I heard it through the grapevine that the Library of Congress is accepting bids to become their #WebArchiving vendor. The documents provide a little window in on some of the details of how they currently do web archiving (transferring Bagit packages from S3) and the reports they generate to monitor it.
https://sam.gov/workspace/contract/opp/a2c5551af2b74c3d84c775032c83a55e/view
It's interesting they want to be delivered a sample crawl to evaluate. You can see the list of seeds they want crawled over a 48 hour period in the attached Word file:
* J2b - Sample_Web_Crawl_Seed_List_20250917 - (pp. 13).docx
@ink ooooh nice find 👀
@ink Interesting! From a quick read, WARC/CDX/BagIt are required, Heritrix not.
«large, standard (e.g. Heritrix) crawl, and a smaller browser-based crawl [...] around 16,500 seed URLs across all crawls is 70/30 standard to browser-based»
«Contractor must delete all copies of captured content stored by the Contractor.» Ok for password-protected URLs, strange otherwise.
Background checks for employees.
"Contractor publicity" is why the IA does not (?) publicly say they do this.
@nemobis Yes there are probably .gov rules around that. I'm pretty certain LoC switched to MirrorWeb away from Internet Archive ~5 years ago? I don't remember when and it's hard to find info about it quickly on the internets (for said reasons perhaps).
