"The core technical challenge was clear: How do we allow investigators to move fast while maintaining a forensic chain of custody? A simple screenshot is insufficient for legal or historical proof. We needed a system where an investigator could claim, “at this time and date, I browsed this unique URL which contained precisely this content,“ and be able to back it up with cryptographic proof."

https://dispatch.starlinglab.org/p/pilot-project-on-making-web-preservation

#webarchiving #digitalarchive

duckdb-web-archive-cdx 👀
DuckDB extension to query web archive CDX APIs directly from SQL.

https://github.com/midwork-finds-jobs/duckdb-web-archive
#webarchiving

duckdb-web-archive-cdx 👀
DuckDB extension to query web archive CDX APIs directly from SQL.

https://github.com/midwork-finds-jobs/duckdb-web-archive
#webarchiving

wow, this is huge: World's largest Internet Domain Database https://ip.thc.org/
can be relevant for #webarchiving activities.

bulk data available, in parquet format: https://ip.thc.org/docs/bulk-data-access

"The core technical challenge was clear: How do we allow investigators to move fast while maintaining a forensic chain of custody? A simple screenshot is insufficient for legal or historical proof. We needed a system where an investigator could claim, “at this time and date, I browsed this unique URL which contained precisely this content,“ and be able to back it up with cryptographic proof."

https://dispatch.starlinglab.org/p/pilot-project-on-making-web-preservation

#webarchiving #digitalarchive

Wikidata and decentral archives:

At the workshop Subverting Archival Practices (https://perfomap.de/newspictures/workshop_berlin_nov.pdf), I presented my thoughts on the role of #Wikidata for the now very widespread online archives of projects as websites (e.g., https://talkingobjectsarchive.org/ https://dekoloniale.de/ https://archive.transmediale.de/ https://archiv.ngbk.de/). Linking persons, events, places, objects on these websites to Wikidata would make them more visible, but also allow for research across these different archives.

A new aspect here is also Wikidata's possible role for #webarchiving, i.e. the storage of endangered websites in containers, namely by enabling references to individual pieces of information in the container once the original website is no longer available.

This was also tested with students of @magdalena at Aarhus university in October in the workshop "Data Modelling and Web Content at Risk", they referenced information from the https://dekoloniale.de/ website on Wikidata: https://cc.au.dk/en/c4cdp/news/show/artikel/data-modelling-and-web-content-at-risk-students-of-the-curating-data-course-explore-digital-preservation-through-wikidata-and-linked-open-data

Wikidata and decentral archives:

At the workshop Subverting Archival Practices (https://perfomap.de/newspictures/workshop_berlin_nov.pdf), I presented my thoughts on the role of #Wikidata for the now very widespread online archives of projects as websites (e.g., https://talkingobjectsarchive.org/ https://dekoloniale.de/ https://archive.transmediale.de/ https://archiv.ngbk.de/). Linking persons, events, places, objects on these websites to Wikidata would make them more visible, but also allow for research across these different archives.

A new aspect here is also Wikidata's possible role for #webarchiving, i.e. the storage of endangered websites in containers, namely by enabling references to individual pieces of information in the container once the original website is no longer available.

This was also tested with students of @magdalena at Aarhus university in October in the workshop "Data Modelling and Web Content at Risk", they referenced information from the https://dekoloniale.de/ website on Wikidata: https://cc.au.dk/en/c4cdp/news/show/artikel/data-modelling-and-web-content-at-risk-students-of-the-curating-data-course-explore-digital-preservation-through-wikidata-and-linked-open-data

🤔 Curious about how CiVers will work?

📹 We’ve shared a new (German-language!) video that walks you through the core idea behind CiVers and shows the very first steps of our software development.

If you want to understand what we’re building, and why versioning and PIDs matter, this is a great place to start!

👉 Watch the video:
https://www.youtube.com/watch?v=Jje8Kb8xJL0

#DigitalHumanities #academia #OpenScience #WebArchiving

📣 New blog post! 📝

October 14, we hosted our first #CiVers workshop at the @dai_weltweit in Berlin 🏛️ This was a great opportunity to exchange ideas on citing versioned web resources and managing research data in #archaeology and the #humanities

Read more about what we discussed 👇
🔗 https://www.dainst.org/blogs/noslug/253

#Metadata #Research #OpenScience #DigitalPreservation #WebArchiving #DigitalHumanities

‼️ ATTENTION ‼️

⏰ Proposals for #iipcWAC26, "Sustainable #WebArchiving," in Brussels are due in 1 WEEK (15 OCT): http://netpreserve.org/ga2026/CfP

First-time submitters encouraged! Need inspiration?
https://www.youtube.com/@iipc8855/featured

#WebArchives #WebArchiveWednesday #DigitalPreservation #DigitalHumanities
@webarchives

‼️ ATTENTION ‼️

⏰ Proposals for #iipcWAC26, "Sustainable #WebArchiving," in Brussels are due in 1 WEEK (15 OCT): http://netpreserve.org/ga2026/CfP

First-time submitters encouraged! Need inspiration?
https://www.youtube.com/@iipc8855/featured

#WebArchives #WebArchiveWednesday #DigitalPreservation #DigitalHumanities
@webarchives

I started crawling 80,000,000 web pages for a personal project August 1 after an initial trial "PING-ing" 40,000,000 of them.

Cloudfare limiting is triggered at least twice a day, and almost immediately if I up my own rates.

Currently at 16% as of September 17.

Has rate limiting severely limited web archiving capability in the advent of AI?

Should I be crawling from multiple IP addresses?

What volume pages are likely to be crawled in day-to-day archiving processes?

#WebArchiving #Digipres

A vital record of Black life on the Mississippi Gulf Coast is now online.

⭐ The Mississippi Star, digitized by Hancock County Library System Community Webs, preserves local voices, culture, & civil rights stories.

Accessible to everyone, these issues now live on the web we built. 🌐

Learn more on our blog ⤵️
https://blog.archive.org/2025/09/03/local-voices-lasting-impact-digitizing-a-community-magazine-in-hancock-county-mississippi/

#CommunityHistory#Libraries#WebArchiving

A vital record of Black life on the Mississippi Gulf Coast is now online.

⭐ The Mississippi Star, digitized by Hancock County Library System Community Webs, preserves local voices, culture, & civil rights stories.

Accessible to everyone, these issues now live on the web we built. 🌐

Learn more on our blog ⤵️
https://blog.archive.org/2025/09/03/local-voices-lasting-impact-digitizing-a-community-magazine-in-hancock-county-mississippi/

#CommunityHistory#Libraries#WebArchiving

Social media web archiving 3/3

Additionally it would be helpful to process these feeds again and record author, date and post in a database.

I'd appreciate pointers to forums, chats, mailing-lists where this is discussed...

I suspect that the webarchiving and digital preservation community could (and should?) take inspiration from the scraping scripts of the tech industry, on Github I saw some of those tailored at extracting content from feeds.

#webarchiving #digipres

One thing I am wondering with current web archiving: are there any new approaches for how to deal with social media content?

I mean this problem:
Platforms are full of scripts. We use automated browsing to trigger those scripts and record them in a format to replay them. ( #webrecorder #browsertrix etc.). Which makes sense from a historian's point of view, you record the info in its media format. But often the scripts don't replay, we don't see the content.

thread 1/3

#webarchiving #digipres

📣Start planning your proposal for #iipcWAC26, “Sustainable #WebArchiving,” today! 📣

🗓️ Proposals due October 15
🇧🇪 20-23 APR 2026 at KBR, Royal Library of Belgium

For more info: https://netpreserve.org/ga2026/cfp/

Need inspiration? Check out past presentations: https://www.youtube.com/@iipc8855/featured

#webarchives | #DigitalPreservation | #DigitalHumanities | @webarchives

📣Start planning your proposal for #iipcWAC26, “Sustainable #WebArchiving,” today! 📣

🗓️ Proposals due October 15
🇧🇪 20-23 APR 2026 at KBR, Royal Library of Belgium

For more info: https://netpreserve.org/ga2026/cfp/

Need inspiration? Check out past presentations: https://www.youtube.com/@iipc8855/featured

#webarchives | #DigitalPreservation | #DigitalHumanities | @webarchives

Google’s shortened goo.gl links will stop working next month
https://www.theverge.com/news/713125/google-url-shortener-links-shutdown-deadline
https://en.wikipedia.org/wiki/Google_URL_Shortener

I cannot say if it is a trusted source. Here, they say that 3.6 billion URLs have been shortened. https://dub.co/blog/googl-links

A single small server could have served as an HTTP redirector, for lifetime.
Nice job, Google!
#webarchiving