I talk to people about web archiving a lot, and it feels like so many people think it’s not something *they* can do - they just have to wait and hope someone else saves a site for them. But *you* can save web history too! Here’s how! https://blog.mistys-internet.website/post/2026/10/29/if-you-love-a-website-back-it-up/
Look, that’s an entire archived website embedded right into this blog post. And I did it all with free and dead easy open source tools! If you’ve got any websites or social media posts you care about, please consider going out there and archiving it yourself while you still can.
I would love, love, love to see thousands of small web archives pop up all across the internet as people collect the things that are important to them and make sure they’ve got backups under their own control. The internet is decentralized - web archives should be, too. Do it yourself, or band together with groups of your friends to take care of sites that are important to you as a group.
And if you're wondering about the dead site that inspired this blog post? I *did* recover something from it... just not using these tools, since the original site is no longer online. I'm writing a separate post on how I did it, but in the meantime, if you’d like to see teapuppies dot com for yourself: https://teapuppies.mistys-internet.website/
@misty also: Whenever you have old computer harddrive images, look for the browser cache and visited urls.
Sometimes urls are just no longer found these days and they might reveal hidden treasures in archives.
@misty i've been trying to back up pages myself, too, despite my slowly-decreasing SSD space! i keep thinking about a flash-based website i loved that didn't archive properly on IA which makes me wish it was able to be preserved better....
@swiftyshq Yeah… it’s a shame tools for crawling flash didn’t get better before the sites themselves largely disappeared.
@misty
@lavaeolus
For people who are concerned about websites' contents but shy away from 'nerdier' tools, #Obsidian Web Clipper is a nice suggestion. Of course, you need to be using Obsidian.
@misty ohh zhis is so cool. a little website in a browser in a website in a browser
@misty@digipres.club I stumbled into a rare opportunity a few years ago where I was not only able to resurrect a dead site, but also restore it to its original domain, so all old dead links got fixed!
@natalie Oh wow, amazing! I’m glad you were able to do that.
@misty The archive.page browser extension you mention in this looks like the perfect tool for us to use, except that we refuse to ever use a chromium-based browser (which includes the embedded browser in its desktop program, which i don't think would really be viable for us anyway as it would add too many extra steps to the process)
Are you aware of any equivalents for firefox-based browsers, or whether the extension you linked has any plans to make a firefox version?
-F
@Hearth Unfortunately, the reason it’s Chrome-specific is because it’s using the Chrome debug protocol in order to puppet the browser and retrieve data. I’ve seen a bunch of other tools written around the Chrome debug protocol specifically because it happens to be very well-featured for this kind of task, even if it wasn’t intended that way. I’m not sure if Firefox’s equivalent has everything it would need to build a tool like this around it.
@misty I wanted to do this - but the site was badly built - pre mobile devices - fixed sizes. It disappeared. Found the remnants of it on Internet archive. So I rebuilt it and hosted it. HTML and gifs. That cartoonist was an inspiration to me in my formative years. I will look at these tools.
https://stooryduster.co.uk/lobey-dosser/index.htm
@misty i’m curious if you have an opinion about ArchiveBox. It’s the archiving tool I started using recently, specifically to make more durable backups of my Wallabag saves. It’s been a bit processor intensive on my home server tho
@CodingItWrong I haven’t used it myself, but I’ve heard good things. I know it uses wget for WARC generation, which makes me wonder how it handles JS-heavy sites.
@misty Save Page Now on the Wayback Machine is the lowest bar and still the one most people have never heard of. Paste a URL, wait a minute, and there is a dated copy anyone can cite.
@marintkael Yes, definitely! It’s great for single pages. I’ve found it’s a little spotty for social media posts recently though.
@misty archive.today often copes better with the script heavy ones, since it renders the page before saving. The catch is that nobody quite knows who runs it, which matters once you want to cite the copy later.
@marintkael Also depends what you want from script heavy sites - the DOM capture is really good imo for stuff like articles, but it’s not great for capturing interactive social media profiles.
@misty TIL about the browser extension for it. However, the default link in the article goes to a Google Chrome extension.
For Firefox, I found: https://addons.mozilla.org/en-US/firefox/addon/wayback-time-machine/ , thanks for the tip!
@davidrevoy Yea, unfortunately Archive Web Page is using the Chrome debug protocol to control the browser so it doesn’t work on Firefox.
@davidrevoy (The difference between the one I mentioned and the one you linked is that the one I mentioned does the archiving locally on your device and lets you control the process, while the other one has the work done remotely on the Internet Archive’s servers, with you having a bit less control over the process.)
@misty Thank you, good to know!
@misty I have questions! (...which you are of course under no obligation to answer, but this seemed like a good place to ask them.)
- How does this method compare with asking archive.org or archive.is to archive a page?
- Is there a way to automate it -- like an API, or CLI utility, or library or something?
I'm particularly interested in that second question because I run a wiki which links to a lot of external sites for reference, and those links often go dead over the course of the years (it's now 21 years old) -- so it would be nice if it could automatically archive every page linked.
Thanks for this useful post! :-)
@woozle Those are good questions! Some answers -
1. Asking those sites to do it means the web archives live on their servers - I’d love to see people hosting their own web archives. Both of them are suitable for single pages, but less suitable for full sites or sets of multiple pages like the tools I linked. The DIY approach also puts you more in control over the process, especially for interactive sites.
@woozle archive.is does captures via DOM snapshots rather than the original documents + requests, which is great for some kinds of sites but does mean that interactive elements on sites aren’t preserved. I think it’s a useful tool for articles (esp. loginwalled ones), but less for some kinds of websites. Just worth being strategic about.
Wayback Machine Save Page Now currently has trouble with some kinds of social media sites, whereas the tools I mentioned are better at those right now.
@woozle 2. Yep! The browsertrix crawler is what the Browsertrix web app calls, and it’s a separate CLI app you can run on its own and script/automate around however you want. https://github.com/webrecorder/browsertrix-crawler
As for other CLI tools, wget can produce WARCs but I don’t believe it has the same support for working with interactive/JS-heavy sites since there isn’t a full browser in the picture. Another CLI tool, ArchiveBox, uses wget under the hood.
@misty Awesome info, thank you!
...and the issue with highly JS-driven sites like fedi and some other social media is a particular concern; right now I don't have any tools which can reliably archive a fedi thread as text, though there is a tool on Mathstodon which creates a thread-map as an SVG... but I'm wandering off the topic a bit. 😅
@misty Figured you'd be a good person to ask: is there an "agreeable" way to download pages from the Way back Machine? I've seen the "wayback" Python lib, but I was wondering if there was a more accepted method of mirroring things from there. I have a non-CGNAT connection and I'm somewhat afraid of being IP banned or something by the IA for looking like an AI scraper or something of the sorts.
I agree with this, maintaining my own website, means I get to pick / choose what goes on the site and where. Pages / posts may be subject to removal at my own whim, so as I get very little feedback on anything, then I may remove a page or move the content elsewhere.
When looking at other websites I may e-mail myself a link to a site or page within a site for later reference.
So thank you for these tips. they may prove most useful in this process.
@misty aaah this makes so much sense and yet I never considered it, thanks!!
@misty I haven't used your tools (yet), but I've used some of the kiwix scrapers to get some stuff that's important to me.
And I'm keeping some of the kiwix library seeded.
@misty #digipres