Discussion
Loading...

Post

  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
jonny (good kind)
@jonny@neuromatch.social  ·  activity timestamp 4 months ago

check this out if you want to help preserve the archive of "most local newspapers through most of US history" that had its funding pulled, even if you only have a couple dozen gigabytes to spare, you can
a) make an account on https://sciop.net/ ,
b) run a qbittorrent instance, go to preferences>web ui and click enable,

and just do this

python -m pip install sciop-scraping
sciop-cli login
sciop-cli client add
sciop-scrape chronicling-america --next

and that's all.

if you have spare storage, you can sort by seeders, ascending, and start from there. or subscribe to the rss feed and auto-download it.

this is an archive funded by the library of congress (threatened) and the national endowment for the humanities (actively being eliminated). the alternative is that an enormous amount of US history that doesn't percolate into history books is owned and operated by lexisnexis and other for-profit data brokers.

this is the first run of some tooling to lower the bar for participatory scraping - at the moment, the archive is still online, and the scraper will automatically embed a webseed URL in the created torrent. so even if you don't have space to seed, you can scrape the data, upload the torrent, and make it possible for waiting peers to become mirrors

#sciop

  • Copy link
  • Flag this post
  • Block
Moritz Negwer
@moritz_negwer@mstdn.science replied  ·  activity timestamp 4 months ago
@jonny this entire thread is amazing, top-notch tool development for a noble cause.
@ #academia : if you feel desperate about the wholesale breakdown of science under the current US administration, consider helping out with #SciOp: Decentralized backups of datasets under threat, in a torrent swarm.

Have a disused laptop or Raspi? Make it part of the swarm and take the data outside the US (or any) administration's grasp!

#scienceunderattack #bittorrent #decentralizedbackup #libraryofcongress

  • Copy link
  • Flag this comment
  • Block
jonny (good kind)
@jonny@neuromatch.social replied  ·  activity timestamp 4 months ago

bug reports here: https://codeberg.org/Safeguarding/sciop-scraping

  • Copy link
  • Flag this comment
  • Block
jonny (good kind)
@jonny@neuromatch.social replied  ·  activity timestamp 4 months ago

the qbittorrent client part and the sciop-cli client add part are optional if you want to just manually add torrents to your client the old fashioned way, but yeah the goal is to start automating bittorrent clients using coordination from trackers that serve as hubs for information organization. so we're starting here and going to use this route to force mutable torrents to exist by sidestepping the stodgy client ecosystem lol

  • Copy link
  • Flag this comment
  • Block
jonny (good kind)
@jonny@neuromatch.social replied  ·  activity timestamp 4 months ago

we manually approve all uploads to sciop.net unless we know you personally, so if you are joining on the great kronkelianation please dm me and lmk your username on sciop

  • Copy link
  • Flag this comment
  • Block
jonny (good kind)
@jonny@neuromatch.social replied  ·  activity timestamp 4 months ago

automate the computer parts (scraping)
manualize the human parts (moderation)

  • Copy link
  • Flag this comment
  • Block
Log in

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.0-rc.3.21 no JS en
Automatic federation enabled
  • Explore
  • About
  • Members
  • Code of Conduct
Home
Login