@stefano

Maybe that wave of bots is hyperscalers way of telling you: "you need to buy more powaaaa".
Think about what would average so called "cloud engineer" do ? Answer: scale her/his K8 clusters to the sky to cope with load.

😎

@stefano we need the Internet2 which cannot be controlled by corporations and stuff lile advertising/cookies can be switched off by users.

@stefano
Making web.archive.org (Wayback Machine) as "only single service that is allowed to crawl using robots in the world" and any others commercially want to use data on Internet to purchase data from them (of course, only allowed to do ones. Would need i.e., "no AI, no commercial" option in robots.txt not to be sold to AI things) seems to be the way to go.
And force purchasing data from Wayback Machine to be outside Internet (dedicated leased line, for example) would significantly lower the unndeeded traffics.
Keeping Wayback Machine in good manner (strictly obbey robots.txt, restricting crawling frequencies, contracting with authors directly by Wayback Machine if contents are allowed to be sold, and so on) would be needed, too.
This way, all "allowed" contents that are not too often (over once a day, for example) to be updated could kept public even when the server services are gone disregarding the intentions of authors there.

@stefano packet radio 😅 Uhm… yes another layer of network “tor style” seems to be necessary. It is painful to give them the internet, and having to use another “stealth” layer. We will need to import vital resources to this. No pb for the hobbyists, but I don’t expect banks to be open to the idea of existing on some over network. I would applause my bank if it had an .onion :-)

@stefano This is so frustrating and infuriating to see.
I had a discussion with a few folks on here just yesterday and we came to the same conclusion... just have a private separate network and leave the internet to the robots and corporations. It seems more and more like a lost cause...