Post by @flaki@flaki.social

The arms race chugs along...

https://matthiasott.com/articles/webspace-invaders

We were actually relying on Cloudflare's bot protections for a while, but the AI scraping got *so bad* lately (with mitigations turned on!) that we have been forced to spend most of our time just trying to keep the site running and not on actual feature work.

Eventually, our CTO implemented a POW scheme inspired by Anubis that was able to bring things back to normalcy but we shouldn't be the ones cleaning up Big AI's mess!
Unfortunately, as @matthiasott said "[people running AI crawlers a]re externalizing their costs and aren’t particularly concerned about the collateral damage"

Two graphs showing Target Response Time and CPU Utilization for a ~12 hour period. Target response time is showing high volatility, certain percentiles reaching into the 2-4 seconds on peaks, CPU utilization shows elevated CPU usage around 20% with spikes to in once instance 50%. There is an abrupt change two-thirds in, with CPU utilization dropping and smoothing out significantly, and response time targets flatlining around a couple hundred milliseconds (marking the deployment of custom bot mitigations)

Matthias Ott – User Experience Designer

Webspace Invaders

There’s something happening on the Web at the moment that almost feels like watching that old arcade game Space Invaders play out across our servers. Bots and scrapers marching in formation, attacking our servers wave after wave, systematically requesting page after page, relentlessly filling their data stores while we watch our access logs fill up.

⁂

More from

Matthias Ott

6 Replies

1 Boost

Flaki

@flaki@flaki.social 5 mo. ago

So, we are almost two months into deploying bot detection on our site, which have indeed helped alleviate the performance issues (a.k.a our infra teetering on the brink of collapse), but six weeks in we are *still* fixing issues and weird edge cases this change has brought upon us and our users.

Every moment we spend on just trying to keep the site running in the face of the machine onslaught is time we cannot spend on features and bugfixes making our human users' life better, simple as that. It sucks.

https://mas.to/@vrandecic/116300242806761207

Flaki

@flaki@flaki.social 3 mo. ago

Shot https://weirdgloop.org/blog/clankers

Chaser https://social.vivaldi.net/@everton137/116613317111584835

AI scrapers both have a disproportionate toll on the content provider (both in terms of infrastructure cost and cost of time/effort spent just keeping the lights on) AND at the same time extracting value from the commons reducing the viability of the very sites themselves.

This is not new https://en.wikipedia.org/wiki/Tragedy_of_the_commons

Tragedy of the commons

self-interests causing depletion of a shared resource

Weird Gloop

Aggressive AI scrapers are making it kinda suck to run wikis

Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.

Flaki

@flaki@flaki.social 3 mo. ago

Should you be wondering how's our Homegrown Super Botfight Mode faring:

Partial chat screenshot: "Might be worth trialling some 'honeypot' endpoints, have an invisible link to '/you-will-be-banned-for-accessing-this-url' on every page"

3 months later

Flaki

@flaki@flaki.social 2 hr. ago

8-month update on the Crawlpocalypse at $dayjob is basically this blogpost from the ops folks behind the Linux kernel(!) hosting architecture:

https://people.kernel.org/monsieuricon/creepy-crawlies

As I said in chat the other day - when you are not the one paying the cost (of Anubis and similar proof-of-work schemes) you don't actually _care_. It will slow down the onslaught, but it will not stop it.

Enter Residential Proxies, and what you got is not only "when you aren't paying for it, you are the product", but "even though you probably actually paid for it your device is still basically mining the equivalent of bitcoin, but you are not getting any of it while at the same time contributing to train someone's new slop machine, and destroying human-created value and the user experience of human browsing on the internet"*.

2026 is a helluva year

___
* https://spur.us/blog/smart-tv-apps-residential-proxy-sdks but also many "free" apps like VPNs

View

Spur Intelligence Corporation

Nearly Half of LG Smart TV Apps Contain Residential Proxy SDKs

Spur scanned 6,038 LG and Samsung smart TV apps and found 2,058 with residential proxy SDKs, exposing privacy and home network risks.

~2 more replies

Luddicus Mus

@algernon@come-from.mad-scientist.club 2 hr. ago

@flaki FWIW, serving identifiable crawlers garbage with poisoned URLs, and then firewalling off anything that hits poisoned URLs does a great job at not only slowing down the crawling, but stopping a pretty darn significant part of it.

Still sucks that we have to do that, but we can do that.

Flaki

@flaki@flaki.social 2 hr. ago

@algernon yeah that _used_ to work but, as the blog post says:

> There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.