Post · bonfire.cafe

@algernon@come-from.mad-scientist.club · 8 hours ago

Another day, another person asserting confidently that determined crawlers will get through defenses, and I'm sitting here, calmly sipping my evening hot chocolate, watching iocaine having served over 50 million requests in the past 24 hours.

Sure, that's about the max my VPS can handle, but if I turned off zstd compression for garbage responses, it would be able to serve about twice as much, if not more. But I like to keep my bandwidth down, so ~650req/sec is what I'll top out at.

It is entirely possible to tell the bots to fuck off. It works, it scales well, it isn't even very hard.

solo

@solonovamax@tech.lgbt replied · 3 hours ago

@algernon and sure, maybe some determined bots can get through it

but like, the vast majority of bots are not determined

and in order to be made "determined", these companies need to spend a bunch of engineering hours on this, which can then be easily undone with some relatively minor changes (read: wasting their time & money)

Byte

@Byte@rage.love replied · 7 hours ago

@algernon without subjecting visitors to the one thing computers are famously better than humans at: PoW challenges

famfo

@famfo@frogs.lgbt replied · 8 hours ago

@algernon my main defense is still just giving requests a cookie without further checking and that works so well that I still didn't bother adding more checks

famfo

@famfo@frogs.lgbt replied · 8 hours ago

@algernon sure, bypassing it is easy but nobody does

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 8 hours ago

At the time of this toot, iocaine saw 50.96 million requests in the past 24 hours, 0.38% of those weren't served garbage. That's about 193.7k requests. Of those 193.7k, 141.6k is various fedi software, leaving ~50k requests.

Not counting fedi software and other good bots, iocaine let about 15k requests through.

Some of those might be bots, but compared to the 50 million? Even if 90% of those 15k requests were bots (they weren't), the system still works.

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 8 hours ago

Looking at those 15k requests: ~6.5k were Firefox/144 and Firefox/145, all of the requests look legit. That leaves us with ~8.5k requests. Of those, 1.6k is a python crawler that is definitely a bot, and it's crawling URLs that only existed in the maze ~6 months ago. While it did not get caught, it never saw anything but 404.

We have 6.9k, about 1k of that is my CI. 5.9k left, mostly various Firefoxen and Chromes, with reasonable access patterns.

So, yeah, most of that 15k is legit requests.

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 8 hours ago

While 15k might sound a lot, consider that loading the Mastodon frontend on my old instance is ~300 requests on load alone.

Loading my old profile is ~280 requests.

When you factor in that a visit usually ends up doing 4-5 requests, we're down to ~3-3.7k visits in the past 24 hours, and across a dozen or so hosts, that's not very much. Maybe a few hundred visits / host. That used to be about my normal visitor count in the mid-2010s.

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 8 hours ago

Now, I am using a complicated defense script. But iocaine 3.0's built-in script would have stopped ~38 million of those, and a firewall level block of the Huawei & Alibaba ASNs would have blocked another 11m.

That's 49 million requests stopped out of 50.96 million, or 96.67% of all requests. That's a very good rate for something as trivial as this pseudo-code:

If requester's ASN is Huawei or Alibaba: block
If user agent is in ai.robots.txt: block
If user agent has Firefox/ or Chrome/, and does not have a sec-fetch-mode header: block

Two ifs in a trenchcoat and a firewall rule, and your bot troubles are gone.

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 8 hours ago

And you know what? I'll teach iocaine's built-in handler to allow matching by ASN. Point at a Maxmind mmdb-format ASN database, and boom, it does 99% of what my elaborate defense script does, in ~1000 lines of roto (including a test suite).

algernon, the mad

@algernon@come-from.mad-scientist.club replied · 7 hours ago

Oh, about resource use:

At this moment, my VPS is handling ~400 req/sec. Load is ~3.5. Most CPU use is by Caddy (~100-130%), iocaine (~40-50%) and Vector (~20-25%). Memory use is not relevant, there's ~1.5GiB available for anything that needs it. Some of it is used for filesystem buffers & cache & whatnot, but that's how memory should be.

The bottleneck is CPU, and the majority of CPU is used by Caddy, which hogs an entire core and then some. A large part of that is zstd compressing most responses - that's not cheap. But it saves a lot of bandwidth, and I consider that a fair trade.

iocaine itself is using half a core at most, Vector is just chugging along. Meanwhile, my 2014-era Mac Mini with a spinning disk and 4GiB ram is happily ingesting all those logs (2 loglines for every request: one from Caddy, one from iocaine), so about 100 million lines ingested today. It's sitting at 1.31 load, with VictoriaLogs using ~25% CPU.

1+ more replies (not shown)

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances

Bonfire social · 1.0.0 no JS en

Automatic federation enabled