Post · bonfire.cafe

Almost 29 million requests from AI crawlers defeated by essentially one simple check: if the user agent contains Chrome/ or Firefox/, and doesn't have sec-fetch-mode, it's going into the maze.

Billions of dollars poured into AI, yet, their crawlers are broken by two ifs in an nginx config.

If this all wasn't so sad, I'd laugh.

ayushnix

@ayushnix@social.ayushnix.com · 7 months ago

@algernon Thanks! I added this to my HAProxy configuration in a named defaults section for HTTP(S) services and it works

acl firefox-or-chrome hdr_sub(User-Agent) -i 'Chrome/'
acl firefox-or-chrome hdr_sub(User-Agent) -i 'Firefox/'
acl empty-sfm req.fhdr(Sec-Fetch-Mode) -m found
http-request silent-drop rst-ttl 60 if firefox-or-chrome !empty-sfm

I wonder if the following user agents are legit 🤔

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582
Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1

The client using these user agents has Sec-Fetch-Mode header values as well.

2something@transfem.social

@2something@transfem.social · 7 months ago

@algernon@come-from.mad-scientist.club

Hiya,
I am curious to see what the "maze" looks like. Is there a way I (a human) can preview it?

a very weeny construct 💀

@pho4cexa@tiny.tilde.website · 7 months ago

it's no iocaine but here's @algernon's heuristic for detecting many bots, in Caddyfile syntax

@ai-scrapers {
 header User-Agent *Firefox/*
 header User-Agent *Chrome/*
 header !Sec-Fetch-Mode
}
respond @ai-scrapers "🖕( ︶︿︶)" 467 {
 close 
}

... recently installed on orbital.rodeo 😏

Niko

@niko@gts.niko.lgbt · 7 months ago

@algernon 0.02% human is just straight up depressing

Hagen Bauer :fckafd:

@hbauer@social.caserio.de · 7 months ago

@algernon where can I finde some documentation about this?

aburka 🫣

@aburka@hachyderm.io · 7 months ago

@algernon I just hope you aren't popular enough that they see this and fix it