Discussion
j'essayais de mettre iocaine mais en fait y'a pas de système de reconnaissance des crawler ia, c'est juste un truc qui enquiquine les crawler IA une fois qu'on les a identifiées
ça m'arrange pas des masses pour l'instant même si j'aime l'idée aussi, là j'ai déjà un blocage basé sur les useragents mais lol elles s'en foutent
j'essayais de voir s'il y avait des ip mais j'ai pas trouvé de projet qui répertorie un max d'ip utilisées par les bots
@PhieLaidMignon
Les IP c’est mort, les crawlers utilisent des IP domestiques dynamiques, ingérable.
Anubis ou Iocaine franchement c’est bien, la plupart des bots généralistes sont cons et taillés pour justement se faire passer pour des navigateurs donc avec un UA comme tel. Moi ça bloque 80% du trafic en moyenne, je trouve ça plutôt pas mal.
@barbapulpe @PhieLaidMignon ce qu'a découvert @algernon c'est que lorsqu'on ne bloque *pas* les bots (par exemple dans robots.txt), ceux-ci se déguisent moins et il est plus facile de les détecter ensuite.
par ailleurs, le «truc» de iocane est de placer des liens cachés qui redirigent vers un labyrinthe. vu que seuls les robots peuvent «voir» et donc «cliquer» le lien, on n'a plus qu'à enregistrer/bloquer l'adresse ip du client qui visite le labyrinthe.
@dt @barbapulpe @PhieLaidMignon Apologies for chiming in in English, my French is... pretty much non-existent. I kind of understand what the thread's been about so far, but writing Frech is not something I can do - please correct me if I misunderstood something.
The trick iocaine employs is that it catches the trivial bots (either via user-agent, or via simple heuristics like a missing sec-fetch-mode header over HTTPS if the user agent includes Chrome/ or Firefox/), and feeds them an infinite maze of garbage, with poisoned URLs. Those poisoned URLs are the real trick: a lot - most, in my experience - of the bots the simple UA / sec-fetch-mode check does not catch, use an URL queue collected by the trivial bots. If we fill that queue with garbage, with URLs we can identify as only ever existing in the maze, we can trivially identify remote controlled Chromes from domestic IPs too.
Recent versions of iocaine also come with a firewall feature: you can firewall certain requests off automatically. For example, if anything hits a poisoned URL, they'll get firewalled off after the first request. After, because we still want to serve them a set of poisoned URLs. But we stop the rest.
This firewalling cut down my daily requests from 100m+ to a mere 2.5-3.5m (and I still serve garbage to 90%+ of them, just to keep those queues filled with trash).
So, yep, iocaine does not have a very complex bot detection mechanism - because it doesn't need one. Half the bots are trivially detectable, and the rest use URL queues collected by the other half.
@algernon
then there is something I don't understand, to me iocaine was only about the maze, not the catch part, in my nginx conf I redirect to iocaine only if useragent is of a certain type
will try to understand this better, thanks
@dt @barbapulpe
@PhieLaidMignon It used to be a tarpit only, it started like that, and was a pure garbage generator for a while.
The detection part came later, in version 2.2.0 (released in June 2025), and the whole URL poisoning scheme came even later (arriving to iocaine's built-in script in 3.1.0, a day before last Christmas; though, it was available in the non-default Nam-Shub of Enki script since ~July 2025).
(If you, or anyone else in this thread, or outside of it needs help, or wants to know more about iocaine, feel free to ping me, I'll do my best to help!)
@algernon hello !
So I've setup iocaine, it seems to be filtering traffic fine but my peertube is... broken on some parts, can't find out why, I think it could be related to the use of error page
a token isn't transmitted anymore
do you think there are other ways than relying on error_page for nginx config ?
edit: just saw this https://nginx.org/en/docs/http/ngx_http_auth_request_module.html
could be useful
@PhieLaidMignon I don't think there's any other way than via error_page currently1. Can you share your nginx config with me? And/or peertube paths that don't seem to work? Those would help me narrowing down where the problem might be.
I'm working on an nginx module, but that's a 2027 thing, most likely. ↩︎
@algernon I'm almost there using the auth request module !
My only problem is that Iocaine returns 200 if it's an AI, 421 if human
so for now my nginx conf blocks... humanns and not AI
I was able to change 412 to 200 through nginx but not able to change 200 to 401
I was wondering if we can customize status code ?
@PhieLaidMignon Technically, yes. iocaine itself is a glorified script runtime, the "brains" is the script you run. It ships with a built-in script, you can configure both the garbage status code and the fallback, see here:
declare-handler default {
garbage {
status-code 401 // the status code to use when serving garbage
fallthrough-status-code 200 // the status code to use when signaling the reverse proxy to serve the real stuff
}
}
If you're using Nam-Shub of Enki, you currently have to edit the code (NSoE's, not iocaine's).
@algernon thanks !
Hmm just wondering, I guess I might need two iocaine
one for bot detection that returns 401 if this is a bot, and one to generate garbage that returns 200
or I turn 401 into 200 in case of error
@PhieLaidMignon Running two servers should be fine, though, currently, if you do that via nginx, it will generate garbage twice, which is a bit of a waste.
The trusted-decision-header setting can be used to ferry the decision made by one handler to the other. But the http-server thing in iocaine is set up that it will always try to generate garbage.
A bit of hacking around in the Roto sources could address that1, or if you put HAProxy inbetween (iocaine's HAProxy SPOP agent thingy only runs the decision making, and if you set the trusted decision header, the http-server's script will only do garbage generation, and trust the decision in the header).
Install bonfire.cafe
Get the full app experience