Dear fedi, I am thinking of building-in some LLM scraper bot traps into my website.
One of the ideas is links down near the bottom of each blogpost or page that are hidden in CSS (so that no human would click them) that when clicked immediately put the client IP address on naughty list.
I want to understand better how CSS-hidden links work for #Blind visitors and others using screen readers or other assistive technologies.
The last thing I'd want is to inconvenience any human! :blob0w0:
1/3
Also consider how someone using an LLM to workaround accessibility issues in your site may react. Remember that they have the right to choose their own tools and if you are going to remove one. You should consider making it clear to them via other accessibility tools that LLM based tools will not work.
Also, LLMs could use that to avoid those links as well.
Part of my project is feeding Markov-generated slop to these LLM scrapers. And doing it very very slowly, to waste their resources (open socket, etc) on something that will make them actually worse.
Check out this curated list of tools and methods titled "Sabot in the Age of #AI" by @asrg: it might include some approaches that could help your case.
https://algorithmic-sabotage.github.io/asrg/posts/sabot-in-the-age-of-ai/
Right now I have the code and it works, so the next step is deploying it, and for that I need good ways of trapping the bots.
Hence this thread.
(and I reverse-tip my hat to the idea that a scraper feeding data to train LLMs is not different than a person. Ayn Rand would blush at this. Someone's been trapped in an entirely different tarpit there...)
@pixelcode the approach is two-pronged:
- long-term IPs (like the ones mentioned in OpenAI's documentation) are hard-blocked semi-permanently;
- random IPs that are heuristically determined to be LLM scrapers (for example, they "clicked" a link no human had access to) are fail2banned for a certain time.
That way those residential IPs will eventually be able to access the site again.
@pixelcode we will never know until we try.
The most recent and imo clever approach I've seen was Anubis though: https://anubis.techaro.lol/
@iris I mention the first two projects here:
https://git.rys.io/libre/markov-tarpit
I am aware of Anubis but I want to avoid any JS on my website, and as far as I understand Anubis requires JS.
Great minds etc etc.
Which is ready for deployment, I just need to figure out how to trap LLM scraper bots into it.
The bots parse content fully rendered, with CSS applied and a couple of seconds after onLoad fires or after the last layout shift.
The bots also use natural language processing to parse the content, so hiding things using CSS generally doesn't affect them.
As long as it does not negatively affect users using assistive technologies, I am still willing to test it out and see if any LLM scrapers get caught into that net.
Worst case scenario, I will have hard data on this myself.
But if you zoom out of the problem: you're never going to pull this off, sorry.
A bot is really just a user. One with "accessibility challenges". A bot may be "visually impaired", or may lack the ability to operate a mouse etc.
So any hurdle, or misdirection aimed at bots, will inevitably hurt "organic users". And will punish those who rely on assistive technology, more than others.
And really: isn't AI, a form of assistive technology?
Is there a group of users that I want to forbid reading my work? Do I want to show a group of users false or misleading content?
If yes, then I'll have to define this "group of users": geografic, based on mental and physical abilities, based on technical abilities, usage of soft- and hardware? The easiest discriminatory trait would be "has an account" or "has paid".
But, for me, that's never the web I fought for.
> So any hurdle, or misdirection aimed at bots, will inevitably hurt "organic users". And will punish those who rely on assistive technology, more than others.
This is obviously, demonstrably false.
Looking at my webserver logs I can tell you that I can set a rate-limit that would *never* be reached by any human user, but which would deal with a bunch of bots.
> And really: isn't AI, a form of assistive technology?
No.
> No.
To me it is. Personally. As to some elderly family members who never got the hang of computers and only little of phones can use chatgpt. A cousin with dyslexia who can interact with internet through voice. An analphabetic foster brother who can now finally be part of whatsapp groups through tts and stt.
Saying it isn't assistive either shows how entrenched you are, or how uniform your social environment is.
Demonstrably hurting humans. We've had this. Hurt users that shared an IP (university, shared student housing, vpn) or subnet even.
Hurt humans that were offline for a while and when finally had coverage, would barrage events to our API.
Hurt humans that legitimate used TOR.
Hurt humans that automated tasks through our API with amateurish scripts.
Anyway, I had tried to gently suggest you go somewhere else have your opinions – you are more than welcome to have them in your own thread for example.
I am now directly asking you to stop bothering me with them. There is a reason why I wrote that last paragraph in the second toot of my thread.
I am obviously doing research about this myself, so no, I am not going to only throw this out into the void and expect free advice. :blobcattea:
But, if you do have advice or suggestions or notes or input here, I would love to hear it. :blobaww:
Especially if you use assistive technologies, or have experience/expertise in that area. 👀
If you are building your own LLM scraper or are otherwise an AI bro, and have Opinions about the Open Web, please feel free to go suck a lemon instead. 🍋
2/3
Context is: my Markov Tarpit is ready for deployment, and I need to figure out the heuristics around trapping LLM scrapers in it
https://git.rys.io/libre/markov-tarpit
3/3
@rysiek@mstdn.social For your information, some people also have CSS disabled on their browser or use HTML-only browsers that do not support CSS, so they would also be at risk of being caught by the trap.
What i strongly suggest is a warning text that humans can easily understand, like "do not click the following link, it will blacklist you"
Would be AMAZING is someone could build an CSS-modificator to make pages static
For my site, I opted for a CSS of "visibility:hidden;", and a link that is leads to the Iocaine honeypot.
Thank you for asking the question, because on review with a text browser this shows up, so I expect screen readers would see it as well.
I'll update my honeypot entrance with a human-readable description to reduce confusion.
A space for Bonfire maintainers and contributors to communicate
