Discussion
Ah shit. Been hit with a wave of 37k bot hits on my blog. As much as I'd like to believe that thousands of people in Uzbekistan and Iraq love my posts from 2008, I think I'm being scraped.
Can anyone personally recommend a PHP/WordPress way to block AI scrapers?
(Please don't tell me to use a different blogging platform. I also don't want to use a CDN. Sympathy not required but practical advice welcome.)
@Edent I used to use “Blackhole for Bad Bots” https://perishablepress.com/blackhole-bad-bots/, but it seems it doesn't play well with caching, and apparently kinda silently stopped working for me at some point last year.
I just went looking for some tar-pitting options, and found “JTZL's Bot Maze” https://wordpress.org/plugins/jtzl-bot-maze, which I am now trialling. On the surface, it does the trick (fake content, tar-pitting, and outright banning after a while), but we'll see.
It's got a very small user base, which isn't great, but not necessarily a red flag, and the codebase looked ok from a cursory look. I have automated updates disabled for now. We'll see.
Ended up deleting about 36k AI bot visitors.
Turns out most of them use comparatively ancient versions of major browsers - so I just whacked anything more than a few years old.
Not exactly sophisticated, but did the trick.
I'm reluctant to add anything annoying for human visitors. I really don't want to be beholden to a massive CDN.
Will do some thinking on it.
@Edent amongst other things I tend to use the user-agent lists from https://github.com/ai-robots-txt/ai.robots.txt and have a little script for turning those into a small apache htaccess conf file that matches the browser against them and denies.
I know a bunch of the bots are faking their user-agent but this approach still catches and blocks between 2-21k connections a day on my small personal hosting machine.
@Edent I haven't tested but I saw that you could use some flag in robot.txt and that should stop Ai bots, though shortly afterwards I read that most Ai bots ignore the robot.txt
@Edent sympathy offered! I use a CDN because I couldn't find a better way, same set up. I'm going to bookmark this because I'd be curious to know what others have figured out. I find it so frustrating to have to manage this problem at all.
@Edent Try Z! I've never used it, but it might work with N. :-p
@Edent I've noticed that a lot of the scrapers coming to my site recently seem to be using variants of this user agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/605.1.15
Blocking variants on that might help a bit. I also block anything coming from *compute*.amazonaws.com, and I use a honeypot plugin on top of that.
@Edent This is not practical help, just *me too*. On Friday my Newsgrouper site started getting hammered by what I believe to be AI-driven scrapers which appeared to have been *taught* how to navigate my guest login process. Since then I've had guest login disabled, I'm now trying to devise and implement my own captcha-type gating before I turn guest logins on again.
(Not practical help because I run my own Tcl-powered server.)
#webdev #ai
@Edent I use two methods:
1. Apache rewrite rules that block the typical requests from distributed AI scrapers - they often add lots of query string arguments, or make requests for event listings from a long time ago.
2. Firewall blocks on countries that typically wouldn't have people who would be interested in my websites. Brazil and Vietnam have been top scrapers in the past, apparently their ISP-provided routers are easily hacked to work as proxies.
It's a serious annoyance. The sooner the "AI" bullshit bubble bursts, the better.
@Edent
Hi Terence.
Sorry to hear about your problem.
I use a free WP plugin called - creatively - 'Block AI Crawlers'.
Its got a lot of good reviews but its a little like shark repellent spray: There are no sharks around but you don't know if its because its working or just because the sharks haven't come for you yet!
I also geoblock a small handful of 'risky' countries: Iran, Russia, North Korea for example. It just seemed sensible to reduce the attack surface.
@Jaimieserotica ta, I'll take a look 😀
@Edent If you want a suggestion on an IP to block, start with these assholes. *sigh*
@web_goddess from the looks of things they're using hundreds, if not thousands, of different IP addresses. Probably residential proxies.
@Edent Ah, that sucks. Other than some things at a CDN level, the other thing I tried was copying Robb Knight's robots.txt since it seems pretty comprehensive: https://rknight.me/robots.txt
@Edent why not anubis?
@runawaydevil does it work on a PHP only web host?
@Edent If it’s shared hosting that allows PHP only, without the ability to run your own services or configure a reverse proxy, then no. Anubis is a separate service that needs to run in front of the website. The website itself can be PHP, but you need some level of control over the server/proxy to place Anubis in front of it.
@Edent do you have control over the proxy?
@runawaydevil no. That's why I specified PHP and in my original post.
@Edent put #vinylcache in front. Remove all cookies. Put /admin/ behind http auth
@Edent I am dealing with a similar problem for a client at the moment. Unfortunately it's very difficult to do anything in WordPress or PHP - partly due to the fact that you've already paid the cost of dealing with the request at that point.
The things I've had luck with are:
1. Add known scrapers to robots.txt (helps a bit)
2. Disable any plugins you don't need
Serious action does require you to login to the server though and tweak things in Apache/nginx etc.
@Edent it's not quite what you're after but possibly a useful thought... I put go-away in front of my gitea instance last week -- it's a reverse-proxy that requires proof-of-work to let clients through to your application. It may not suit how you deploy Wordpress though.
@fooflington imagine you asked me for a personal recommendation for a nice place to get some coffee in London.
I reply by telling you to to fly to Malaysia where you can cultivate your own tea leaves.
Would that be a helpful answer?
@Edent
I block them at server level. I have several of them blocked regularly. PHP script in general just eats up more resources.
hosts.flossboxin.org.in
@peterrenshaw which one do you personally use which works with WordPress?
@Edent my wife’s Wordpress site has the Defender plugin (free version) no idea how effective it is.
@Edent maybe try something like this: https://github.com/Andrei-Geana/nightshade
A hidden link that only scrapers will find can trigger a defensive response.
@ketumbra you see how that isn't PHP, right?
@Edent I was thinking more about the idea than the specific implementation. I'm sure you can adapt it to your preferred lang.
This one is PHP but not clear what triggers it. I guess as well, you don't want to use even more resources serving rubbish
https://github.com/msigley/PHP-HTTP-Tarpit
I would probably experiment with a fail2ban approach with requests for a hidden honeypot resource or even AGENTS.md
@ketumbra so when i said "personally recommend" - what I was looking for was personal recommendations. As in something you have successfully used. I know how to Google.
@Edent I would love to know myself. @Podnews has some hidden anti-email scraping code, but otherwise AI seems pretty impossible to block. I blocked Russia and Belarus at the CDN level, if only because Podnews is a US company and I didn’t want to break any trade block. (And, really, a capable CDN is the right way to block this but I understand you not wanting that).
I don’t suppose it helps, but https://podnews.net/AGENTS.md is there and a public file. (I should probably make it more visible in the sitemap).
@Edent I use (in php) such things as geoip2 to get the country of ip and make decisions based on that, referer being impossible or not present (sad but them’s the breaks), user agent string, block various ASNs entirely, that kind of thing
@dracos pretty much my approach so gar. Was working well until today 🫠
@Edent
Personal server, yesterday, all traffic, I blocked about that amount, yeah; about 20k from traintimes, 15k from theatricalia (I don't log iptables but have a lot of IPs blocked as well).
(At work, yesterday, c. half of traffic, 296k 418s (upgrade your browser), 526k 503s (just blocked), and 2.35 million packets dropped.)
Thanks for making me look, looks like my fail2ban isn't working!
@Edent @dracos This may be worth researching: https://github.com/TecharoHQ/anubis
@retrofan64 how do you use it on PHP?
@Edent I use the Wordfence plugin, which has a built-in firewall. Sometimes I use that to block problematic IPs. But obviously it’s better if you can stop them before they get to that point. Interested to hear what other suggestions you get…
Install bonfire.cafe
Get the full app experience
