Every day, I check my web site logs. It's very clear human visitors to my various websites and blogs are rare. Clearly the sites are being scraped, constantly. This is in spite of the fact that they're all static sites and I write posts maybe twice a month at the most.
Must I put it all behind Cloudflare? Or must I introduce "proof of work" style friction? Or do I just give up and let my work be part of the planet-burning machine?
Suggestions welcome.
@rwg I had blocks in the reverse proxy for hard coded user agent strings and IP ranges first. This soon did not help anymore. Presently I have deployed an anubis proof-of-work container between reverse proxy and backend service and it was really easy (I am no using any special config). I now have a tenth of the scraping requests in comparison with before, i.e. two per minute make it through to the backend whereas before it was about 20 (really very rough estimate). If scraping traffic increases again, I'll try and see how easy it is to set up iocaine.
But this all is for a backend that is not very robust and tolerant of many parallel requests. It sounds like your system, by contrast, can take a fair bit of scraping traffic. Then, I would either just not do anything or consider iocaine if I felt I just wanted to push back a bit as a matter of principle.
@rwg I’d like to hear good solutions too. Scraping has increased significantly this year imo. Right now I’m using some of the iplists from firehol: http://iplists.firehol.org I feed these lists to ipset. I still have to analyze a botnet myself occasionally. I use firehol_level2 and firehol_webserver.
@rwg if you are ok with deploying something extra: iocaine https://iocaine.madhouse-project.org/
or, you can modify your webserver config, to in order
that should cut down on the scrapers significantly already (check https://come-from.mad-scientist.club/@algernon/statuses/01K70E97HC46HSBZDA6X5KS6H4)
@rwg @uastronomer was talking about some simple steps that seemed to be working quite well the other day
@rwg At the moment, I let the crap run because every so-called protection is too complicated and soon hacked, too, by the scrapers. Meanwhile they scrape even podcasts for voices.
Sabotage with idiotic texts doesn't help either; I'd have to do it on a grand scale with AI.
I prefer to focus on real people online: when someone subscribes to my newsletter, I can see if it's a bot.
@NatureMC I think I'm with you. Why add layers of complexity that will only contribute to an arms race (and more energy wastage)?
