I'm encountering more and more websites - mostly businesses - that block access from countries that aren't the one they're based on. This makes perfect sense given the scourge of AI crawlers. If your customers are in just one country why would you accept a massive amount of traffic from elsewhere? It also shows the increasing Balkanization of the internet, a virtual space that grows smaller and more isolated every day.
6
Replies
0
Boosts
@gabrielesvelto I sometimes just block all of Russia and China, mostly for AI-bot control, but also just the good old-fashioned dumb bots that go looking for exploits or forms to spam. It's a sledgehammer, but also the best tool I have sometimes with a budget of $0.
@gabrielesvelto I can understand the AI crawler incentive. But this is a bad business decision. People visit websites from other countries for plenty of reasons: temporarily out of country for business or holidays, acting on someone else's behalf, preparing a visit in the country, etc.
Allowing malicious crawlers to steal content and use local resources, without permission, is a very bad business and security position.
@simonzerafa @anthony it's a bit of both really. Just today I was looking for some stuff for a friend to buy and being barred from a page prevented me from doing it, that's the downside for the business. The upside is that crawlers can literally make your web page functionally offline, effectively closing it down for any customer. Barring access from abroad might limit your market a bit but if it keeps your regular customers around then that's a reasonable compromise.
@gabrielesvelto @simonzerafa I don't see how doing country based filtering helps with AI crawlers. A lot of them are using botnets (also called "residential proxy") to get around any IP based filtering.
There are other techniques to block AI crawlers that are less disruptive for your audience.
@anthony @simonzerafa residential proxies are scattered around the world. If your audience comes largely from one country it's very effective to filter traffic from other countries given the vast majority of it will be bots. Heavy levels of proof-of-work filtering are very disruptive to users but seem to have become the norm if you want to keep the bots out:
https://people.kernel.org/monsieuricon/creepy-crawlies
I can see why people would chose a simpler solution that doesn't really affect their audience much.
