I've had it with the aggressive #AI #crawlers now. Some bot has been hitting #MacPorts with a legitimate enough user agent that I can't block it without also blocking users.

Yesterday, it sent 377k requests (62 % of the total), 369k to URLs forbidden in robots.txt from 274k unique IPs. Most of it for content that could be analyzed quicker using svn checkout or git clone.

Dynamic content on the #web is broken. There's just no way to do that anymore. What a waste of energy.

I've had it with the aggressive #AI #crawlers now. Some bot has been hitting #MacPorts with a legitimate enough user agent that I can't block it without also blocking users.

Yesterday, it sent 377k requests (62 % of the total), 369k to URLs forbidden in robots.txt from 274k unique IPs. Most of it for content that could be analyzed quicker using svn checkout or git clone.

Dynamic content on the #web is broken. There's just no way to do that anymore. What a waste of energy.

If your crawler is spoofing its identity because I have disallowed it from my sites, I consider that an attempted server breach. You are no better than someone trying to compromise my SSH connection.

#IT#SysAdmin#Perplexity#Crawlers

If your crawler is spoofing its identity because I have disallowed it from my sites, I consider that an attempted server breach. You are no better than someone trying to compromise my SSH connection.

#IT#SysAdmin#Perplexity#Crawlers

🖥️ My ultra-budget server powering http://websysctl.alfonsosiciliano.net has been running smoothly for the past 2 months. So far, so good!

📈 #Crawlers hit tens of thousands of sysctl parameter pages daily. That's fine, since robots.txt allows it. But why keep requesting non-existent pages as if the site were built with WordPress 😤 ? Fortunately, the stack (#FreeBSD + #OpenResty 🌐 + #Lapis ✏️ + a custom-built #database 📦 ) stays well within the limited resources of my $5/month cloud server.

The code might soon be #OpenSource stay tuned!

#UNIX #sysctl #WebDev #WebServer #ThePowerToServe #coding #Lua #kernel