OK here's a theory: #ChatGPT's #Atlas browser is not a really browser but fact a way for OpenAI to circumvent scrape blockers. It's more a distributed human-based scraper rather than anything else.

Given how widely loathed AI and how damaging AI scrapers have become #OpenAI's IP ranges ended up in quite a lot of block lists, many servers outright terminate any connection to them. Then there are things like #Anubis or #Iocaine that further frustrate #LLM scraping.

But what if you DIDN'T neeed to bother about all that? What if you could use civilian IP addresses with "organic" traffic patterns, and have humans solve Captchas, provide proof of work for Anubis, or get around Iocaine? All this for free -- you don't even need to pay people for it?

I would be REALLY interested to see what telemetry Atlas sends back. 100% certain it will send back things like URL and rendered HTML output, possibly user interaction patterns ("a normal human on this website moves their mouse first to the 'I am not a bot' captcha then clicks it). They do not have to respect robots.txt because, well, it comes from organic visitors...

Am I crazy?

OK here's a theory: #ChatGPT's #Atlas browser is not a really browser but fact a way for OpenAI to circumvent scrape blockers. It's more a distributed human-based scraper rather than anything else.

Given how widely loathed AI and how damaging AI scrapers have become #OpenAI's IP ranges ended up in quite a lot of block lists, many servers outright terminate any connection to them. Then there are things like #Anubis or #Iocaine that further frustrate #LLM scraping.

But what if you DIDN'T neeed to bother about all that? What if you could use civilian IP addresses with "organic" traffic patterns, and have humans solve Captchas, provide proof of work for Anubis, or get around Iocaine? All this for free -- you don't even need to pay people for it?

I would be REALLY interested to see what telemetry Atlas sends back. 100% certain it will send back things like URL and rendered HTML output, possibly user interaction patterns ("a normal human on this website moves their mouse first to the 'I am not a bot' captcha then clicks it). They do not have to respect robots.txt because, well, it comes from organic visitors...

Am I crazy?

#iocaine has been up for 11days 15h 5m 24s, and spent 3days 8h 21m 41s dealing with - gestures hands wildly - everything.

In the past 24 hours, it served 27.94M requests, 99.46% of which were garbage, 0.53% passed through unscathed, and 0.007% were fed to the Cookie Monster. This required about 79.41MiB of memory on average, and 64.72GiB of absolute trash was generated for the nastiest visitors.

Top garbage consumers were:

1. Disguised bots - 22.85M
2. Claude - 1.52M
3. Facebook - 976.18K
4. Huawei - 871.42K
5. OpenAI - 811.89K
6. Amazon - 332.62K
7. Commercial scrapers - 260.51K
8. Alibaba - 62.96K
9. Google - 27.44K
10. Enthusiastic guestbook visitors - 1.34K
11. Perplexity - 45.03

Various other agents slurped through 50.20K pages of unhinged junk, bless their little hearts.

In these trying times, 0.04% of all requests were likely of human origin: I hope you enjoyed your stay, and will visit again! Of all requests iocaine let into the garden, 68.51% were from Fediverse software. Thank you! #FediHug

#AIStatsPorn

Previously, I set a date for the #iocaine 3.0 release, October 23rd, that's about 1.5 weeks. I may have to push it back a little again, so I can write the necessary documentation.

Already started on it, but a lot has to be written still, and even more reviewed - and this endeavour includes a redesign of the iocaine homepage, so that I can host docs both for the stable and the in-development branch too.

#iocaine has been up for 15h 5m 29s, and spent 3h 46m 11s dealing with - gestures hands wildly - everything.

In the past 24 hours, it served 28.00M requests, 99.44% of which were garbage, 0.54% passed through unscathed, and 0.003% were fed to the Cookie Monster. This required about 139.48MiB of memory on average, and 65.90GiB of absolute trash was generated for the nastiest visitors.

Top garbage consumers were:

1. Disguised bots - 22.77M
2. Claude - 2.55M
3. OpenAI - 810.08K
4. Facebook - 699.76K
5. Huawei - 340.27K
6. Amazon - 333.69K
7. Commercial scrapers - 237.63K
8. Alibaba - 67.17K
9. Google - 6.96K
10. Enthusiastic guestbook visitors - 2.12K
11. Perplexity - 3.00

Various other agents slurped through 8.92K pages of unhinged junk, bless their little hearts.

In these trying times, 0.05% of all requests were likely of human origin: I hope you enjoyed your stay, and will visit again! Of all requests iocaine let into the garden, 62.13% were from Fediverse software. Thank you! #FediHug

#AIStatsPorn

A scraper is getting past my #iocaine install and hitting forgejo, (several others are getting put into the maze) how do I go about trying to improve my filter?

I will be cutting an #iocaine 2.5.1 release this coming weekend, to address a few issues & footguns.

All of the issues it will address have workarounds, but a fix is better than a workaround, and as 3.0 is still a while away, and is going to be a disruptive change, another 2.x release is necessary.

In completely unrelated news, I have a half-finished branch that can render frames from Doom. That branch will never be part of iocaine, but if I end up making a plugin system, then it will be a supported plugin.

Why? "Because a game engine is anything that can execute Lua scripts" (context), and iocaine can execute Lua scripts, and as such, is a game engine, and a test of a game engine is to see if it can be used to run Doom. iocaine can run¹ Doom.

¹: For some limited values of "run".

#iocaine