Discussion
Loading...
@kfitz
Kathleen, Cloudflare offers a free technology specifically to prevent LLM scraping.
https://blog.cloudflare.com/cloudflare-ai-audit-control-ai-content-crawlers/
It was designed specifically to help with use cases like yours. I am a Sr. Solutions Engineer and can help set you up if you like.
@suzannealdrich @hello Hi, and thanks! We're already behind Cloudflare but still getting clobbered. I'll pass your offer to our dev team, though -- fine-tuning may help!
@kfitz @hello sadly, this comes from the top of the pyramid (as bad as the metaphor is). KC is based on Wordpress, and Wordpress leadership now actively promote AI as a foundational tool for the future of the web bla bla bla 😰 https://wordpress.org/news/2025/12/sotw-2025/
@kfitz @hello I hear you on this. It's been a total nightmare for Reclaim as well, what's worse is that blocking them is a joke, no one honors it. We had a site go from 1-2 million hits on avg per month to 7-8 million hits on avg basically overnight and it's all bots. Makes running a site with significant content and history exponentially more resource intensive.
@kfitz @jimgroom @hello yep. The #PleiadesGazetteer has gone from being perpetually in the #mapbox free tier to owing them more than $500/month because of javascript-capable scrapers that request all tiles available to a map. For the first time in ~20 years we may have to pull a basic, read-only user capability behind a login wall. Non-trivial for us and for our users, but this is unsustainable.
@paregorios @jimgroom @hello Yikes. That’s horrible. I’m so sorry. And increasingly infuriated.