sysadmins/webmasters of fedi:

I am looking for suggestions of which search engine crawlers I should consider permitting in my robots.txt file.

There can definitely be value in having a site indexed by a search engine, but I would like to deliberately exclude all of those which are using the same data to train LLMs and other genAI. More specifically, I would only like to allow those which have an explicit stance against training on others data in this fashion.

Currently I reject everything other than Marginalia (https://marginalia-search.com/). Are there any others I should consider?

#AI #search #robots #crawlers #scrapers

sysadmins/webmasters of fedi:

I am looking for suggestions of which search engine crawlers I should consider permitting in my robots.txt file.

There can definitely be value in having a site indexed by a search engine, but I would like to deliberately exclude all of those which are using the same data to train LLMs and other genAI. More specifically, I would only like to allow those which have an explicit stance against training on others data in this fashion.

Currently I reject everything other than Marginalia (https://marginalia-search.com/). Are there any others I should consider?

#AI #search #robots #crawlers #scrapers

Scraping for AI training may or may not be legal. But the effort crawlers put into evading detection and blocking is a smoking gun, an admission this scraping is not fair.

https://arstechnica.com/tech-policy/2025/08/ai-industry-horrified-to-face-largest-copyright-class-action-ever-certified

#AIscraping #scrapers #ai

Website owner? Not keen on the Mellowtel browser library building a botnet of untraceable scrapers from unwitting users who are using a browser plugin that contains Mellowtel? I've raised a GitHub issue for them to explain how much contempt they have for our consent. Join in, politely, make them look like the jerks they are. https://github.com/mellowtel-inc/mellowtel-js/issues/41

#Mellowtel #botnet #scrapers

Website owner? Not keen on the Mellowtel browser library building a botnet of untraceable scrapers from unwitting users who are using a browser plugin that contains Mellowtel? I've raised a GitHub issue for them to explain how much contempt they have for our consent. Join in, politely, make them look like the jerks they are. https://github.com/mellowtel-inc/mellowtel-js/issues/41

#Mellowtel #botnet #scrapers