Discussion
@sammy @junes @oblomov that's correct, we indicate this using both robots.txt and widely-used robots http headers, so even crawlers who can't be bothered fetching and parsing robots.txt can check robots http headers instead and throw away the stats if the http headers indicate they shouldn't be crawled
@dumpsterqueer @sammy @oblomov
what's the point of relying on a robots.txt, which has a very downwind syntax and can be circumvented by anyone who wants to do so, or its http headers, for which the same applies, when you can just develop an option to simply not expose nodeinfo, other than flaunt numbers you don't have?
@junes @sammy @oblomov the option to just return 0 stats also exists. If a crawler can't parse and handle robots.txt then they can check http headers, as I mentioned above. If the crawler implementation can't even be bothered doing that then... :smh: The whole point of the baffled stats option is to return accurate stats to crawlers who respect consent, and to sting crawlers who don't.
@junes @dumpsterqueer @sammy poisoning the well for bots that are not well-behaved?
@oblomov @dumpsterqueer @sammy
if you tell "i have big numbers" when you don't, while you could just not tell any number, you are stupid.
@dumpsterqueer @oblomov @sammy
the option to tell numbers bigger than they actually are should not exist. life itself would be greatful.
@oblomov @dumpsterqueer @sammy
anyway, life itself laughs of all of this.
Install bonfire.cafe
Get the full app experience
as I said elsewhere in the thread, the option to return 0 for stats also exists in gotosocial. And again, only crawlers that ignore "do not crawl" requests get served bad stats. You can say "you are stupid" all you like, but it won't improve the behavior of your crawler.