It's been a few days since I posted about https://readily.news AKA "open.news", a service which:

1. asks for complete access to your Mastodon/fedi account

2. ingests whatever your account can see via your account and summarizes it using LLMs (seemingly from OpenAI?)

3. sends you a daily, personalized newsletter

It's a particularly bad kind of scraper because it basically hijacks existing community infra to do the scraping for it.

Because accounts' host instances are the actors gathering up all the content there's no way for remote servers to detect which of their followers' accounts have been compromised, nor to block their posts from ending up in the hands of the upstream LLM providers.

We'll probably need admins of affected instances to run a database query to detect and revoke permissions granted to this service via OAuth to limit its access.

I asked the guy who
the guy who appears to be behind it (https://mastodon.social/@librenews
) if he could confirm his affiliation, but he doesn't actually seem to be very active on Mastodon (preferring Bluesky) and so he still hasn't responded.

I'm actually a little surprised at how little reaction there's been to this based on how quickly other scrapers were run off the network, but I get that people are busy.

If you want more details, the specifics of my investigation are in this post:

https://cryptography.dog/blog/what-little-i-know-about-readily-news/

...and I'd appreciate if others could corroborate my findings.

#infosec #fediscrapers #scrapers #LLMs #AI

Guarding My Git Forge Against AI Scrapers

https://vulpinecitrus.info/blog/guarding-git-forge-ai-scrapers/

#HackerNews #GuardingMyGitForge #AI #Scrapers #Cybersecurity #OpenSource #DeveloperCommunity

I just published a blog post summing up my most pertinent thoughts about dealing with badly-behaved web-scraping bots:

https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/

It isn't exactly a Hallowe'en-themed article, but today is the 31st and the topic is concerned with pranking people who come knocking on my website's ports, so it's somewhat appropriate.

#infosec #bots #halloween #scrapers #AI #someMoreHashtagsHere

It's been a few days since I posted about https://readily.news AKA "open.news", a service which:

1. asks for complete access to your Mastodon/fedi account

2. ingests whatever your account can see via your account and summarizes it using LLMs (seemingly from OpenAI?)

3. sends you a daily, personalized newsletter

It's a particularly bad kind of scraper because it basically hijacks existing community infra to do the scraping for it.

Because accounts' host instances are the actors gathering up all the content there's no way for remote servers to detect which of their followers' accounts have been compromised, nor to block their posts from ending up in the hands of the upstream LLM providers.

We'll probably need admins of affected instances to run a database query to detect and revoke permissions granted to this service via OAuth to limit its access.

I asked the guy who
the guy who appears to be behind it (https://mastodon.social/@librenews
) if he could confirm his affiliation, but he doesn't actually seem to be very active on Mastodon (preferring Bluesky) and so he still hasn't responded.

I'm actually a little surprised at how little reaction there's been to this based on how quickly other scrapers were run off the network, but I get that people are busy.

If you want more details, the specifics of my investigation are in this post:

https://cryptography.dog/blog/what-little-i-know-about-readily-news/

...and I'd appreciate if others could corroborate my findings.

#infosec #fediscrapers #scrapers #LLMs #AI

does anyone know who's behind "open.news"?

https://open.news/

> Open.News is the command center for the decentralized newsverse.

Looks like they're ingesting people's fediverse feeds into LLMs and feeding slop to people. I only noticed because it was mostly visiting non-existent or malformed URLs.

> We index live conversations across RSS, Bluesky, and Mastodon so you never miss the story behind the story. FeedBrainer's conversational AI transforms the firehose into a calm, contextual briefing tailored to you.

#slop #scrapers

@Codeberg my problem ain't #scrapers - otherwise I'd #SelfHost my stuff in my own home LAN - but rather #bots flooding #Issues and #PullRequests with garbage.

• Cuz I do expect #bots to scrape #FLOSS which I permissively licensed...

The problem I dread is once people start abusing their " #AI" #bullshit and #FloodTheZoneWithShit aka. " #AIslop" for no good reason.

• Kinda like @bagder had to deal with "AI" slop #SecurityReports that didn't even try to show #ProofOfConcept or actually evidence their claims in a scientifically reproduceable fashion but merely wasted lifetime of maintainers!

And @Erpel 's original issue is just that: #spam in the #IssueTracker...

does anyone know who's behind "open.news"?

https://open.news/

> Open.News is the command center for the decentralized newsverse.

Looks like they're ingesting people's fediverse feeds into LLMs and feeding slop to people. I only noticed because it was mostly visiting non-existent or malformed URLs.

> We index live conversations across RSS, Bluesky, and Mastodon so you never miss the story behind the story. FeedBrainer's conversational AI transforms the firehose into a calm, contextual briefing tailored to you.

#slop #scrapers

AI scrapers request commented scripts

https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/

#HackerNews #AI #scrapers #commented #scripts #technology #automation

I just published a blog post summing up my most pertinent thoughts about dealing with badly-behaved web-scraping bots:

https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/

It isn't exactly a Hallowe'en-themed article, but today is the 31st and the topic is concerned with pranking people who come knocking on my website's ports, so it's somewhat appropriate.

#infosec #bots #halloween #scrapers #AI #someMoreHashtagsHere

The developer of Bear Blog on the latest wave of scrapers he faced.

They've depleted all human-created writing on the internet, and are becoming increasingly ravenous for new wells of content.

[...]

I'm still speculating here, but I think [mobile] app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers.

https://herman.bearblog.dev/agressive-bots

#ai #scrapers #crawlers

The developer of Bear Blog on the latest wave of scrapers he faced.

They've depleted all human-created writing on the internet, and are becoming increasingly ravenous for new wells of content.

[...]

I'm still speculating here, but I think [mobile] app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers.

https://herman.bearblog.dev/agressive-bots

#ai #scrapers #crawlers

sysadmins/webmasters of fedi:

I am looking for suggestions of which search engine crawlers I should consider permitting in my robots.txt file.

There can definitely be value in having a site indexed by a search engine, but I would like to deliberately exclude all of those which are using the same data to train LLMs and other genAI. More specifically, I would only like to allow those which have an explicit stance against training on others data in this fashion.

Currently I reject everything other than Marginalia (https://marginalia-search.com/). Are there any others I should consider?

#AI #search #robots #crawlers #scrapers

sysadmins/webmasters of fedi:

I am looking for suggestions of which search engine crawlers I should consider permitting in my robots.txt file.

There can definitely be value in having a site indexed by a search engine, but I would like to deliberately exclude all of those which are using the same data to train LLMs and other genAI. More specifically, I would only like to allow those which have an explicit stance against training on others data in this fashion.

Currently I reject everything other than Marginalia (https://marginalia-search.com/). Are there any others I should consider?

#AI #search #robots #crawlers #scrapers