Discussion
Loading...

Post

Log in
  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
David Gerard
David Gerard
@davidgerard@circumstances.run  ·  activity timestamp last week

@froztbyte @osm_tech yeah i'm getting the same AI assholes

as is @RationalWiki (i'm the sysadmin trying to keep the site up in the face of the hammering - we can either lose Google search listing, or we can be literally unusable for humans)

as is @corbet at Linux Weekly News - OSM might be relevant to LWN, a free content project getting hammered by the AI bots

they botnet suburban Android boxes

covered it a bit previously on Pivot:

https://pivot-to-ai.com/2025/06/02/fighting-the-ai-scraper-bots-at-pivot-to-ai-and-rationalwiki/
https://pivot-to-ai.com/2025/09/07/the-ai-scraper-bots-are-hammering-pivot-to-ai-again-please-test/

Pivot to AI

The AI scraper bots are hammering Pivot to AI again — please test

Pivot to AI lives on a small and cheap virtual server at Hetzner (who are great). The site’s been slowing badly of late. If you guessed it’s our AI bot friends, you are correct! I put some fi…
Pivot to AI

Fighting the AI scraper bots at Pivot to AI and RationalWiki

We’ve covered the AI scraper bots before. These just hit web pages over and over, at high speed, to scrape new training data for LLMs. They’re an absolute plague across the whole World Wide Web and…
  • Copy link
  • Flag this post
  • Block
JP
JP
@froztbyte@mastodon.social replied  ·  activity timestamp last week

@osm_tech might be a thing @davidgerard could do on pivot

  • Copy link
  • Flag this comment
  • Block
David Gerard
David Gerard
@davidgerard@circumstances.run replied  ·  activity timestamp last week

@froztbyte @osm_tech yeah i'm getting the same AI assholes

as is @RationalWiki (i'm the sysadmin trying to keep the site up in the face of the hammering - we can either lose Google search listing, or we can be literally unusable for humans)

as is @corbet at Linux Weekly News - OSM might be relevant to LWN, a free content project getting hammered by the AI bots

they botnet suburban Android boxes

covered it a bit previously on Pivot:

https://pivot-to-ai.com/2025/06/02/fighting-the-ai-scraper-bots-at-pivot-to-ai-and-rationalwiki/
https://pivot-to-ai.com/2025/09/07/the-ai-scraper-bots-are-hammering-pivot-to-ai-again-please-test/

Pivot to AI

The AI scraper bots are hammering Pivot to AI again — please test

Pivot to AI lives on a small and cheap virtual server at Hetzner (who are great). The site’s been slowing badly of late. If you guessed it’s our AI bot friends, you are correct! I put some fi…
Pivot to AI

Fighting the AI scraper bots at Pivot to AI and RationalWiki

We’ve covered the AI scraper bots before. These just hit web pages over and over, at high speed, to scrape new training data for LLMs. They’re an absolute plague across the whole World Wide Web and…
  • Copy link
  • Flag this comment
  • Block
The Orange Theme
The Orange Theme
@theorangetheme@en.osm.town replied  ·  activity timestamp last week

@davidgerard @froztbyte @osm_tech @RationalWiki @corbet An aside, but I had no idea you keep Rational Wiki running! I love that site. Thank you for all your hard work! I'm sorry the slopbros are trying to ruin it.

  • Copy link
  • Flag this comment
  • Block
David Gerard
David Gerard
@davidgerard@circumstances.run replied  ·  activity timestamp last week

@theorangetheme @froztbyte @osm_tech @RationalWiki @corbet i quit the sysadmin job nine years ago, so of course i still have it

  • Copy link
  • Flag this comment
  • Block
JP
JP
@froztbyte@mastodon.social replied  ·  activity timestamp last week

@davidgerard @osm_tech @RationalWiki @corbet Also getting and handling them (as you know), but I’d be pretty interested to hear how bigger projects have to handle them

Quick check on latest status since last #iocaine restart: 1.49TB across 1.05B requests served

they never ever stop…

  • Copy link
  • Flag this comment
  • Block
Dan Goodin
Dan Goodin
@dangoodin@infosec.exchange replied  ·  activity timestamp 2 weeks ago

@osm_tech

Please contact me on Signal: DanArs.82

  • Copy link
  • Flag this comment
  • Block
sjvn
sjvn
@sjvn@mastodon.social replied  ·  activity timestamp 2 weeks ago

@osm_tech Tell me more. You can reach me at sjvn01 <at> gmail.com

  • Copy link
  • Flag this comment
  • Block
David Gerard
David Gerard
@davidgerard@circumstances.run replied  ·  activity timestamp last week

@sjvn @osm_tech do contact sjvn!

  • Copy link
  • Flag this comment
  • Block
Baloo Uriza
Baloo Uriza
@BalooUriza@social.tulsa.ok.us replied  ·  activity timestamp 2 weeks ago

@osm_tech I wonder if there's a way to fail2ban requests coming in faster than typically found in human requests.

  • Copy link
  • Flag this comment
  • Block
OpenStreetMap Ops Team
OpenStreetMap Ops Team
@osm_tech@en.osm.town replied  ·  activity timestamp 2 weeks ago

@BalooUriza We use fail2ban to handle some of this with custom rules, but eventually fail2ban becomes a bottleneck after 100,000 IP addresses.

  • Copy link
  • Flag this comment
  • Block
Cassandrich
Cassandrich
@dalias@hachyderm.io replied  ·  activity timestamp 2 weeks ago

@osm_tech @BalooUriza For IPv4, a bitmask of the entire address space is a viable "efficient" implementation of blocking. I wonder if there are tools that can do it that way rather than needing a gigantic list.

  • Copy link
  • Flag this comment
  • Block
InsertUser
InsertUser
@InsertUser@en.osm.town replied  ·  activity timestamp 2 weeks ago

@osm_tech The proxy SDK providers need to be treated like the DDOS providers they are and prosecuted.

  • Copy link
  • Flag this comment
  • Block
Floris
Floris
@floris@freiburg.social replied  ·  activity timestamp 2 weeks ago

Vielleicht ist das ein Thema für die @lagedernation?

  • Copy link
  • Flag this comment
  • Block
Lage der Nation
Lage der Nation
@lagedernation@chaos.social replied  ·  activity timestamp last week

@floris
@osm_tech Hi, please get in touch, we've covered OSM many times before and would love to learn more: team a lagedernation org

  • Copy link
  • Flag this comment
  • Block

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.2-alpha.7 no JS en
Automatic federation enabled
Log in
  • Explore
  • About
  • Members
  • Code of Conduct