Discussion
Loading...

Post

  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
Michał "rysiek" Woźniak · 🇺🇦
@rysiek@mstdn.social  ·  activity timestamp 2 days ago

> In this specific instance, the Bot Management system has a limit on the number of machine learning features that can be used at runtime.
> (…)
> When the bad file with more than 200 features was propagated to our servers, this limit was hit — resulting in the system panicking.
https://blog.cloudflare.com/18-november-2025-outage/

So it's "AI" when it's good for business, "machine learning" when it's bad for business, gotcha! 🤡

#CloudFlare

The Cloudflare Blog

Cloudflare outage on November 18, 2025

Cloudflare suffered a service outage on November 18, 2025. The outage was triggered by a bug in generation logic for a Bot Management feature file causing many Cloudflare services to be affected.
  • Copy link
  • Flag this post
  • Block
Urzl
@gooba42@mastodon.social replied  ·  activity timestamp 2 days ago

@rysiek So they don't verify the payload on receipt and this is going to happen again, on purpose next time, now that the vulnerability is revealed.

  • Copy link
  • Flag this comment
  • Block
Michał "rysiek" Woźniak · 🇺🇦
@rysiek@mstdn.social replied  ·  activity timestamp 2 days ago

@gooba42 there was no payload. The data was pulled from their own database. They changed some permissions and unexpectedly that led to duplicated data being pulled from that database, hitting certain limits.

  • Copy link
  • Flag this comment
  • Block
Urzl
@gooba42@mastodon.social replied  ·  activity timestamp 2 days ago

@rysiek That feature file was the payload in question.

They limited what they could send but not what they could receive which left them vulnerable to this disruption when the send limit was exceeded.

  • Copy link
  • Flag this comment
  • Block
Tariq
@rzeta0@mathstodon.xyz replied  ·  activity timestamp 2 days ago

@rysiek

I'm no expert and I'm no professional developer ...

but does this mean they "pushed direct to prod" without going via test?

or is that now old school, and rapid fail / rapid fix is now part of the continuous deployment ideology?

  • Copy link
  • Flag this comment
  • Block
Michał "rysiek" Woźniak · 🇺🇦
@rysiek@mstdn.social replied  ·  activity timestamp 2 days ago

@rzeta0 they pushed to prod (the database permissions change) without fully understanding the consequences of that change. Automation did the rest.

  • Copy link
  • Flag this comment
  • Block
MxFraud
@mxfraud@tabletop.social replied  ·  activity timestamp 2 days ago

@rysiek I probably got my P45 prepared for sharing this on the work slack, but it felt great :)

  • Copy link
  • Flag this comment
  • Block
Michał "rysiek" Woźniak · 🇺🇦
@rysiek@mstdn.social replied  ·  activity timestamp 2 days ago

Literally called it a couple of days ago when quoted in The Guardian article about Anthropic's "AI-driven attack" bullshit:
https://mstdn.social/@rysiek/115566158042786232

blobcatcoffee

  • Copy link
  • Flag this comment
  • Block
Brad Macpherson
@brad@1040ste.net replied  ·  activity timestamp 2 days ago

@rysiek Can't be tainting the "AI" brand, now! Someone might think the hype is .. just hype! 😉

  • Copy link
  • Flag this comment
  • Block
Log in

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.0 no JS en
Automatic federation enabled
  • Explore
  • About
  • Members
  • Code of Conduct
Home
Login