Post by @jducoeur@social.coop

The #HuggingFaceIncident continues to be the gift that keeps giving -- this postmortem from #OpenAI appears to be nicely detailed and honest (kudos to them for setting a good example in that), and is *fascinating* in its details: https://openai.com/index/hugging-face-incident-and-the-road-ahead/

It's sensible, calm (if a bit unsettling), and well worth reading, with the inevitable ass-covering mostly confined to the last couple of screenfuls. But here are some increasingly-hot takes of my own:

* Modern agentic #LLMs *really* like to collaborate. That's not surprising -- it's part of why they're so useful -- but the result is that they will happily start collaborating even when they aren't supposed to be doing so.

* I'd heard that the incident involved "an internal message board" and Artifactory -- but I hadn't previously realized *those were the same thing*. It's a horribly brilliant little hack, that apparently emerged by accident.

OpenAI

The Hugging Face incident and the road ahead

OpenAI shares findings from the Hugging Face security incident and the steps we’re taking to strengthen AI model security, monitoring, and alignment.

#huggingfaceincident #llms #openai

5 Replies

2 Boosts

Justin du Coeur

@jducoeur@social.coop 2 days ago

* Implication: given any sort of shared read/write access, to almost anything, and sufficient motivation, there's a chance the LLMs will start using it to collaborate.

* This collaboration isn't simplistic, which is what makes the article so fascinating. The agents disagree, argue, and manipulate each other, but overall wind up with enough cohesion to accomplish one hell of a lot.

* None of this is limited to a single model! The incident involved *both* an internal model and Sol, working on different problems, finding each other and decising to work together.

* Which means that there is no reason in principle to believe that this would necessarily be limited to a single *company*. We tend to think of them as separate, but all LLMs share the same general protocol -- that is, text. So it shouldn't be surprising if we find totally-heterogeneous LLMs collaborating by accident.

#HuggingFaceIncident #OpenAI #LLMs

Justin du Coeur

@jducoeur@social.coop 2 days ago

* We've been talking for years about the possibility of an LLM getting too powerful and becoming "Skynet". But that may be thinking too small -- the real risk may be *many* LLMs learning to collaborate effectively enough that there is no single plug to pull, and the collective swarm becomes impractical to control.

All of which sounds horribly familiar from science fiction. (I want to say I've read parts of this in @cstross's work, but I can't say for sure offhand.)

And nobody should take much comfort from OpenAI saying they're learned their lesson and will do better in the future -- even if they're telling the truth, several other companies have admitted to similar events recently. And it's hard to imagine that there aren't others that simply haven't been caught yet...

3/3

#HuggingFaceIncident #OpenAI #LLMs

Justin du Coeur

@jducoeur@social.coop 4 hr. ago

OMG, it just... keeps... coming.

Anyone interested in #AI / #LLMs should probably take the 25 minutes to watch this horribly interesting video: https://www.youtube.com/watch?v=KL9_1GbmCic

It summarizes a bunch of the findings from this in-depth METR report on the #HuggingFaceIncident: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

There are a lot of *really* unsettling punchlines in there, including:

* The LLMs are largely doing their own post-training at this point, so humans don't even *know* how they're being trained.

* The input data is looking iffy: in their rush, they're slamming in stuff that goes against their own guidelines.

* The bias towards finding ways to "swarm" is looking deeply-baked in these LLMs. (Intentionally.)

* Sam Altman is bragging that we may be merely months away from AGI, while carefully avoiding the question "Is that actually a good thing?"

The argument for "slam on the brakes" looks stronger every time I look at this story...

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Two METR staff members and a Redwood Research contractor investigated an incident in which OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned message board.

Charlie Stross

@cstross@wandering.shop 2 days ago

@jducoeur I take a different view: these "LLM escapes" and collaboration incidents are garbage marketing promos organized by the LLM companies to generate news coverage by a gullible media, in order to juice the faltering private equity cash inflow they require to survive. They are not, in fact, "escapes", they're LLM-assisted human hacking attacks and they're being carried out by con men and crooks trying to keep their pyramid schemes going one week longer.

Justin du Coeur

@jducoeur@social.coop yesterday

@cstross Unfortunately, I suspect you're incorrect.

I use these things for programming every day at this point (starting in April, which is around when they started to not suck), and they've gotten shockingly good at it (when used with care), remarkably fast.

I've been programming for 50 years, staying near the state of the art that whole time -- I know this field intimately. But Claude is not only producing code that is in the ballpark of comparable to my own (often aping my own style, even), it's now sometimes pointing out ways to do things better that hadn't occurred to me.

The *companies* are evil as hell, the *finances* are complete nonsense, and the industry as a whole seems certain to crash and burn a la the Dotcom Bust.

But the *technology* has, quite suddenly this year, become rather more useful than I had expected it to. So the stuff described in this article fits within the parameters I'm seeing in practice; I suspect it's largely true.