Discussion
OpenAI claims that their own experimental model broke containment and caused the Hugging Face incident.
If this is true, the following is also true: they were working with weapons-grade offensive models, or so they hoped, and yet had insufficient telemetry from the lab, and couldn't be bothered with an airgap.
Hard to take that seriously.
https://openai.com/index/hugging-face-model-evaluation-security-incident/
@mttaggart Every one of these incidents reads like a scripted commercial. "Oh, no, we created a horrible monster, so horrible that we can't give you the details of how it all went down. But trust us, monsters were created, so terrible and bad, and you can rent access to that monster..."
@mttaggart now imagine this was an employee and a physical warehouse.
with reduced cyber refusals for evaluation purposes
Ah, so they pulled down the safeguards.
@Laukidh I mean model guardrails are one thing. If they were this concerned about their model, there should have been physical network gaps to prevent exactly this eventuality. I gap my malware labs; why don't they?
@mttaggart I knew a guy who worked for a big AV company (I think Norton) and as I recall, if you walked into their test lab with your cell phone, you didn’t get to walk back out with it.
Call me naive if you want, but I really don't think this is marketing. And if I'm wrong, it's not a better look for OpenAI. In the first case, they essentially discharged a weapon without proper safety controls. In the second, they're making up a story about doing the same to sound scary and cool.
Either way, ghoulish as hell.
You can absolutely make the case for this being scripted, and working backwards from the facts, there is a lot of convenience for OpenAI. The model's supposed power, the fact that the "other models" got in the way of response because of guardrails, etc.
But either there was collusion with the victim, or an assumption that the victim would not respond with a charge of negligence or something else. Big gamble, and quite the conspiracy.
I tend to believe in the cutting power of Hanlon's Razor.
@mttaggart They are openly admitting negligence. Must have gotten too big to contain the information so posting publicly was better than the inevitable leak of same.
@mttaggart or they did it on purpose but their dumb ai failed to hide it's track so they claim the "bug", because they always claim it's a bug, even for intended action #microsoft-backdoor
@mttaggart their marketing is getting sloppy
@mttaggart gonna start selling artisanally spraypainted cardboard boxes labeled "doomsday device" and achieve the same material effect
@mttaggart I just finished reading their blog post and they really don't seem to be taking this seriously. Their key take-away appears to be that the "model alignment" was insufficient, when it's been proven fairly rigorously that trained-in model alignment isn't a strong deterministic control.
@mttaggart yeah i took a quick look and my initial reaction was "clever marketing"
@mttaggart or, it's a marketing stunt
@ciaranmak Maybe, but this is not the same as the Mythos hype. There was a victim here, unless Hugging Face was in on it.
@mttaggart I don't point my pentest automations at websites unless there is intent, are they somehow above the law and are random automated attacks totally fine now. There's an arrogance in the act imo
Install bonfire.cafe
Get the full app experience
