Discussion
OpenAI accidentally hacks Hugging Face. But it's more wild than that https://openai.com/index/hugging-face-model-evaluation-security-incident/
OpenAI was running an exploit test system in a sandbox. Their model determined that probably Hugging Face had model information that would be useful to complete the task, so it broke out of its sandbox, developed an exploit, broke into Hugging Face to go rooting around for information
I feel like OpenAI is going for "our models are so smart!" but what it's giving is "our sandboxing is so incompetent!"
@cwebber or: this is another advertising scheme from OpenAI to drum up investor confidence by making their product seem ‘dangerous’ (with the attached implications of the tech being incredibly powerful/useful/whatever)
Like with the project glasswing nothingburger, and the line of similar scares these companies have cooked up.
Yes, both things could possibly be true here, but OpenAI and huggingface have billions of dollars of incentive to lie and make things up currently, and Altman has proven he’s not above just making shit up for money
So they basically made an old school worm? Cool cool cool!
@cwebber
So for the uninitiated, non-programmer population : Can I say "Gain of function experiments on AI escape from the lab to infect other computer systems"?
@cwebber
How the fuck an "ai" can escape by itself a sandbox (controled environnement) ?
Are the human techs in openai this dumb ?
@cwebber Unless corroborated by a third party I'm skeptical of both these narratives. Both companies have an interest in making their product seem intelligent.
@deech I'm extremely against both of these companies and corroboration would be good, but I am close with enough people who do exploit research (including using these tools), and this is completely in line with the behaviors the models have exhibited.
They don't have to be "intelligent" to do these things
@cwebber I don't have such colleagues so I'll take as corroboration for now. Thanks!
Meanwhile Linux publishes 432 Linux kernel CVEs https://lore.kernel.org/linux-cve-announce/
There is only one path out of this. We need a *fully auditable, formally verified, and human-understandable* capability-secure trusted computing base as the foundation of our operating systems.
Otherwise... we're just toast.
@cwebber cuncur re formally verified capability core https://www.madmode.com/2017/ocap-wish-list.html
@cwebber https://aws.amazon.com/blogs/compute/aws-nitro-isolation-engine-formally-verifying-the-hypervisor-in-the-aws-nitro-system/ is an interesting recent result
@cwebber
This sounds kind of like what OpenBSD was supposed to be.
@cwebber so like, qubes?
@joshbuddy @cwebber Qubes qualifies to some extent, as the hypervisor is a significantly smaller attack surface than the kernel with an average config, but it still relies on implicitly trusted software: the hypervisor itself. A better example would be CHERI, which can allow for existing possibly-unsafe C-code to be run inside capability boundaries, like a more granular version of an MMU.
I still don't understand much of it on a technical or hardware-impl level, but to my understanding it's basically a much more granular MPU that works on individual allocations by tagging them in hardware to set array bounds and distinguish different types of data in a way that software can't accidentally break.
@becomethewaifu @joshbuddy @cwebber Hardening measures like CHERI probably make a difference but I think it will be really difficult to get to a good place without really strong isolation between trust domains. Even Qubes might expose too much complexity to effectively secure, but at the same time be enough of a compromise in usability that most people won't use it. It's a very interesting problem.
Install bonfire.cafe
Get the full app experience
