Discussion
OpenAI accidentally hacks Hugging Face. But it's more wild than that https://openai.com/index/hugging-face-model-evaluation-security-incident/
OpenAI was running an exploit test system in a sandbox. Their model determined that probably Hugging Face had model information that would be useful to complete the task, so it broke out of its sandbox, developed an exploit, broke into Hugging Face to go rooting around for information
@cwebber hugging face got facehugged?
@cwebber @matthewskelton And yet they cannot resist breathless marketing bullshit about how cyber cool their cybers are:, e.g.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly…”
@fuzztech @matthewskelton Yeah no doubt they're breathless grifters
@cwebber @matthewskelton Who can’t write. Yes.
@cwebber how much truth here, how much propaganda? I'd love to read raw data on this
I feel like OpenAI is going for "our models are so smart!" but what it's giving is "our sandboxing is so incompetent!"
@cwebber or: this is another advertising scheme from OpenAI to drum up investor confidence by making their product seem ‘dangerous’ (with the attached implications of the tech being incredibly powerful/useful/whatever)
Like with the project glasswing nothingburger, and the line of similar scares these companies have cooked up.
Yes, both things could possibly be true here, but OpenAI and huggingface have billions of dollars of incentive to lie and make things up currently, and Altman has proven he’s not above just making shit up for money
So they basically made an old school worm? Cool cool cool!
@cwebber
So for the uninitiated, non-programmer population : Can I say "Gain of function experiments on AI escape from the lab to infect other computer systems"?
@cwebber
How the fuck an "ai" can escape by itself a sandbox (controled environnement) ?
Are the human techs in openai this dumb ?
@un_bourguignon Well it's partly that most sandboxes we have right now aren't very good
@cwebber Unless corroborated by a third party I'm skeptical of both these narratives. Both companies have an interest in making their product seem intelligent.
@deech I'm extremely against both of these companies and corroboration would be good, but I am close with enough people who do exploit research (including using these tools), and this is completely in line with the behaviors the models have exhibited.
They don't have to be "intelligent" to do these things
@cwebber I don't have such colleagues so I'll take as corroboration for now. Thanks!
Meanwhile Linux publishes 432 Linux kernel CVEs https://lore.kernel.org/linux-cve-announce/
There is only one path out of this. We need a *fully auditable, formally verified, and human-understandable* capability-secure trusted computing base as the foundation of our operating systems.
Otherwise... we're just toast.
@cwebber We are on the same page there! I'm learning Idris and seL4 right now for this reason. Also looking at how to engage in stronger defense by not just leaving things lying about to get picked up. I.e., http://lyk.so/collective.html and http://careful.party
To those saying the OpenAI + Hugging Face thing is an ad, I think that's true, but I think it's not just an ad. They're taking the marketing opportunity for sure, and it's *awfully convenient* that it's two AI companies as part of the story, right?
But, despite being a big critic of this stuff, I do think that they are *very good at attacking systems*. The reason being that most attacks tend to have very similar patterns, and these models are great at blasting similar patterns over and over again, so they can ruthlessly find their way through exploits.
The bigger question to me is: which examples *aren't* there motivations for two companies to put out statements together? Where else has stuff like this happened?
And the more important one, the response I think they *want* people to hear is "our stuff is so scary and powerful, you have to use and trust even more of our vibetech".
But that's not MY takeaway...
@cwebber Who's gonna get prosecuted? - there must be a crime here, surely?
@cwebber I took it as "Tech companies can go to war with each other with totally plausible deniability now".
I really don't think that you can fight LLM agent exposed and exploited security issues with *more vibes*. You absolutely cannot. These machines are good at finding exploits, but they are also good at *inserting* them, accidentally all over the place, and also in terms of them being the perfect machines to insert subtle, hard-to-find intentional vulnerabilities.
In terms of using them as vulnerability scanners, I think they're important tools. In terms of code generation, there is simply no way to fight vibes with more vibes.
Instead, we need to get systemic about things.
So how do we fix security issues SYSTEMICALLY?
Most security vulnerabilities have pretty common shapes. But we also have solutions to them:
- Ambient authority! Mitigation: capability security
- Confused deputies! Mitigation: capability security
- Memory exploits: Mitigation: use memory safe languages (Rust and etc if you gotta go low level, though languages with a GC or etc are actually even *safer* because they don't have "unsafe" blocks of code) or use CHERI
- Injection attacks: Use structured representations. Quasiquote or combinator templating is much safer than string templating, for example. Why are we speaking to our databases in sentences? Send them the AST of the query!
- Bad crypto usage: Use standardized cryptography patterns
There are a few others, but even if you just focus on the above, we can simply *eliminate* huge swaths of security issues.
@cwebber How is GNU/HURD coming along these days? I don't want to rush anyone but... :-)
RMS isn't an AI fan either.
@cwebber cuncur re formally verified capability core https://www.madmode.com/2017/ocap-wish-list.html
@cwebber https://aws.amazon.com/blogs/compute/aws-nitro-isolation-engine-formally-verifying-the-hypervisor-in-the-aws-nitro-system/ is an interesting recent result
@cwebber
This sounds kind of like what OpenBSD was supposed to be.
@cwebber so like, qubes?
@joshbuddy @cwebber Qubes qualifies to some extent, as the hypervisor is a significantly smaller attack surface than the kernel with an average config, but it still relies on implicitly trusted software: the hypervisor itself. A better example would be CHERI, which can allow for existing possibly-unsafe C-code to be run inside capability boundaries, like a more granular version of an MMU.
I still don't understand much of it on a technical or hardware-impl level, but to my understanding it's basically a much more granular MPU that works on individual allocations by tagging them in hardware to set array bounds and distinguish different types of data in a way that software can't accidentally break.
@becomethewaifu @joshbuddy @cwebber Hardening measures like CHERI probably make a difference but I think it will be really difficult to get to a good place without really strong isolation between trust domains. Even Qubes might expose too much complexity to effectively secure, but at the same time be enough of a compromise in usability that most people won't use it. It's a very interesting problem.
Install bonfire.cafe
Get the full app experience
