Discussion
My friends, I have spent months assessing "AI security" products, especially ones that claim to prevent jailbreaking, prompt injection, etc. Here's what I know:
It's really a sight to behold.
@badsamurai Dude if regex were really all this was, then fine. The problem is they're using other LLMs for classification of text, and trusting it.
And it turns out that power users making detailed prompts are semantically so close to "jailbreaking" that no model can tell the difference.
@mttaggart Sounds perfectly sustainable.
@mttaggart Would your rather:
1) An "AI Security" product that tries to reinvent EDR from first principles
2) An existing EDR that aggressively re-brands as AI by shoving LLMs into everything
@tehfishman 3) Money in my pocket for skilled engineers
@mttaggart It'll be a tough sell to VC/Shareholders
@mttaggart yeah, but it looks good on an audit
Y'all are reaffirming my decision to step away from my previous role as a senior security computer toucher earlier this year.
I looked forth over the landscape, and amongst other reasons, saw a number of these LLM security issues, and said "no, I don't think I wan't to be responsible for dealing with this."
The oldest, classic, attack surface is confusing data plane and control plane. Like smashing the stack with a NOP sled
For LLM, this combining is a feature
@pseudonym @dave_cochran @mav What's more, the data plane is unreliable when built as they intend. And the view, however you describe it, can sometimes just make stuff up.
@mttaggart man I'd love to have a salesperson explain how to secure AI for prompt injection.
Sounds great. Like powder water.
@khwevin The basic idea is a small language model classifying prompts in real time, and actioning any detections.
So you can imagine how that goes.
@mttaggart the only really valuable security model for this sort of problem (stopping attacks, not securing/guardrailing the agents) is one of the oldest approaches.
Positive security models rise from the grave to enforce allowlisting.
But that’s on really to manage AI automated attacks.
@thegibson I've firmly concluded that sane and aggressively isolated and monitored architecture is the only way to interact with these tools, along with rather stringent formal guidelines for their usage.
None of that has to do with asking another LLM to classify content.
@mttaggart@infosec.exchange https://role-confusion.github.io/ it's so funny how people think prompt injection is a solvable problem
@mttaggart It is wild watching folks who keep trying and failing to have any respect for expertise continue to engineer an "expertise machine."
Install bonfire.cafe
Get the full app experience
@mttaggart Even just looking at harness security, e.g. OpenCode or similar, is zany. The first thing people recommend for these harnesses is to setup tool deny lists. Which cybersecurity knows is an ineffective control. And while cybersecurity is still grasping at straws for AI governance, the industry is influencing users to run agents on their primary devices and allow the agent to run with their user’s device account and browser profile, e.g Hermes Desktop. My examples aren’t picking on those 2 specifically, there are many, many options doing the same thing. Convenience > Security. And LLM capabilities are growing quickly. They can parse the available action space even better. As LLMs get “smarter” the brittle cybersecurity practices around them get weaker.