RE: https://dair-community.social/@timnitGebru/117209766801690706
Even security engineers often fall into the trap of thinking there's a way to "find all the bugs." We can be fooled into thinking there's a methodology or a tool by which we can simply enumerate the problems. We build whole ontologies around vulnerabilities thinking that there must be some way, if we just know all the classes of issues, to find every instance.
But static analysis already fulfilled part of that promise, only to reveal an entire other layer. Sure, we can look at the AST and find every syntactic vulnerability but there's a lot of noise like from unreachable code paths. Ok, we can SAT solve to find reachable code but that can collapse at high complexity. Well, we can use feedback based fuzzing, maybe with SAT assistance, to get more coverage? Great. But we still have to build the harness and go through all the findings at the end. Assuming we know all the compile options, the actual configuration, and any additional controls then we should be able to filter all that stuff down to usable bugs. Great, maybe an LLM could help build the harness (assuming it doesn't mess up in an undetectable way, like humans often do, that wastes hundreds of hours of cpu time). Maybe an LLM could help us filter down exploitable bugs.
Let's assume we can perfectly do all of that. We have all the best technology looking for known classes of vulnerabilities across a code base. We're proving them with LLMs. Let's pretend those LLMs don't need to be carefully guided to get good results. Let's pretend that problem is solved.
We still have gaps. This thinking always misses or ignores, at the very least, business logic. Find every syntactic vulnerability you want, semantics still matter.
Security is always about humans. Human experience can't simply be encoded into a stochastic process. Humans intuitively have an idea what other humans want and don't want. Good security engineers don't just find bugs. We find things that are scary for users. We find things that threaten business stability. "AI" can never replace humans for the same reason diversity is important in a security team.
Vulnerabilities only matter when people care about them, and things that are otherwise syntactically perfect *become vulnerable* when the behavior violates the expectations of the user in a way that threatens the relationship between the consumer and vendor.
There is no statistical representation of that because it comes from meat and bones. Fear lives in your lizard brain, encoded through millions of generations of evolution. Security is ultimately QA that scares people. LLMs can't feel fear because emotional experience can't be represented with statistics.
Anthropic also claims that Mythos can replace security experts. It's one thing to claim that you've built a useful tool, another to claim that you can replace experts. Some security experts have even said that it's dishonest to say your tool is superior to security experts because it found bugs in old codebases and we don't know how often people audit them for bugs and fix them. Again a misleading claim that is repeated by those outside of the company.
