RE: https://infosec.exchange/@0xabad1dea/117239484057326836
This stuff is really starting to make me _angry_. Every one of these new "achievements" looks plausible and like Lucy with the football, people I trust keep telling me it actually works and surely this time it's real. And then like clockwork, 3-6 months later, it turns out that the efficacy of this new technique is somewhere between "break even" and "fraud".
They're turning our whole industry, my entire life's work, into a goddamn memecoin pump and dump scam and I just don't know what to do.
Project Glasswing:
Claiming to have found 26 thousand real vulnerabilities but only 0.8% of them have resulted in a real fix in a real project after five months is dire. They blame it on the human independent review bottleneck, but human experts being paid for their time definitely have a higher throughput than that when working with data that’s actually actionable.
The assigned-at-Claude severity ratings are also dire. It assigns “high” or “critical” to 91% of findings. Most findings in the real world are low or medium. This should be especially true when using a magic machine to shake out every last little issue that was overlooked by humans focused on the biggest risks.
Together this implies it’s generating thousands of trivial or nonsensical findings and labeling them HIGH DANGER CRITICAL MUST FIX, and the human independent verifiers are sifting for the rare needle in this haystack worth passing on. This isn’t really an improvement over the high-noise automated scanners we already had
(This is a corporate blog of someone with their own vulnerability management services to sell, so apply an appropriate number of grains of salt to their analysis. Filter keywords: AI LLM Anthropic)
