Updated chart showing the share of #curl vulnerabilities caused by "C mistakes" as a share of all the vulns we know existed in code over time.
2 Boosts
0 Likes
@bagder Beautiful chart! Do you also happen to have a split with the proportion of issues that were found in new (like same year) code vs older code?
@metalhead I have this chart that shows the age of every vulnerability at the time of reporting:
@bagder @metalhead see, the solution is not LLMs but time travel. A good Time Machine would have enabled the removal the bugs at conception.
@bagder looks like my c++ project at work needs to copy your CI playbook!
@bagder They are 70% as documented by Microsoft and Google. Everyone says so. Resistance is futile.💁🏻♂️
@mrunesson @icing I should hope so!
How do you determine what is a C mistake and what's not?
@ivcanet.bsky.social human assessment. It's rarely much of a debate and usually quite obvious by the flaw itself.
@bagder yeah, that graph doesn't look good for the Rust lovers, huh
nothing against them but eh
@bagder what happened in 2017 to cause that steep decline? Did you adopt some specific tool/process at the time?
Flaws listed as "C mistakes" are vulnerabilities that we deem are likely to not have happened should we have used a memory-safe language rather than C. The C mistakes are divided into the following areas: OVERFLOW, OVERREAD, DOUBLE_FREE, USE_AFTER_FREE, NULL_MISTAKE, UNINIT and BAD_FREE.
@bagder I was wondering, would you qualify one of these happening over the memory pool the same as you would if it happened over an object pool? Would it still be a "C mistake"?
@Archivist if you look at the categories there are no considerations for "pools" or "objects vs memory". I can't even figure out how we would differentiate (or why).
@bagder Well, many people differentiate, that is why they say Rust is memory-safe and C is not.
@bagder Would the big drop in 2017-2019 be explained by the generalized use of sanitizers, and the plateau after that the combination of sanitizers and test coverage? How many of these bugs remain in the code to be found by the LLMs?
@huitema yeah, the drop has no particular isolated event that explains it that I have been able to find. Just better tooling and proper engineering.
The LLMs of today find C mistakes at similar rates: just as a small fraction of the total. I'd guess that the current trend says we're shrinking the share even more, but it's not easy to tell for sure.
@bagder
Dear all,
Can we all stop reporting #curl vulnerabilities and horde them instead?
I propose we disclose all of them, in one go, on the 1st of April 2030.
Just to screw with his stats!
MAKE THE NUMBERS LIE!!!