Post
I'm not sure what to do about this. With LLMs, it's easier than ever to test things and the time/cost it takes to get a piece of software to a particular quality bar has gone way down and, simultaneously, software quality seems to be getting much worse.
I've sat down with a couple people and wrote a fuzzer with them (pre-LLM) in 15-30 minutes and converted them for life, but I haven't figured out a framing that actually works when I write it as a blog post.
@danluu what does fuzzing/random testing look like for business logic? It feels like a lot of the examples are of the “thing crashes” variety but with CRUD apps where we tend to be reading data, transforming a bit, then throwing it back into the DB, it feels like the failure states are way less obvious
Have you had success in that space?
I'm not sure what to do about this. With LLMs, it's easier than ever to test things and the time/cost it takes to get a piece of software to a particular quality bar has gone way down and, simultaneously, software quality seems to be getting much worse.
I've sat down with a couple people and wrote a fuzzer with them (pre-LLM) in 15-30 minutes and converted them for life, but I haven't figured out a framing that actually works when I write it as a blog post.
@danluu I enjoyed the post, even if some references and parts went over my head some.
I think I’d really love to see a video recording of you setting up a fuzzer with someone who’s not a convert as it were, going through the process of set up, through to success.
Other than that, it might be interesting to have people who you’ve helped blog about the process and what they’ve ended up doing
@danluu tests & fuzzers good. LLMs bad: unreliable, privacy hole, IP theft actor, legal provenance ambiguity, then its amorality, career risk, dependency increase and de-skilling.
Maybe I should write an updated version of that post for the LLM era?
I don't know that it would be more convincing, but I recently tried using a text editor and hit a bug in it, so I tried vibe coding a fuzzer and it took ~6 minutes of my time to find the first bug (text/data corruption).
With LLM triage and review of the output, there have been zero false positives so far and I'm getting "my" patches merged. I'm blown away by how easy this is to do now and yet, software quality is declining.
@danluu "Maybe I should write an updated version of that post for the LLM era?"
Please do!
@danluu I think we may rebound re: quality, but right now we’re just accelerating that software path dependence.
LLMs were (in most orgs imho) introduced top down and need ROI, EMs are measuring easy thing which is tokens and features. Quality is still as hard to quantify as ever and bugs are still culturally hard for orgs to manage, so…it will just take some time.
I think a lot of the claims about LLMs increasing productivity by some huge factor (10x, 100x, 1000x) are overblown, but on testing/quality, the speedup is pretty incredible.
It took me about an hour to do what would've previously taken me a week or two given that I'm not familiar with the project. It's not just startup cost, additional time is absurdly productive (maybe even more so) because a lot of time consuming manual work is automatable, which puts you in the right side of Amdahl's law.
@danluu I had a similar experience with LLMs and was writing fuzzers for Firefox for several weeks. And then I discovered that for 80% of the issues I was finding, I didn't even need to write the fuzzer. I can just ask the LLM where the bugs are and it finds them with basic code auditing. My pattern now is to have an LLM do several passes of auditing and then recommend places to hit harder with an LLM-generated fuzzer.
@danluu Even if we accept the premise of corporate productivity we've always had a group of people who cared mostly about how their work performance looks like rather than what it actually is.
If you assume we only have competent people who care about their results - sure. At a meager cost of pricing out a family in OHIO out of utilities you can speed up the repetitive part of development.
If you look at the minority which never cared about quality - you speed their "work" up by factors more.
@danluu what do you think about the ethical concerns?
@danluu Have you used Hegel yet? I think "fuzzing" is a little bit different in spirit than property-based testing, but the tools can seem similar.
And the arguments about LLM are the same for both fuzzing and PBT.
Hegel's shrinking and caching story has worked out really well for me in practice. And I'm in a pretty normal language of just TypeScript. It's grown well past Haskell.
@shajra I've found libraries/frameworks like that are often counterproductive because the performance isn't there and I get tests per second by writing a custom thing.
Never tried Hegel, but it looks like it has a Python backend, so I expect the same issue there. It seems like there's a plan to re-write that to be faster at some point, so I might give it a shot then. I'm using the term fuzzing loosely and probably doing what people call PBT, I guess? I've never paid much attention to the terms.
@danluu It might be less about LLMs and more about the quality is thinking and planning of people who build things. Tools and automation can improve speed, but if plans have bigger mistakes, end results often lack a lot.
@danluu Timothy Fitz had a very similar experience after writing this blog post: http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ - e.g. "Sure it works for IMVU, but it'll never work for me"
Install bonfire.cafe
Get the full app experience
