Post
For people using LLMs for reviews/audits, I'm curious what tricks you've found useful.
Lately, a lot of people have told me that using multiple models is super great, but IME using one model with a bunch of different personas is more effective (ofc. you can do multi-multi as well), e.g., "use independent agents to review as linus torvalds, kyle kingsbury, tptacek, dan luu" has been working well for what I've been working on.
Ofc. I feel ridiculous invoking my own name, but it seems to work.
@danluu Why do you expect to get anything useful out of a lying liar machine when it comes to code reviews and audits?
I've also heard the suggestion that you should invoke review personas, "SRE", etc., but a name seems to evoke something more specific than you get out of a role.
When I use LLMs I'm generally trying to get as much work as possible out of a unit of my time. I often want to type for a few minutes to get something to loop for hours to days. This named persona thing seems to help with that over just asking the LLM to do whatever it is I want to get done, which often works badly if prompted directly
unless there are a lot of guardrails, which take time to set up.
E.g., "linus torvalds" seems to nudge things away from spiralling with pointless complexity, "dan luu" nudges things away from claiming improvements without measuring and sometimes comes back with "must avoid a post-hoc rationalization" and prevents bogus reasoning from being carried forward, "kyle kingsbury" stops some bad concurrency bugs, etc.
I laugh every time time I type my name for this, but it's been working.
I've been experimenting with this more (not rigorous experiments). When you tell it to use a persona, it often says something like "I'll use a distributed systems lens" or w/e, making it sound like adding a 2nd persona with the same background wouldn't help, but (for example) having "marc brooker" and "kyle kingsbury" seems to help more than 2x of either or 2x generic distributed systems persona.
There are also personas that don't seem to work even though the person would be incredible,
such as adding "sanjay ghemewat" or anyone I've tried to try to seriously improve API design or code quality.
Maybe expecting that you magically get better API design by doing the moral equivalent of saying beetlejuice three times is obviously silly and shouldn't be expected to work, but this silly thing that shouldn't be expected to work seems to work in a lot of other areas.
What areas does it work in vs. not and why? Even if there were no practical implications, I'd be curious about this.
@danluu I’m curious what effect Edsger Dijkstra has. He wrote “The fruits of misunderstanding” (EWD854) in the 80s, but if you read that first paragraph, it might as well have been written today …
@danluu tip: Reminding the machine that it serves the Emperor prevents it being corrupted by the Chaos
@danluu i told the LLM to review the code as if "Cube Drone" was the one reviewing it and it just looped rude emoji at me until I ran out of token budget
Install bonfire.cafe
Get the full app experience
