@jamie @toddsundsted examples of things we learned: handing over a PR should include the plan doc. The plan should be checked off by the AI as you go so it needs no other context to start up again. Plans should have sequence diagrams so you can understand what you’re asking it to do and also see what it built, and so it can understand the overall picture in not that many tokens. We’ve often handed off PRs and happily completed them.
Discussion
Loading...
@jamie AI usage is like a placebo effect. Everyone believes they've been “cured” now. 😂
@jaredwhite I really like the idea of comparing it to a placebo. It's easy to *feel* productive when using AI coding tools. It's a very similar dopamine hit to when you're really in the zone and I think a lot of folks are having trouble telling the difference.
But if they were more productive with it, January would've dominated previous months.
@jamie I can't think of a single product or service or tool which has noticeably improved in the past 12 months that can be attributed to gains due to AI. Not a single one.
"I'm more productive than ever!"
My dude, you're not even as productive as you were in October.
I know PRs per month isn't the ideal metric for measuring productivity. But it's a metric that AI should be absolutely crushing.
The top red line in the graph is yours truly. I rarely use any AI tooling for development. If AI made these folks faster they should be stomping me, but that's not what's happening.
It's easy to feel so much more productive when you "got a lot done" without really doing anything. It's one reason executives think everything is so easy. Don't use vibes. Measure.
@jamie any sense of what the contributing factors are (or are not)?
in my case, i'm still seeing a tremendous amount of variety in terms of what "using AI tooling" means for a team—everything from people still prompting "You are a senior software engineer..." to mammoth PRs to attempts at project/spec-driven development—so it's hard to normalize anything.
in my own case, i think i'm literally my own bottleneck. coding happens faster, but it still takes the same amount of effort to figure out what needs to be written.
@toddsundsted Yeah, it definitely seems to vary. The team's been working on putting together a place to share their prompts.
And I think that's a big part of where their time is going. Talking about AI seems to be a sort of minigame that folks are spending a lot of time on.
Your last point is also huge. Typing isn't where we spend the bulk of our time. We have a lot of legacy code and big clients with diverse and sometimes conflicting requirements. Validating change is critical.
@toddsundsted Even if AI-generated code is speeding up the writing of the code, the team still needs to understand it. I've been very vocal with questions like "Who is an expert in the code that nobody wrote?" "Who understands why we're using a table schema that nobody chose?".
It's like how, as the team has grown, we've had to take over a few of each other's in-progress PRs. Every time, that's been harder than writing it from scratch even though the code's mostly there.
@jamie @toddsundsted I can attest that I and my team have become lots more productive: it’s visible in shipped product. But it means working together on the strategies for coding with AI, sharing commands, learnings, etc. Likewise building tools to leverage AI to understand the code it or others wrote. I posted earlier about the Mermaid tool I built. I have the AI draw me sequence diagrams of flows in the code, ask questions, etc. AFAICT it’s even better for understanding code than writing it.
1 more replies (not shown)