@simon

I am going through exactly the same thing with actual humans.
The lack of decision RECORDING is a huge impact on even simple code.
The lack of validation of WHY we reach a decision is massive.

With humans the individual remembers, and it also remember it’s ow hallucinations (I THINK they mean this)

Ironically I had Agentic AI spitting out every decision made into Markdown, that is a tangible teaching tool to my team.

@simon Hello friends, this is Diaa from Gaza. 💔
I’m in severe pain, and doctors say I might lose my leg if I don’t get urgent surgery soon.
Even $10 or any help you can give would mean the world to me. 🙏😭
Please share my story and help me survive this. 💔🕊️
https://chuffed.org/project/133500-support-diyaa-a-home-a-future-and-hope-lost-in-gaza

@simon "Claude, walk me through this part of the implementation" can be helpful, both to understand what it has been up to and identify things that can be consolidated.

And particularly useful in other people's code bases.

It's a fine line between getting out of the way, and losing control. Why waste time on an out of date mental model, if the details aren't actually important?

@simon
Oh boy... I have enough cognitive debt in my personal projects even with no AI involvement whatsoever.

@simon I think that using AI reinforces our "System 1" mode (Kahneman) and weakens "System 2," so we lose the ability to reason properly.

@simon The tooling is improving so quickly that it makes sense to regenerate code from the spec and tests if you’ve been away from a project for a while. I would prompt something like: review the existing spec and tests, create additional BDD and integration tests then re-implement all the code.

@simon I appreciate you writing about this, specifically because a lot of people in LLM-positive coding circles look to you for advice.

This is a key part of why many open source maintainers (myself included) have been averse to these products from the jump (reaffirmed by our interactions with people who have thrown in). We cannot faithfully execute the role of maintainer without clear understanding of tacit assumptions, limitations, and nuance of leaky abstractions. While some can be addressed through documentation and tests, PR review is an important medium in which to disseminate understanding within the community, and possibly conceptualize better approaches. The trust of the community depends on performance of these cognitive processes.

I believe the effect you describe becomes more insidious in larger projects, with distributed developer communities and bespoke domain knowledge. Such conditions are typical in research software/infrastructure (my domain), and the cost of recovering from such debt will often be intractable under public funding models (very lean; deliverables only for basic research, not maintenance and onboarding). Offloading to LLMs interferes not just with the cognitive processes of the "author", but also that of maintainers and other community members.

@simon
Sounds like what a typical technical manager in IT is like, in my experience. They often don't really have a mental model of how things work, how they're put together.

@simon I am sure you’ve read this, but in case you haven’t, Peter Naur’s article on “Programming as Theory Building” has been very useful to me in talking about the maintenance of ai-augmented systems:

https://pages.cs.wisc.edu/~remzi/Naur.pdf