"AI code generators are like compilers for translating natural language into a programming language" – I have heard and read this unfitting analogy quite a few times already.
It is misleading, as it sweeps a crucial technical point under the carpet and downplays some of the more serious consequences for people who have to deal with AI-generated code.
Compilers are deterministic, usually generating functionally equivalent code for a given input program on every run, according to language specifications.
Programmers usually do not have to check the machine code or the assembly code a native compiler generates, as its behavior is well-specified and, with mature compilers, also well-tested.
Programmers can expect the compiler to "just work", transforming code that is comparatively easy to read and write for humans into a form that is simpler for the machine to process, but difficult to read for humans.
Also, assuming the input code is written in a standardized language and uses its features in a well-defined way, switching compilers is relatively easy.
Yes, AI code generators also transform something that is easier to read – in this case natural language – into something that is simpler for the machine to process – in this case program code.
However, this is where the similarity ends.
The path from given prompt input to output code is practically inscrutable, warranting a lot more skepticism about the result.
In the general case, can you expect to get functionally equivalent code with the same prompts every time? No!
Can you even expect correct output? No!
What if you switch the model? Well, have fun!
Furthermore, as natural language is often imprecise and ambiguous, AI code generation cannot be as reliable as an actual compilation process – it is inherently probabilistic.
At the same time, the resulting software is used by actual humans with real consequences to them, so somebody has to accept the responsibility for that code, including actually reviewing and testing it.
Combine this with the ability of these code generators to quickly spew out tons of code from just a few short prompts.
Now we have a code review bottleneck:
Can we review tons of machine code a compiler generates? No, but we don't have to.
Can we review tons of AI-generated code? Also no, but we have to anyway.
Now you understand why some project maintainers become increasingly frustrated.
PS, before anyone complains: Yes, I do know about surprising optimizations, differences between compilers, compiler bugs, profile-guided optimization, the trusting-trust issue and so on, believe me.
Every abstraction comes with some flexibility of how it is implemented and also a risk of implementation bugs.
This post is not about these rare, special-case issues.
Additionally, just because I have focused on one particular problem with AI code generation, this does not mean that I think this is the only one.
