I hereby coin the term "Ptolemaic Code" to refer to software that appears functional but is based on a fundamentally incorrect model of the problem domain. As more code is generated by AI, the prevalence of such code is likely to increase.
1/7
Discussion
Loading...
@dk That's a bit unfair to Ptolemy and friends. Their model was purely descriptive. Epicycles are much like Fourier analysis: you can describe any periodic orbit with them. There is no "world model" implied. Observe, fit, extrapolate, that's all. Hardly surprising, given how little people knew about the universe beyond Earth in those days.
Building software on an incorrect domain model is much worse, because today's techies *should* know better!
Like the ancient Ptolemaic model of the solar system, which tried to overcome its fundamentally incorrect understanding of the solar system by adding complex "epicycles" to force its Earth-centred model to match observed reality,
2/7
this code passes all its tests and satisfies, its specifications, yet is built on a fundamentally flawed logic.
AI code generation, which relies on examples, will likely produce significant amounts of Ptolemaic Code.
3/7
It attempts to fit solutions from potentially very different domains together with corrective code "epicycles" until it satisfies its context, the user, and tests.
4/7
This is a trap: while the code works within its parameters, its internal model is incoherent. When it inevitably fails, the incoherent basis for debugging or correction will lead to additional 'epicycles' being added. This process increases system complexity and brittleness.
5/7
The Ptolemaic model of the solar system, though completely incorrect, allowed sailors to navigate the globe for centuries, its failure was not practical but explanatory, similar to how Ptolemaic Code works until deeper issues arise.
6/7
Ptolemaic code works, but when it breaks, it can only be patched, not made correct.
7/7
~2 more replies (not shown)
This is exactly how I have described the limitations of black box models to others.
A good scientific model should do two things:
1) Provide accurate predictions of outcomes given certain inputs, and
2) Enable greater understanding of how a system works.
Simpler machine learning models, like logistic regression or decision trees, can sometimes do both, at least for simpler phenomenon. The models are explainable and their decisions are interpretable. For those reasons among others, applied machine learning researchers still use these simpler approaches wherever they can be made to work.
But in our haste to increase accuracy for more complex phenomenon, we've created models that merely provide semi-accurate predictions at the expense of explainability and interpretability. Like the ptolemaic model of the solar system, these models mostly work well in predicting outcomes within the narrow areas in which they've been trained. But they do absolutely nothing to enable understanding of the underlying phenomenon. Or worse, they mislead us into fundamentally wrong understandings. And because their training is overfit onto the limits of their training data, their accuracy falls apart unpredictably when used for tasks outside the distribution of their training. Computational linguists and other experts that might celebrate these models instead lament the benighted ignorance left in their wake.
Or how it was more eloquently stated in the great philosophical film Billy Madison:
"Mr. Madison, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul."