@VeeRat
In general, machine learning is good for situations where the value of a correct answer is significantly higher than the cost of an incorrect one. Prefetching is my favourite example (not least because that’s what I used it for in my PhD): if you prefetch the right data, the processor avoids a long idle time. If you prefetch the wrong data, you discard it and you’re not in a worse position than if you hadn’t used it.
Machine translation is a great case study for this: for any given situation, is an incorrect translation worse than no translation? For translating a toot, the worst-case cost is that I am briefly confused and move on. I don’t have time to learn every human language and I don’t have the money to pay someone to translate every toot I see, so machine translation is better than nothing. For translating a user manual or a contract, the worst-case cost may be someone dies or I am exposed to unbounded legal liability (or both), so machine translation is worse than nothing.
The corollary is that machine learning is also useful if the cost of generating an optimal solution is high and it’s possible to mechanically generated solutions. Stochastically generating plausible solution-shapes things then checking if they actually are solutions and picking the one that works best with a cheap-to-run machine-learning system providing the inputs is great. Vulnerability discovery with guided fuzzing works like this, for example.
The big issue with LLMs is that, generally, text is something we use for high-stakes problems. The set of things where the desired output is a pile of text, where the cost of incorrect text is low or the cost of checking the text is low, is very small. This includes programming: if you have a sufficiently detailed spec that you can mechanically check whether an implementation satisfies it, you almost certainly have something that can be used to synthesise a program already.
Some cases, such as searching documentation, are right on the cusp. Is it better to have a machine that answers a question about a system but gives you a right answer 60% of the time and plausible nonsense 40% of the time than nothing? What if it’s 90%? At some point, the Paradox of Automation kicks in: system efficiency does not correlate with component efficiency. You get a significant dip when the system is good enough that people stop checking the answers but it’s still producing a lot of mistakes, only now those make it further along the process.
