RE: https://unstable.systems/@jneen/116618931097778342
Worth looking at both the quoted text here and •especially• the linked page, which is quite good.
I’ll add another item of my own. The first screenshot mentions giving an LLM the task of “implementing an HTTP server in JavaScript from scratch” in 90 minutes. Sounds impressive, right? Until you remember that every open-source Javascript HTTP server in existence ••was in the training data••.
1/
IT WAS IN THE TRAINING DATA. Your gave the machine a pile of correct answers and free license to plagiarize.
I remember people being wowed that Claude Code could implement a complete C compiler. But somehow it doesn’t sound quite as impressive when you phrase it as “given every existing C compiler as input, the LLM can produce a C compiler as output.”
2/
It’s just •astonishing• how many eye-popping stories of LLMs doing amazing omg-verge-of-magical-superintelligence things turn out to be just unvarnished plagiarism.
In the first months after ChatGPT’s release, I remember a French dept colleague being amazed that GPT could translate and summarize a passage of Le Petit Prince.
It was a lot less impressive when I dug up the 2 or 3 online passages which it had copied almost verbatim and stitched together (sprinkling in a couple of extra words that made it less accurate).
3/
To be fair, competent plagiarism is a nontrivial task. It takes skill for a human to do: finding relevant fragments from an input dataset and stitching those fragments together in a well-formed way is not nothing! It’s impressive and interesting that people have figured out how to make machines do it.
But •that• is not the promise on which investors are valuing the AI megajuggernaut at trillions of dollars.
4/
The reasons we consider plagiarism to be cheating in school and malpractice in professional contexts are many of the same reasons that LLMs are not going to replace all knowledge-based human labor. (Details left as an exercise for the reader.)
And yes, a whole lot of what LLMs do •would• count as plagiarism if one of my students did it manually, and •should• count as plagiarism just the same if they use a machine to do it — not just in a “that’s cheating!!” sense, but more importantly in a “that’s not really doing the work” sense.
5/
One last example:
The first LLM code example that really made my eyes pop was early after the release of GPT, when somebody got it to combine Breakout with Conway’s Game of Life (a truly delightful idea). It worked!
Funny thing: the Breakout code and the Life code had a •completely• different style and flavor. Red flag. In about 15 minutes of web searching, I was able to find one of the projects (can’t remember if it was the Breakout or the Life half) which it had copied wholesale, with just a few variable renames. And the other half? It was in Python, but it used dictionaries where it really should have used objects — tons of `thing["prop"]` where it should have said `thing.prop`, and lots of other un-Pythonic stuff besides. It was a machine translate of code from another language, very likely Javascript.
The entire thing was a plagiarized Breakout and a plagiarized Game of Life, one transpiled, and all stuck together in a single run loop. To be fair, figuring out how to (1) run both halves of the logic from a single loop and (2) count the Life cells as Breakout bricks is work I'd cheer on from a second-semester intro CS student! It's not, however, quite what's being sold by these companies.
6/
Install bonfire.cafe
Get the full app experience
