One more thing and then I'll stop on this topic. In 1995 our Computer Vision / Robotics lab at UVA had a waist-high, trashcan-shaped robot with lidar, sonar, and bump sensors. It was powered by something like an 80486 CPU, and probably had something generous like 32Mb of RAM. (Megabytes, not gigabytes). And this robot could do the "R2D2 bench."

In Return of the Jedi in 1983, there's a brief scene where Jabba the Hutt has R2D2 serving drinks on his barge. R2D2 is autonomously roaming around the party floor with a tray of drinks and people are picking them up. Our robot could do this in 1995 using far less than a billion dollar data center.

It's absurd to use such horrendously wasteful technology (LLMs) to solve a problem that has already been solved far more inexpensively.

When people spend time to make DOOM run on something absurdly underpowered or unexpected like a photocopier, I get it: do it for the sake of doing it.

This experiment burned outrageous amounts of resources doing it with an LLM, when it's a well-known problem with far better existing solutions. And then the LLM does it worse than basically everything we already have (40% completion rate, to say nothing of how well it did it), the "do it for the sake of doing it" reasoning doesn't make sense. Why try to refine this and make an LLM do it better, when it can't possibly be better or cheaper than what we already have?

The objective of the stupid experiment was to use LLMs, not to make cost-effective robots that work in practical home situations.