Discussion
How hard could it be to deterministically log the one kind of data that your whole program is based around to a file?
Well, in the land of LLMs, the answer turns out to be "very hard" - Fable apparently drops around 25% of all its messages: https://github.com/anthropics/claude-code/issues/74260
@jonny
Throw expensive nondeterminism at the wall to see what turnip.
@jonny you know, it would probably be harder for them to make the logging deterministic than to have it statistically imagine what was lost
@jonny I'm not sure von Neumann had any idea whatsoever how weird and intense the state of sin would be when he made the observation about those who consider arithmetical methods of producing random digits.
It's like imagining Dante being shoved face first into 8chan.
The fix, as posted in thread is of course to ask the model to not drop the messages.
@jonny the fix always seems to be "pretty please don't make mistakes"
@jbaert it pretty much is the only possible thing you can do: there are some other fixes in that thread that attempt to use hooks into different event types, but a) one of them bottoms out in a prompt saying "use this tool next" anyway, and b) the model can achieve the same behavior in so many different ways that there is no way to deterministically do anything at all: the model could just use the shell instead of its tools at any time to do anything, i've even seen it try and print a user message to me by invoking its shell tool and echoing a string literal
I also love this one - how hard could it be to select an item from a list, particularly when that selection is the major thing that determines how much someone pays? Again, apparently very hard: people are getting billed for fable usage despite never using it.
https://github.com/anthropics/claude-code/issues/73597
This shit is so funny to me man, I could read these bug reports forever
This person reports how they tried to get fable to build a thing to process some data, but instead burned 12 days of usage on it generating thousands of commits and hundreds of GB of data doing absolutely nothing. It was supposed to build some validation thing that wraps the data processing pipeline, but it never built the pipeline itself, so it then went into a delirium of validating the validator and doing everything except the thing it was supposed to
@jonny the slop writing is so unbelievably grating. How, how, how can people think this is good text?
@jonny Fable: Good news. I was able to build the data processing pipeline you requested.
Is it actually a cactus?
Vimeo
The thing that is extra funny to me is that many of these are also clearly written by the LLM, so its the LLM complaining about how the LLM is broken, and the person that presumably is somewhere in the process doesn't see anything wrong with a machine that is clearly failing to do even the most basic task being the thing explaining why it is failing even the most basic task
Install bonfire.cafe
Get the full app experience
