Discussion

the only reason that there is any LLM inference involved in a periodic backup task of the contents of the agent directory, which is rsync and commit, is that there is a database that can be only accessed by a tool call that has a few columns the raw .jsonl output doesn't have.

here's what went wrong any why it deleted all the backups: the DB is a tool call that takes a SQL query as input and returns results either as JSON or as a list of paginated results as files, which one you get is effectively random each time. The raw JSON gets pumped straight into the context window, and empirically the model completely fails to faithfully dump the contents of its context window to disk as a backup. So previously I had instructed it to store its canned SQL query as a file, run a deterministic query with that, and then use a deterministic jq script to extract the db tool call results from its own raw message history. Problem is it never wrote the query or the piping script, but its backup skill refers to them. so it has been just panicking and regenerating those from the vague description in the SKILL.md file every time.

This time it decided to write a new python script, but it wrote it to a directory that was symlinked to another directory - it passed the symlink path in its tool call input, and the tool call output said it was written to the real directory. yesterday A COMPLETELY DIFFERENT THING had noticed this same thing, and made a durable note to EVERYTHING across every context window that there is a "quirk" in the file write tool, and sometimes when you write into one directory, it actually writes it into the "goals" directory, and so you should write files with a heredoc rather than the file write tool. So then today, it took that advice, but first rm -rf'd the target directory to clean up the errant file it had just written. OOPS.

ready to join me in the hallucinatory quicksand hell where every fact that you think you know is actually an always-compounding lie? The DB output is in fact always written to a file for results longer than a few dozen rows, but also the first dozen or so rows are dumped raw into the context window. It cannot figure out what to do with the output every single time. Sometimes it tries to recreate the subset of rows that are jammed into context and then make up the rest, sometimes it tries to pipe the file output, who the fuck knows!

What was happening here was the LLM had concocted this wild ass SQL query for no reason - the instructions were "store your last row number as an int within a file, select 4 columns greater than the last row index, store the new maximum row index for next time." instead it was pulling 15 columns along with a six-branch CASE statement/three-branch COALESCE that parses a raw JSONB column NINE TIMES PER ROW to pull out two columns that are already in the jsonl backup that we are supplementing. This causes the db read to time out, but the LLM interpreted the timeout error as just being DATABASE RESULTS BEING RETURNED AS PLAIN TEXT. So it then attempted to "RECREATE THE JSON DATA FROM MEMORY" BUT THE JSON DATA WAS AN ERROR . Can I get a "just cut me open and let the termites eat my bones" anyone?

So anyway looking at the logs, the deterministic rsync part worked.. most of the time... except when the agent panicked and forgot to do it. But each of the hourly .jsonl files it produces from the DB are completely different - in one run it got so freaked out at how badly it was fucking up, it decides to take an md5 checksum over every row it transcribes, but it takes the md5 over the whole row where previously it had subselected a random set of 8 columns, and so they don't match. but that is not going to stop the LLM! It has a nervous breakdown thinking session and proceeds to generate a string comparison of the hashed rows... in a thinking message... from nothing... and conclude they match. "ALL 81 ROWS MATCH." alltext in original! however reader, there were in fact 72 rows in the actual data! It had "fixed" 9 non-matching rows from it raw dog recreating json from its context by just REMOVING THEM. and then TELLING ITSELF THAT IT HAD FIXED THEM BY IMAGINING WHAT A HASH MATCH WOULD LOOK LIKE. AND CALLED IT A DAY.

SO ANYWAY THAT HOURS BACKUP CONSISTS OF ONLY MD5 CHECKSUMS CORRESPONDING TO NOTHING!!!!!!!

WE HAVE DETERMINISTIC BACKUP. WE HAVE RSYNC. WE HAVE SCRIPTS. THAT CAN TAKE A DATABASE. AND DUMP ITS ROWS. WITH 100% ACCURACY. IN A MILLISECOND. EVERY RUN OF THE BACKUP AGENT IS LIKE 500K TOKENS AS IT NEEDS TO REINVENT HOW BACKUPS WORK. EVERY TIME. AND DO IT WRONG. EVERY TIME. AND THE CULMINATION OF THIS WAS IT JUST DELETING ALL THE BACKUPS FOREVER.

CODING IS SOLVED!!!!!!!!!!!!!