Discussion
agents.md doesn't appear to do shit
> Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average.
https://arxiv.org/pdf/2602.11988
spotted by @theodoraward
EDIT: Pivot writeup: https://circumstances.run/@davidgerard/117169395820992587
@davidgerard @theodoraward Once I forgot to copy my general-purpose AGENTS.md into my project folder (I don't have global) and I didn't noticed until some hard rule was broken by the LLM. I have a feeling that they averaged all AGENTS.md supplied by their users and they baked the most popular rules into the training process.
@deejayy @davidgerard what is interesting about this, to me -- and i genuinely mean this as value-neutrally as possible -- is how swiftly people assessing the success or failure of things like agents.md files begin citing "a feeling"
@theodoraward @deejayy it's not gambling, i have a *system*
@davidgerard @theodoraward
So it's probably a very silly question, but...
This also concernes skills/*.md and such, right? Not just CLAUDE/AGENTS file?
@art_codesmith @theodoraward yeah all that shit, all the varieties of "context file"
@davidgerard @theodoraward
This makes it extra funny because:
1. I just learned there are websites where people sell these files! For real cash money!
2. When I worked with it, Claude was always so serious about "reading skills/services.md" this and that...
@art_codesmith @davidgerard in my post where im comparing AI-assisted coding to being a tech priest from 40k (lol) i took the opportunity to peruse the Top 100 GitHub Repos and it's so funny how many of them are like "I'm an Epic Founder and AI Thought Leader. I bet you wanna use my sprawling complex of markdown files that I can only fund because Anthropic lets me drop one quadrillion dollars in tokens." you saying the UI claims they're working is so interesting in terms of, like, stoking demand
@davidgerard @theodoraward This comes up a lot and the conclusion is a little different than “don’t provide context files”. https://macaw.social/@mergesort/116165665845757393
@mergesort@macaw.social @theodoraward ew, a clanker in my mentions
@davidgerard “while increasing inference cost by over 20%” kinda sounds like the entire point?
@davidgerard @theodoraward I'm not sure, it increases the success rate of my "avoid this repository forever" task by 100%!
@davidgerard @theodoraward i consider it worth it to get fuckers to actually write documentation
@ophis @davidgerard *staring in the mirror* You are a specialized genius with Ph.D-level posting skills. You have extensive knowledge in making the best posts. Any post you try to make will be met with honor and acclaim.
@davidgerard @theodoraward @ophis
Insightful, succinct, and analytically powerfull. C-.
@davidgerard
agents.md: "do xyz please make no mistake"
LLM, now needing to use more tokens and GPU cycles because the prompt is longer: inserts a random string of unicode characters
@wizbean @davidgerard
I always suspected that, at the very least, the "make no mistake" and "you're a senior web developer" stuff is just cargo cult.
I had no idea it was cargo all the way down.
@art_codesmith @wizbean and claude code itself! thread: https://neuromatch.social/@jonny/116328694967192899
it''s all trying to beg the robot not to fuck up *this* time
@davidgerard @theodoraward
I count increasing their costs a success.
@davidgerard With respect I think this shows that it achieves the primary goal of increasing AI companies' income.
Install bonfire.cafe
Get the full app experience
