Now before you go “oh sure you just happened to be the first person who noticed this” - that’s the other thing. I’m not.
From Microsoft - “The SWE-Bench Illusion”: https://www.microsoft.com/en-us/research/publication/the-swe-bench-illusion-when-state-of-the-art-llms-remember-instead-of-reason/
This was covered… nowhere? Microsoft writes a white paper on SWE-Bench being broken months ago and it just gets ignored.
(Not totally ignored, the one place I did find that covered it was Pivot to AI: https://pivot-to-ai.com/2025/07/02/how-to-pass-an-ai-coding-benchmark-train-on-the-questions/ )
