I enjoyed this paper by @UlrikeHahn! It's very well written, and I think it's correctly identifying and dissecting an important stumbling block in our discussions of what LLMs are capable of. I really appreciate the attempt to make this dialog more productive
The point is that "reasoning" is a hopelessly ambiguous term, with many different meanings within and across fields. If we want to debate whether LLMs can "reason," then we ought to get much more precise about what we mean.
I strongly agree with most of this, though I do have a few objections. They're all from section 5.4, exploring arguments about "the means" by which LLMs complete reasoning tasks, or whether they lack the means to do what we call "reasoning."
This really is the value of a paper like this: because of Ulrike's careful analysis, I can point to precisely the point where I think the problem lies. Interestingly, "reasoning" may be a red herring.
