What is reasoning anyway? A closer look at reasoning in LLMs
Ulrike Hahn
2026-01-13
There is a remarkable degree of polarisation in current debate about the capacities of Large Language Models (LLMs). One example of this is the debate about reasoning. Some researchers see ample evidence of reasoning in these systems, while others maintain that these systems do not reason at all. This paper seeks to shed light on this debate by examining the divergent uses of the term reasoning across different disciplines. It provides a simple clarificatory framework for talking about behaviour that highlights key dimensions of variation in how ‘reasoning’ is used across psychology, philosophy and AI. This highlights not just the extent to which researchers are talking past each other, but also that common inferences about model capability that accompany classification decisions are, in fact, far less compelling than they might seem.
URL: https://zenodo.org/doi/10.5281/zenodo.18231171
DOI: 10.5281/ZENODO.18231171
@UlrikeHahn Lots to say on this but alas more thoughts than time! Some quickies - the System/Type 1/2 distinction seems unhelpful when trying to analyse task performance - it's always a bit of both and I reckon more useful to discuss more specific processes (at some comprehensible level of abstraction) like parsing, memory, surface level text heuristics, beliefs, etc. I had a brief digression in my 2009 PhD thesis on this... Personal/subpersonal maybe helps!