Post by @UlrikeHahn@fediscience.org

@UlrikeHahn@fediscience.org · 2 hours ago

Just heard the first talk at the conference on reasoning and agency in AI at the Institute of Philosophy by Nick O’Shea. 🧵

https://www.sas.ac.uk/news-events/events/philosophy-ai-conference-2026-reasoning-agency-ai

He looked at outcome-directedness and goal-directedness in biological systems and compared them to AI systems. The crucial point was that both biological and current AI systems can exhibit outcome-directedness and goal-directedness, and both show capacities for improving themselves by finding “better” solutions to those outcomes or achieving those goals, but biological systems are forced, simultaneously, to find *both* new solutions and the resources to effect those solutions *simultaneously*. Current AI systems do not and this has implications for ‘doomsday scenarios’

#philosophy #AI. 1/

#philosophy #ai

TheTootingTitan

@seaborgium1234@mastodon.social · 55 minutes ago

What if the very formation of an idea was contingent on also discovering its own resources?

Ulrike Hahn

@UlrikeHahn@fediscience.org · 2 hours ago

2/ It makes a standard `doomsday’ scenario where these systems become ever more powerful and seek to replace us unlikely, for the simple reason that the critical resource (energy for compute) is controlled by us. A much more likely scenario of concern is an undesired development of free-riding to greater (and potentially damaging) complexity in the way viruses (who lack metabolism of their own) show: basically, AI systems develop better goal-directedness by parasitically drawing on human behaviour, by modifying human behaviour.

Ulrike Hahn

@UlrikeHahn@fediscience.org · 1 hour ago

3/ Crucially, these types of process of improvement arise in biological systems and can equally arise in non-biological AI systems simply through synergistic, causal feedback loops. They do not require *intentions* on the part of the system itself. Biology provides us with many clear examples of this. Likewise, current AI technology already contains examples of mechanisms that could potentially serve as causal ratchets for system self improvement.

Ulrike Hahn

@UlrikeHahn@fediscience.org · 55 minutes ago

4/ some reasons why I found this talk thought-provoking:

1. there's a strand of critique of AI capacities that emphasises that they are not living organisms (e.g., to argue they don't have agency, or don't 'really' reason or 'think'), but the question there, to me, is always but what about being a living thing matters to those capacities? As Nick O'Shea himself put it, this can come periilously close to simply positing some utterly mysterious, but essential, "life force'.

highlighting the need of biological systems to simultaneously optimise both 'improvements' to capacities and required resources (e.g., metabolism) makes the role clear in ways that are relevant for thinking about concerns about AI....

Ulrike Hahn

@UlrikeHahn@fediscience.org · 50 minutes ago

5/ it makes some concerns less plausible, while also pointing out that the decoupling of these two functions in current AI systems doesn't preclude the possibility of comparable causal feedback loops toward (adverse) greater capacity in practice

At the same time, locating that function of 'living thing' brings into sharper relief (for me) the distinction to intuitions about 'living thing' and capacity itself- an intuitive sense that if it's not alive it's not *really* thinking or planning or reasoning, say, because it's not 'conscious'

1 more replies

Ulrike Hahn

@UlrikeHahn@fediscience.org · 2 hours ago

relatedly- for a recent paper that explores the theme of how biological evolution might inform our concerns about AI developments, this is worth a read:

https://www.pnas.org/doi/pdf/10.1073/pnas.2527700123

Albert Cardona

@albertcardona@mathstodon.xyz · 2 hours ago

@UlrikeHahn

Call me naive but what prevents anyone from pulling the plug? As in, as you mentioned earlier, resources are a constraint that biological systems must always consider and LLMs so far don't. Perhaps it's a failure of my imagination, but I can't quite see where the alarming part is. A physical substrate and an energy source are always required; that's how geopolitics work too, controlling vast human populations.

Ulrike Hahn

@UlrikeHahn@fediscience.org · 1 hour ago

@albertcardona I wish I had access to the OpenAI plug, but I don't ;-). ..so probably not 'anyone', might be part of the answer

the other part of an answer, I guess, comes from thinking about how viral free-riding could actually occur (see @suzannealdrich reply): the systems are `giving' human users something they value enough to let the systems persist

but most importantly, what makes this interesting to me is that all of this is relevant long before a 'takeover doomsday scenario': it simply points out non-mysterious causal pathways to unintended development trajectories of increasingly harmful AI...

from that standpoint, I personally feel we are already living in the problematic `viral' world scenario where perceptions about value (and concrete interests) are sufficiently diverse that real harm is not being stopped