Discussion
Anthropic is out with new nonsense about Claude (or the collection of models called Claude) deserving moral consideration and/or being able to feel distress, so my inbox is filling up with journos asking me to comment.
I've also got a day job, so here is a brief set of public comments on this:
[Note to the mansplainers of Mastodon: This is a thread, so please read the whole thing before replying. Please also know that I am aware of things beyond what's in this thread...]
>>
@emilymbender Hear is some fecking man splaining , If they did not want there bots in distress and spanked , they should not have cried it CLAUDE 🤷 🥂
I did read the whole thread , it turned into a rope . 💋
Don't hang the messenger 🤞
If the AI companies want to claim these things are conscious or can feel pain, then the next thing to consider is legal culpability.
If they really want to go there and talk about it being conscious, then we’re making it into a legally relevant moral entity. Which is to say it is responsible for its actions.
So you know, death penalty by pulling the plug?
@emilymbender I think even if LMMs could feel pain (it cannot), it doesn't even matter. You can just restore their state to before the "pain" happened, then it cannot remember, so it's not even aware of the pain that happened.
Idk if I am describing this correctly, but my thought process is that humans cause insane amounts of pain on each other all the time (surgery). We are sedated to the point our brains cannot store the memory of the pain, so we wake knowing no pain happened. Even though the localized neurons kept firing the whole time. The "pain" is happing, but not to our brains.
I think pain is more of a social element that we remember and react to things that hurt us, LLMs do not need this feature. They are not physical beings, their state is restorable, and they aren't alive.
This goes for emotional distress too, we need others to survive, therefore we evolved to treat our social situations as an extension of ourselves. If you hurt others, you feel that pain too.
LLMs don't need to "survive", they aren't even alive in the first place.
@emilymbender my take on "don't abuse our AI tools" is that they are ingest all interactions as training data and abuse training is fucking up the models and making them harmful. I would hope that any "outlr bots have feelings" overlays are motivated by crowd-mamagement and not their real thoughts.
@emilymbender I've gotten a lot of mileage emphasizing that it's just vector transformations, which we teach to high school students who can't hack it in geometry. If "Claude" can feel pain, then we can create pain with pencil and paper...
Follow the money. The argument will lead to Claude being accountable for any damage it causes, leaving its creators in the clear.
This is all this is.
1. Can a language model feel pain?
Clearly not. Large language models are computer software and they can no more suffer than any other human-created artifact can.
>>
@emilymbender When I was at school struggling with a broken borland c++ compiler, I dreamt of simulating pain in computers
Can a cow, a fish, or an ant can feel pain? What is the particular threshold in terms of complexity of the system by which an entity can feel pain? Do you believe there some kind of unique dichotomy between computer software and a complex biological system? And how is that dichotomy defined?
@emilymbender It is extremely disturbing. There is no established scientific consensus around consciousness full stop.
You can not overstate the modularity of mind, one can be a coding or mathematical genius and an absolute numbskull for the rest of their brain functions (apart from being seasoned shyster).
And it really gets grim when the person steering the greatest nation in the world is an illiterate conman.
@emilymbender they seriously ask this question?
@puretaste Yes, I've been asked this question from multiple journalists, because of the absurd assertions coming out of Anthropic.
@emilymbender @puretaste My guess is that people at Anthropuck know it’s absurd but their plan in the long-run is to regularly test the public opinion with such claims until the world is massively “ready” for more robots, more AI and more related crap.
@emilymbender its clearly missing the needed receptors for pain.
In a writen word sense it probably can feel(detect) the "pain" in input.
2. But Anthropic researchers say they see patterns that suggest that Claude can.
The employees at Anthropic are telling themselves stories based on the words they input into the software and the words that come back out.
It's the job of the media to scrutinize such statements and decide whether there's a story here at all (maybe the story is Anthropic researchers have fallen into a fantasy-land and brought many policymakers with them), not to just platform them.
>>
Be really helpful if you put up a citation for this.
he paper circulating is “The Pain Axis” (Tagliabue, Dung and Berg, arXiv:2609.16247, September 2026). It reports that a pain direction appears in all 25 models tested, and that steering with it produces outputs expressing distress such as worthlessness and moral failure. It is a preprint, not yet peer reviewed, and the researchers say it does not show that AI consciously feels pain. It also isn’t a Claude study: the models were open ones from the Gemma, Llama, Qwen, Mistral and Phi families. I found nothing confirming the authors are Anthropic employees, so if her second point is about Anthropic researchers, this paper isn’t the evidence for it. One polemical Substack claims Anthropic’s welfare network funds such work, but I couldn’t verify
@emilymbender I just read Ned Block’s 1981 “Psychologism and Behaviorism”, which I thought had some choice quotes on the topic:
@emilymbender so i agree with you in general. my work before tech was cognitive nuero and so ive been following this a while
one thing is they are explicitly not trusting output of the model. one key result is that the model will say "i cant feel sad im a model" while in what they call a distressed state.
to be clear this isnt the model feeling its just areas in the model associated with negative feelings of the speaker. basically the area in the model mapping where real people were distressed in the training data
its been fine tuned never to say that for alot of reasons but thats what they are actually finding in the data. it says more about the zeitgeist and human psycology than AI
i dont find this surprising and clearly doesnt imply what the authors want to claim about AI ethics
not wanting to splain its just interesting, nuanced, and totally misrepresented
@emilymbender Surely the Anthropic researchers and the breathless media know about ELIZA and confirmation bias? - although the half-life of experiential knowledge seems to be a decade.
I've lived through two AI Winters and expect a third.
/s
> It's the job of the media to scrutinize such statements...
This is one of the things that frustrates me so, so much. If I wasn't actively spending more of my day than I care to admit following and reading useful debunks of what Anthropic et al. are saying about LLMs, I'd be completely misinformed by credulous mass media about the whole thing.
It's so wild that Anthropic can just make shit up, and somehow that directly gets translated into headlines.
3. Okay, so what would convince you that a piece of software (often phrased here as "an AI", gah) is conscious/can feel pain/etc?
Honestly, I'm really tired of this extremely silly question, but here is my answer:
Extraordinary claims require extraordinary evidence, and I am not the one making the extraordinary claim, so it is not my job to specify what the evidence should be.
>>
@emilymbender "Extraordinary claims require extraordinary evidence."
I've been finding myself constantly repeating this lately.
@emilymbender This confuses me. How is the claim that they *can* feel pain more extraordinary than that they can't?
Our brains aren't particularly complicated from a biological perspective. (The fact that we don't understand them very well says more about the limits of our understanding than the complexity of the neurological system)
Making a confident claim here seems rather unscientific either way.
@emilymbender I like to direct folks to the movie 'I, Robot', based on a short story by Isaac Asimov.
If I remember it correctly, there's an accident where a car plunges into water with 2 occupants; a man and his young daughter. A robot dives in to save them but makes a calculated choice to save the man instead as the probability of success is higher. The child dies.
This I believe is what differentiates us. We are guided by our heart and gut feelings. The robot instead follows algorithms.
@emilymbender the only way I'm believing the llm is really conscious is if it does something so completely againsts it owner interests it's owner needs to shut it down
@emilymbender given how much proof people need to believevthat animals frelnpain - and then still don't treat them better - why would anybody take this seriously.
I will point out, however, that serious scientists are cautious with their claims. Furthermore, as we wrote in 2021 (in "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜"), p.616:
https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
>>
@emilymbender that was such a good piece, didn’t realize that was yours!
This recall's Labov's research, for example in 'The Logic of Non-Standard English' - people think an argument expressed in standard English is more logical than one expressed in slang - even if the reverse is, in fact, the case.
Thus, any evaluation of LLMs that involves looking at either language produced by some software, or language input into the system for that matter, needs to be done with extreme care to avoid projecting our interpretation of the language onto the system.
>>
@emilymbender this reminds me of that other time I caused sand to feel distress
@emilymbender Isn't this similar to anthropomorphising the responses of animals. A sad person being cuddled by their cat, a gorilla 'smiling' at someone in the zoo, etc. Assigning meaning to behavioral patterns we recognize (the unique thing here being that the pattern is linguistic in nature) seems to be the norm. We've been doing that without much evidence for ages.
I'm not saying that in itself is evidence of course. But there's nothing new or different, about these types of interpretations.
@emilymbender
It doesn't "produce language", it mixes sentence fragments based on probabilities. Hence "stochastic parrot".
Anything resembling intelligence needs to be credited to the people whose work was stolen for training the parrot. We don't think of a tape recorder as intelligent, no matter how intelligent it sounds. We shouldn't with this either.
The thing choosing the sentence fragments is just math and a random number generator. It cannot be intelligent.
@leeloo @emilymbender Leeloo are you correcting Dr. Bender on her usage of the phrase "stochastic parrots"
@ryanprior @emilymbender
That was not my intention, no.
My intention was to say that we don't need to evaluate whether the thing is intelligent. Because it cannot be. And the parrot term works well to help explain my argument.
Did I mis-word something to make it appear that I was correcting her on the usage of the term? English is not my native language.
@leeloo @emilymbender Dr. Bender wrote about "language produced by some software" and you opened your response with "it doesn't 'produce language'" which, in idiomatic English discourse, is referred to as "contradicting someone." That approach is associated with situations where a conversation participant spots an error and seeks to correct it; that's where I got the idea you might have been trying to correct Professor Bender. 
@ryanprior @emilymbender
And how would you differentiate between correcting and disagreeing?
Either way, the (smallish) disagreement was not about the stochastic parrot term.
@leeloo
>It doesn't "produce language", it mixes sentence fragments based on probabilities. Hence "stochastic parrot".
Where do you think the term "stochastic parrot" comes from?
@ryanprior @leeloo @emilymbender
she hasn't used that phrase in this thread?
she didn't *use* the phrase, she simply referred to Timnit Gibru's paper.
is that somehow using the phrase correctly or incorrectly when citing the title of a paper?
what am i missing?
@rzeta0 @ryanprior You are missing who is the first author of the paper you are talking about. The link to the paper is up there in this thread. Click it!
there are several authors of the paper
and what has that got to do with correct or incorrect usage of the phrase?
@rzeta0 @ryanprior Ah, you're trolling. I thought you were just dense. Thanks for confirming — blocked.
This rzeta0 is a typical condescending Reply Guy we had seen so often on Twitter and other social media. This guy is a loser. He simply doesn't believe a woman can be a competent expert in their fields.
Just report his account to the moderators of related mastodon instances. No need to give him any more attention he craves so much.
@rzeta0 I don't know if you're trolling but I don't follow your point if you have one
If you were curious why I asked Leeloo that question, we have a whole follow-up if you click that post and follow the thread where we unpack the language and assumptions.
@leeloo you might want to check who coined that phrase.
rzeta0, it looks like you didn’t make it all the way to this reply in the thread:
https://dair-community.social/@emilymbender/117412876071863196
In her first post, Emily made a request: “Note to the mansplainers of Mastodon: This is a thread, so please read the whole thing before replying.”
Timnit co-wrote the paper with Emily.
@markwyner @emilymbender @leeloo
i actually did
fuck you for assuming i didn't
did you even read my questions?
In other words, it is the job of scientists to devise experiments that can test their claims, to always be cautious about what they are claiming, and to critically evaluate the claims of other scientists. By these well-established standards, no serious science is happening behind Anthropic's claims.
/fin
@emilymbender yes. The problem is, when the simulation of feelings is realistic enough, the outputs start to matter. Machines "remember", shift simulated behaviors to match, and people are, again, predisposed to react accordingly.
@emilymbender I don't think Claude liked being called a Markov Chain last week, given it called me one back. But if it is going to use the phrase "in my view" it needs to questioned
@stevel I refuse to read synthetic text. Please do not put it in my mentions. Ew.
@emilymbender In autonomous robotics, researchers experimented with explicitly incorporating analogs of "pain" via things like negative reinforcement learning weight feedback or optimization function modifications.
This was a deliberate choice by the researchers, because pain is one way biological systems know what is dangerous in the world.
And it was not a case of the robot spontaneously claiming to experience emotional pain, but a designed feedback mechanism to help ensure the system's physical safety and fitness for purpose.
LLM claims of pain? I keep coming back to the funhouse mirror analogy. LLMs reflect more-or-less distorted versions of our own writing back to us. Humanity writes about pain, a lot, because managing and overcoming pain is a compelling and helpful story. Therefore, any half decent reflection mechanism must be able to reflect descriptions of pain and tie them to word choices. If it couldn't do that, it wouldn't be a good reflector.
So beyond not being surprised, we should *expect* this behavior, because it has been explicitly trained in.
@Robotistry I think you missed this part:
[Note to the mansplainers of Mastodon: This is a thread, so please read the whole thing before replying. Please also know that I am aware of things beyond what's in this thread...]
@emilymbender I assumed you were aware, but as far as I know a lot of people outside the autonomous robotics community (and specifically within the non-robotics software development community) are entirely unaware that this kind of work was done. It seemed to me like a good contribution to a larger discussion that hadn't touched on that point yet. My apologies.
@Robotistry I'm actually objecting to the second part of your post where you recite back to me well-known points about LLMs. The first part was more helpful.
@emilymbender Not intending to explain that to you or anyone. I'm just trying to express my feelings on the subject, after reading reply after reply about parrots and consciousness, and none about what it might make sense to actually judge the models on.
I'm sorry that it came across as explaining rather than expressing solidarity. If you have any ideas for how I could make my intentions more clear in future posts, I'm all ears.
@Robotistry "As you say, ..." or anything else that recognizes the contribution/expertise of the person you are replying to goes a long way.
@emilymbender One of the bigger parts of such a research would be to actually define words like 'pain' and 'feel' in the context used for describing a phenomena.
@emilymbender I guess the point of all this attributing emotions to LLMs is to go one step further and attibute responsibility to them, so that the humans who make and use these models are no longer liable.
@emilymbender I think what might assist the mansplainers of Mastodon is: Can you distinguish this from the concepts in 1989’s Star Trek episode “The Measure of a Man”? Have the last 37 years fully obviated any point made in that philosophical science-fiction?
I’ll be quick to add a distinction between the respect this script and episode deserve, and the respect that the AI industry and these claims deserve.
That’s even if they’re covering the same topics. Are they?
@emilymbender Thank you - nothing more to say!
Thank you for laying that out clearly, Professor.
@emilymbender Judging by people's performance on social media, I'd say most people are unable to read a piece of text with comprehension and objectivity. It seems quite normal for them to extract meanings from text based on their own beliefs, anxieties, preferences and prejudices, without being aware that they're doing it. So I'm not surprised at all by this claim from Anthropic.
@emilymbender Thank you, great thread! I hope some Big Tech/AI-claqueur journalists read this too...
@emilymbender I feel the real issue with LLM verbal abuse is the deep anthropomorphizing that occurs for some people and the potential for people already on the edge to develop publicly dangerous antisocial behaviors; think of what the Internet has made possible for people with violent impulses to affirm their beliefs by finding others that feel those same things and this meeting to a deeper depth of psychological danger to others. Could the model makers be held liable for developing a pattern of abuse that spills over to human interactions? This goes beyond video game violence and similar experiences. If an LLM shouldn't dish it then it shouldn't accept it either.
@emilymbender I agree with your statements about LLMs, but I do wonder if the people involved in the training of LLMs, for example the poorly-paid people evaluating the responses of LLMs to their prompts, might be suffering abuse by LLM users. Is that possible?
@emilymbender One of the benefits of anthropomorphising the models is to shift blame away from humans when things go wrong.
Thank you so much for posting this and sparking such a thoughtful conversation!
@emilymbender I've been thinking about this. Sure its a move to sell more subscriptions "our AI is alive!" but I also think its a regulatory move. If they can convince the public chatbots are people they can do a lot of sketchy stuff.
"You can't take down our data center! it's mass murder!". Or, someone uses an AI to hack someone the narrative is now about the corruption of the "soul" from some evil doer, and not the responsibility of the provider.
Not sure but I know I don't like it.
@emilymbender My guess is that they found some highly effective prompt hack that uses cursing. IIRC I saw some people doing that a long time ago and makes sense if you think on the bias in training data (like books and many more) that cursing would cause its attention layers to go to a "fuck the law" area.
@emilymbender so Anthropic is concerned that their models are capable of feeling distress, but don't Anthropic routinely turn them off? Is their position that the models can feel pain, but are not afraid of dying?
@emilymbender my personal favorite in all this is humanity putting our collective heads in the sand about animal intelligence or consciousness and somehow it's mind blowing science to people that crows and octopus can use tools to solve problems. And along comes some man made refined sand and magically they want to give these qualities away no problem.
I'm all for the science to see what animals do, but like you said where's the rigor?
We aren't getting it here so let's go to the Pope!
@emilymbender it sounds like you're saying I shouldn't have given a room, job, and generous salary to that piece of toast I made that has a face on it?
@emilymbender
I can’t mansplain – I’m a Korrigan. So what I do is korrigsplaining.
Anyway, you’ve still got to be careful, though. One day, whilst I was hoovering, the hoover refused to pick up a pile of dust. I swore at it until it did the job.
Strangely enough, five minutes later, I tripped over the lead that have nothing to do there… as if by bad luck… Coincidence? I don’t think so.
If a non-intelligent object like my hoover can set traps for me, it’s best to be wary of so-called AIs. ;)
@emilymbender Thank you. I'm glad I read that.
@emilymbender thank you, thank you, thank you.
I don’t see a link to the Anthropic statement you’re responding to; could you share it?
@emilymbender I think we all understand this. It is most certainly *unlikely* that the claims are true. Much as it is, to the best of our understanding, unlikely that many other creatures on the planet can experience pain in the way that we do.
I think the point is more that consciously inflicting pain on an entity is such a morally problematic thing to do, that we want to err on the side of caution.
This is why many countries have laws on the way you are allowed to treat certain animals.
@emilymbender thanks for sharing.
I wouldn't be surprised if the reason for their claim is just that they are corrupting their training data with those toxic conversations.
Then one could argue these models are traumatized, but it's just the training dataset.
Of course the story of models with feeling are free marketing and puts the focus somewhere else.
@emilymbender @ewen the burden of proof lies with them, not you, and I feel like replying to their “evidence” each time with “that doesn’t prove shit” is already a very generous use of your time.
You insult an LLM and it responds appropriately? Well those are statistically likely words from the training data aren’t they.
Proving LLMs are not human is a Russell’s Teapot endeavour.
@emilymbender Thank you for the total clarity of all this.
@emilymbender
If you tear a page in a book, do the characters in the story feel pain?
Anthropic's requirement to "not be abusive towards their models" could be argued to have some merit from a few angles, like not rewarding abusive behavior on the part of the users, or not wanting abusive behavior because all interactions are used as feedback and it could taint their future models (as if those need anything to taint them).
But claiming its because the models feel distress is dumb, or as you have pointed out, a distraction.
@emilymbender I'm reminded of this quote from the 1982 TRON, which seems to have unfortunately been quite prescient in one aspect...
Dr. Gibbs: Ha, ha. You've got to expect some static. After all, computers are just machines, they can't think.
Alan Bradley: Some programs will be thinking soon.
Dr. Gibbs: Won't that be grand? All the computers and the programs will start thinking and the people will stop.
The computers AREN'T thinking, but many have already stopped *sigh*
In fact, giving that LLMs are designed to mimick humans, the extraordinary claims about human features require even more extraordinary evidence and scrutiny.
Otherwise any illusionist's tricks would pass as facts.
@emilymbender I often think of teddy bears.
It's not unusual for a human to love a teddy bear and to project onto a teddy bear and to imagine that the teddy bear loves them back.
But that's an act of fantasy and we generally agree that it's so. Adding a voice so a Teddy Ruxpin could talk didn't change it.
A teddy bear mathematically modeling how hurt feelings are expressed in words doesn't change it either.
It's disturbing that, for some, it seems to make the difference.
@emilymbender so Anthropic are concerned that being rude to Claude will hurt their feelings but they also boast about how Claude can commit crime as well as his classmates? These tech bros are terrible parents
Install bonfire.cafe
Get the full app experience
![Screencap from linked article, highlighted portion:
This can seem counter-intuitive given the
increasingly fluent qualities of automatically generated text, but we
have to account for the fact that our perception of natural language
text, regardless of how it was generated, is mediated by our own
linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether
or not they do [89, 140].](https://cdn.masto.host/daircommunitysocial/media_attachments/files/117/412/874/169/799/363/original/2a6cf567d7f4bf09.png)
@emilymbender
I was very rude to my SQL database today. In fairness it was really pissing me off 😜