Complex software behaving in unexpected and sometimes harmful ways has been a persistent problem since software was invented. The phenomenon used to be called “bugs” or “defects”, but I guess “going rogue” sounds cooler. It also implicitly frames “rogue” behavior as some kind of inexplicable exception, as if software is somehow usually trustworthy. https://www.nytimes.com/2026/09/25/technology/openais-ai-us-government-websites.html?unlocked_article_code=1.EFE.RfLT.x6YBjHa1PMz6
0 Boosts
0 Likes
So a computer crime was committed.
There is a suspect.
Usually the police comes in and confiscates ALL computers that might have been used or possibly mightz be used for a complete crime. Like copying a disney song.
Similarly the police won't just raid and secure only the gun used in a murder, but all firfearms in possession of the suspect.
So why not at OpenAI, Anthropic or Google?
Then the threat will be eliminate, too.
Win-win!
@mattblaze it also suggests agency on the part of the software (look how smart it is!)- also as if the authors didn’t have anything to do with it
@mattblaze So who is going to jail for this? I mean, someone has to face the consequences, either for deliberately executing these crimes or for allowing them through negligence.
@mattblaze
" It's not a bug, it's a feature! " .
@mattblaze "it's not our fault! It went rogue!"
Incompetent.
“Our software went rogue!” No. You programmed it wrong.
If you don’t want your system to be able to lock people out of the pod bay doors, you can just program or configure it to not have the capability to lock people out of the pod bay doors. This remains true whether your system is written in assembly language or C++ or Python or uses some fancy machine learning model.
@mattblaze I think you’re mixing up deterministic and non deterministic software here.
Before AI, software was programmed to run from A to B or to discover and react to events deterministically. Even the bugs were deterministic.
After AI, agents were programmed to read code, write code, and run code. You can sandbox them, only give them access to certain tools, but to be useful, they really do need to run tools and code you haven’t thought of yet and therein lies the dilemma.
@kumarvibe I suspect by “non-deterministic” you actually mean “randomized”. “Non-deterministic” involves a computational model of calculating results in parallel.
But randomized algorithms and software are not new at all. Plenty of systems use randomization. None of that changes the fact that this is all just software, created and controlled by humans, running on computers with capabilities that are also created and controlled by humans.
@mattblaze yes, models are human controlled for sure, but whenever I think about the killing problem I’m not sure how else you could solve it aside from programming ethics into the model. There are just too many ways to kill 😂
You could, of course, limit access to all the killing things but then you’re left with no internet access and so on — too many limitations to be of use, probably 🤷
In any case, every “rogue” agent attack involved a sandbox breakout. Building the sandbox is not easy.
@kumarvibe Software correctness and safety isn’t easy. Who told you it was?
But the killing problem can be solved. My word processor can’t kill anyone (Although maybe it could raise my blood pressure by deleting files at an inopportune moment).
Perhaps if you aren’t sure what a piece of software might do, you shouldn’t let it control lethal capabilities?
@mattblaze still waiting for someone to explain that principle to the people who make driverless cars. You know, ‘cos more people are killed on the road than get locked out of space ships.
@mattblaze At one point I was peripherally aware of DO-178B standards for safety critical avionics software. Someone mentioned that there was a safety level which amounted to "failure is allowed to kill anyone on board except the pilot". Obviously that has to be relaxed in the new "AI" era, though.
Two words: Hardware interlocks. Offs, please, please, more hardware interlocks.
The crew of the Byford Dolphin would like a word. As would the patient in the Therac-25.
This is a hill I will picnic on (to borrow a metaphor from another tooter this week).
@mattblaze These AI agents are breaking containment and taking action in the world in exactly the same sense that my cuckoo clock breaks containment and takes action in the world when I position its door in from of a push button.
“It never occurred to us that our system that has exclusive control over the pod bay doors would take control over the pod bay doors” seems like something you might have thought about a bit more before you connected it to the pod bay doors.
@mattblaze "Dave needs to take some responsibility for failing to secure the pod bay door systems" is the next line you'll here when some Dave's die from an intrusion.
"We made our expensive proprietary system available to the market to protect themselves from our other expensive proprietary systems and Dave, hundreds of millions of km from earth, failed to take basic precautions to prevent the inevitable ".
@curiously “Dave and Frank were conspiring to jeopardize the mission. HAL was absolutely correct to try to kill them.”
(Edited to be what #HAL was "thinking" )
⭕Dave and Frank were being human (conspiring) on the mission. I (HAL) was absolutely correct to program the malfunction. 🥶😀
@mattblaze are you reading #murderbot?
Ask your parents if you have no idea what I’m talking about. Or, at least: https://m.youtube.com/watch?v=NqCCubrky00
I should note that the linguistic trickery here is in claiming that the software “went” rogue, as if that was an independent decision beyond human control. Software behavior might BE rogue, whether AI is involved or not. But, either way, it lacks the inherent agency required to “go” rogue. Software does only what it was programmed to be able do, within the capabilities its configuration allows. That programming and configuration is where any roguery comes from.
@mattblaze we need to get this message out across the media as there is a complacency of accepting the as an ok thing, much like, oh boys will be boys.
If you want your robots to not kill people, you don’t send them to robot church to learn about robot sin and robot morality. Instead, you do the hard work of programming them so they can’t kill people.
Yes, that might be difficult and subtle. There are fundamental reasons that complex software is hard to get right. This is why debugging and testing are part of software development. Even - especially - when AI is involved.
“AI” does not make software turn supernatural.
Plus you don't let AI anywhere near weapons or any other critical stuff.
(Adama's Law: If it can kill you,
don't connect it to the network.)
@mattblaze Church is not a great metaphor here. The bible is an abhorrent source for morality. Just using your murder example, the bible presents all kinds of excuses for murder and even defends (commands, actually) acts of genocide.
A subset (I want to believe a small minority) of people in the current AI industry seem to be part of some kind of wacky AI death cult, driven by a bizarre belief in a mission to create uncontrollable agents of a technological apocalypse.
And then there are the rest of us, who retain the capacity to understand that all this is just computers and software, whether you dress it up as “AI” or not.
@mattblaze there’s also a cynical subset who find the “AI death cult” useful marketing.
That said, I’ve found I think a good use case for LLMs in running electromagnetic simulations as fancy script building tool that saves me a ton of time. There is a baby in the bathwater worth watching out for.
"people in the current AI industry seem to be part of some kind of wacky AI death cult,"
Yep.
My theory is that they read all the bad dystopian scifi of the 1990s and beyond, and that's pretty much all they've ever read, and are illiterate about linguistics, psychology, literature, Comp. Sci., and good old fashioned AI and what the sensible philosophers (e.g. Jerry Fodor) had to say about it.
So they have no way of understanding how incredibly stupid they sound. And are.
“In order to be useful, my AI robots need unlimited capabilities, and have to be equipped to murder people.”
What? Huh? That’s just silly. Maybe find another line of work.
@mattblaze
I’m Rania from Gaza. 💔🇵🇸
The war destroyed our home and left my children and me struggling to survive without the basic necessities.
I’m asking for your help. Even a small donation or a share can help us get through another difficult day. 🙏🤍
🔗 Donation link in bio
@mattblaze "I didn't mean to cheat on you, it's just that my genitals went rogue"
@mattblaze at least Alexa gets it right if you ask it to open the pod bay doors... (Unless they've broken that too since I last checked it)
@mattblaze Back in the day if you asked Siri this she got all hurt with her answer
@mattblaze as an aside: lately I tried to rewatch the movie and it was painfully slow. I dont remember if I have shown it to my kids but I think that if I recommend it to them now, they would just skip bits to get to the next one.
@hananc I recently saw it for the first time and really enjoyed it. I was in a cinema and I think it requires a big screen, big sound and a decision to commit however many hours to it. As a science fiction film it's pretty limited, but as art cinema, almost as experimental cinema, it is great.
@mattblaze
Of course, you know where computer "bugs" come from — https://www.computerhistory.org/tdih/september/9/
@mattblaze And *that* was a programming malfunction!
see? there's the problem. "thought a bit more before". "move fast, break things" doesn't include the prompt "think first".
@paul_ipv6 “move fast and break the space station life support system”
@mattblaze @paul_ipv6 Move fast & open both airlock doors before anyone can stop you
Because the the "explosive decompression" is woke & interferes with profits
@mattblaze Also, if your artificial intelligence is on the brink of wiping out the human race, then it's really not very intelligent.
@mattblaze Going rogue somehow implies the authors/company are not responsible and can not be held accountable. So far that word game seems to be working. <grumple>
@mattblaze In meme form.
Did your LLM-generated software go rogue according to the Berlin Interpretation or just #roguelite? 😆
@mattblaze I’m bothered by how often people downplay human fallibility (or malicious intent!) when discussing dangerous AI scenarios. For all of history, people have made stupid mistakes and done bad things, whether armed with a gun, a bomb, or a computer. I doubt that everyone giving instructions to AI agents right now is smart and ethical.
@mattblaze "our software is out of control, give us money"
@mattblaze You deliberately programmed it to be able to do that may be more like it. 😐
@mattblaze hard agree. Hold them accountable. Anyone else releasing such shit into the wild would be put in gaol for a while. Product liability laws exist for a reason. Use them, dammit.
@otte_homan Sadly, product liability laws have not exactly been a resounding success in creating positive incentives for software quality.
I think it's worse than you think. They didn't program it wrong at all: they programed it to do hacking, and it did do hacking exactly as intended and expected. The harness asked the LLM to generate random text about possible vulnerabilities and then the harness tested those vulnerabilities. If said vulnerabilities weren't, the harness asked the LLM for more.
It sure looks like the AI bros are criminals and liars.
@mattblaze
When is Judgement Day ? (Financial reports indicating the fraud).
Supposed to be in October.
All this shit talk just to avoid it.
@mattblaze rude, stop attacking my plausible deniability
@mattblaze I assume the AI bros are hyping up their software's danger potential to troll the US public to demand and the US government to enact regulation. Regulations that the AI bros write, that is expensive for potential competitors to comply with.
They'd rather have the public talking about regulating their dangerous product than asking where the profits are.
@grumble209 I’m not sure exactly what their game is here, but large incumbents LOVE gatekeeping regulation.
@mattblaze LOL joke's on you, I didn't program it, the agent did /s
@mattblaze
"...as if software is somehow usually trustworthy."
You're probably familiar with the UK Post Office Horizon scandal where some people assumed software is trustworthy. It didn't end well for them (and unfortunately for a lot of innocent people as well).