It's fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You *should* worry about that - and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding.

33/

"AI" - chatbots that wake up, "set their own goals," and "spontaneously" start hacking servers - is fake. It doesn't have "a 10% chance of ending the human race." The Hugging Face hack isn't a mysterious, supernatural occurrence. It's a Python loop and a chatbot. The people responsible didn't accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.

32/

Riley had a very good way of summarizing this: "LLMs are real, AI is fake." LLMs - chatbots trained on things like CTF logs that can break into servers - are real. They're on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to *do* anything about it.

31/

The hacking tools the chatbot companies are developing stand to carry on this stupid tradition. It *is* scary, but not because chatbots are waking up. It's scary because the world's IT systems are indifferently created, poorly maintained and riddled with vulnerabilities:

https://xkcd.com/2347/

This week, I had a couple of opportunities to hash this over in public with Riley Quinn; first at a book launch in London and then on the Trashfuture podcast:

https://www.patreon.com/trashfuture/posts/what-would-do-169247456

30/

The NSA's irresponsible handling of EternalBlue ended up giving a gigantic force-multiplier to otherwise incompetent and inconsequential cyber-criminals. It's as though they found some guy under a Prius removing the catalytic converter with a Sawzall and handed him a piece of software that could shut down major American cities. That was - and is - *very bad*.

29/

Within months, the stupidest people online were shutting down some of the world's most important systems, demanding cash to return them:

https://en.wikipedia.org/wiki/EternalBlue

They shut down cities:

https://en.wikipedia.org/wiki/2019_Baltimore_ransomware_attack

They took hospitals:

https://www.bbc.com/news/technology-35584081

They seized oil pipelines:

https://en.wikipedia.org/wiki/Colonial_Pipeline_ransomware_attack

They stole the British Library, whose postmortem on the attack is one of the clearest, most informative cybersecurity documents ever written:

https://cdn.sanity.io/files/v5dwkion/production/99206a2d1e9f07b35712b78f7d75fbb09560c08d.pdf

28/

NOBUS is a terrible idea. How terrible? Well, in 2017, the NSA lost track of a Microsoft Windows vulnerability that they'd discovered and hoarded, code-named "EternalBlue." After EternalBlue found its way into the wild, some halfway competent hackers spliced it into some boring, everyday ransomware, giving that ransomware a new lease on life.

27/