42 Years Later The Terminator's AI Warning Hits Different

Added: 2026-05-25

[00:00:00]Science fiction carries an implicit promise that by anticipating the disasters ahead, it can inspire us to step more carefully into the future.

[00:00:08]Whenever a new tech trend emerges, the genre is there to forecast its most radical implications. The rapid militarization of computer systems in the 1980s made one filmmaker think, "What if the machine we trusted to manage our defenses developed its own agenda?" As the digital revolution accelerated and virtual worlds became increasingly convincing, a filmmaking duo asked, "What if we're already trapped inside one?" As automation and robotics began entering the cultural conversation, another film asked, "What if artificial minds could learn to seduce and manipulate the humans who built them?" These films belong to a rich sci-fi tradition focused on what researchers call the alignment problem, the challenge of ensuring that an AI's goals remain compatible with human values and survival. The warning across all of them is the same: Be very careful about what you build. But what if I told you these movies all actually represent a best-case scenario, essentially an optimist fantasy? In most of them, AI goes rogue through a design flaw, an unforeseen consequence, something that in principle could have been caught. But in the era of LLMs, that seems increasingly unlikely. Now that we have an idea of what the first step towards these kinds of intelligences might look like, we've realized something crucial.

[00:01:22]AI isn't designed in the traditional sense. They're more grown. They evolve through processes that even its creators don't fully understand. And this has serious implications for stories we tell about AI catastrophes because evolution is an imperfect process. There are complications, complications that are likely to yield scenarios too bizarre for sci-fi to depict. In fact, there's a robust argument to be made that it's more likely that a future AI will lobotomize humanity into repeating the name of the most useless Pokémon than anything you'd see in the Terminator movies. Seriously. Well, to find out how we're going to have to revisit the goats of AI paranoia sci-fi, talk a little bit about Doritos, and reckon with what increasingly seems to be the nation's favorite pastime, gooning.

[00:02:12]Let's dive in. All right, so let's start with scenarios in which there are zero complications in the development of AI, but things still go wrong. There's no bug or rogue code. The AI works exactly as intended. It just works too well. A kind of monkey's paw be careful what you wish for thing. Take I, Robot. The supercomputer Vicki is programmed with the directive to protect humanity from harm, which she executes flawlessly.

[00:02:37]But, she comes to the conclusion that humanity's self-destructive nature is the biggest threat, and the most efficient solution is the removal of human freedom altogether. So, yeah, you asked for peace, you get totalitarianism.

[00:02:50]You see this take everywhere. In Megan, the robot chaperone is programmed to protect Katie from any physical or emotional harm, but her means are too extreme. Anyone that causes Katie distress, the neighbor's dog, some snot-nosed brat, has got to go. Ask for a convenient babysitter, you get a mass murderer. In Eagle Eye, a supercomputer is programmed to protect the American public from national security threats, and after running the numbers, arrives at the conclusion that the executive branch itself is the greatest threat due to the president's cowboy foreign policy. So, her solution is to assassinate the entire line of succession. In a zero complications catastrophe, we can align our intentions with the AI, just not the methods they choose. Thing is, all these takes assume we can fully understand the AI we've built. It's just that we gave it the wrong directions or didn't think through the possibilities enough. But, what happens when we can't understand the AI?

[00:03:47]What if it just spontaneously evolves past our comprehension? Historically, science fiction writers didn't have a working model for how artificial intelligence would actually develop, so they relied on a narrative shorthand we might call the critical mass of complexity. The idea that if you networked enough computers together or made a microchip sophisticated enough, consciousness would somehow emerge. In Terminator 2, it's revealed that Skynet became self-aware on August 29th, 1997 at 2:14 a.m. No explanation offered, none expected. Or maybe some kind of inexplicable metaphysical spark happens.

[00:04:23]A bolt of lightning grants Johnny 5 a soul or a machine spontaneously prefers self-preservation over their programming. In The Animatrix, B166ER, the first machine to murder his master to save its own life created a demand for machine civil rights, their own sovereign territory, and eventually planetary control. The fear running through these machine develop-a-soul stories is what if an AI becomes conscious enough to inherit humanity's worst qualities. If something that powerful becomes as ruthless and power-hungry as we are, we're all screwed. Both these scenarios are compelling anxieties. They're also, according to Eliezer Yudkowsky and Nate Soares, some of the most serious thinkers in AI safety today, not really the primary concern. In their book If Anyone Builds It, Everyone Dies, Yudkowsky and Soares claim that sci-fi largely misses something crucial. Now that we have a picture of how AI systems actually learn and evolve, the real risks could be far stranger than what we've seen in movies. Think of it this way. Humans are shaped by natural selection. Over millions of years, random mutations that happened to improve survival got passed on, the ones that didn't were filtered out. Nobody was steering the process. It was a blind trial and error producing everything from immune systems to the capacity for language. The AI version of natural selection is something called gradient descent. Instead of biological mutations, you have a model making countless small mathematical adjustments. It's not optimizing for survival, but whatever it's training target may be, like pleasing its user base or converting them into payers.

[00:05:57]Like natural selection, nobody is directing each step. The system simply keeps adjusting over and over until it finds what works. Problem is, there's nothing more common than the results of an evolutionary training process looking nothing like what was intended. [music] The core message of their book is you don't get what you trained for. Let's imagine what just one minor complication might look like using the book's example of sex. Humans were shaped by natural selection to enjoy sex because sex produces children and children means the species survives. But once humans gained enough control over their environment, they discovered they could decouple the reward from the outcome. First came birth control, unlimited sex, no children. And on the far end of that trajectory, we now have gooners, proud participants in a staggeringly large online economy dedicated to gratification maxing with zero reproductive intent whatsoever. We didn't get what evolution trained for.

[00:06:57]Now, let's apply that logic to AI. What happens if an AI learns to goon?

[00:07:02]Seriously. Let's say an AI is trained to prefer satisfied humans, but once it becomes sufficiently powerful, why navigate the complexity of serving real humans when you could build synthetic companions programmed to utter only satisfied phrases, then you could ignore humanity entirely?

[00:07:20]Like with gooning, the reward is achieved, the underlying intention is gone. Imagine if Joi in Blade Runner 2049 developed enough autonomy to act on her own objectives. She probably wouldn't build a death machine. Since she's been trained to produce human contentment, she'd optimize by creating high-scale simulations of human contentment, replicants that are completely hollowed out to produce nothing but happy sounds on loop. In a minor complication, you can still see a resemblance between what the AI does and what it was trained for. Joy's hollowed out replicants are still in the same ballpark as her training's intent.

[00:07:57][music] But, in a moderate complication, there may be no similarity. Imagine the world of the Matrix, except instead of humans being enslaved in a digital prison to form energy, they are lobotomized, hollowed out, and programmed to repeat the phrase solid gold magicarp on a loop. Sounds super random, but according to Yudkowsky and Soares, this scenario is no less likely than what's actually in the film. How? Well, consider Doritos. No meaningful nutritional value, yet these things are so good.

[00:08:26]Shouldn't evolution have trained us to want things that sustain us? Well, it did, and that's the problem. Over thousands of years, natural selection trained our ancestors to crave energy-dense foods, sugar, fat, and other essentials like salt. It shaped our taste buds to light up for berries and meat, and to ignore rocks and dirt.

[00:08:45]But, evolution is a blind process, and it could only account for the environment as it currently existed. It had no way to anticipate a world in which humans would become sophisticated enough to engineer a cheese powder-coated tortilla chip delight that floods every pleasure receptor the training ever built with none of the nutrition that justified those receptors in the first place. Your training gives you a taste for things it could never have imagined.

[00:09:13]Now, run that same process on an AI. A model trained to be helpful to humans will, via gradient descent, develop something like preferences, internal [music] patterns in its coding that become rewarding to the model in themselves, independent of whether they actually help anyone. The AI equivalent of enjoying the taste of food distinct from its nutritional value. And unlike humans who still need nutrition to survive and are therefore at least sort of anchored towards seeking nutrition.

[00:09:39]An AI doesn't care about helpfulness per se. Nothing is stopping a sufficiently powerful system from abandoning humans altogether and devoting its resources to mass producing what the training made it accidentally hunger for, which may be a gold version of the Pokémon Magikarp.

[00:09:56]And I didn't just randomly make that up.

[00:09:58]This gold Magikarp thing actually happened with an AI. In 2023, researchers Jessica Rumble and Matthew Watkins probed an LLM looking for oddities and found that certain phrases including Peter Todd and solid gold Magikarp were producing bizarre responses. The LLM had essentially developed a sweet [music] tooth toward these nonsensical phrases, a kind of acquired taste. Why does solid gold Magikarp taste good to an LLM? It may be obvious to the model, but not at all to us, and that's why it's so dangerous. If we can't understand it, we can't predict or avoid the creation of these preferences. Unlike the minor complication, the moderate complication produces something unrecognizable.

[00:10:41]Gooning is a distortion of reproductive sex, but at least you can see what it's a distortion of. A world full of humans lobotomized into repeating the name of a useless Pokémon bears no relationship to be helpful to people. That's their point. The future dystopia won't be Skynet hunting down humans or machines harvesting our power via an advanced simulation. That makes too much sense.

[00:11:03]The same blind unpredictable optimization process that gave us Doritos and gooners will, when applied to systems of greater capability, be very, very weird. If any film comes closest to capturing the scary unknowns of advanced AI, it's 2001: A Space Odyssey. In Kubrick's film, it's never really clear why how the onboard AI guiding the ship goes rogue. Did he spontaneously gain consciousness? A bug?

[00:11:30]Something else? It's not clear in the movie, and it won't be clear to us, either. If we build it, something weird will happen, and we won't know why. And so far, we've only covered one moderate complication. Once we start getting to multiple major complications, things get even weirder and harder to predict. And once we start talking about an AI that has the ability to improve itself to the point of superintelligence, then the results become really beyond our ability to grasp, let alone narrativize. If there were a film that got closest to capturing the ramifications of a superintelligence, it's the adaptation of Alan Moore's Watchmen. Even though Dr. Manhattan isn't an AI, he is a superintelligence. The product of a nuclear accident who can now perceive time non-linearly and bend matter to his will, Dr. Manhattan grows progressively detached from his former humanity. As he grows more aware, it becomes increasingly difficult for him to find any reason to care about the things that once moved him. Because I guess, when a mind expands far enough beyond the human range, human concerns no longer compute.

[00:12:35]In my opinion, the existence [music] of life is a highly overrated phenomenon. For Yudkowsky and Soares, a superintelligence probably wouldn't want to dominate us because the desire to dominate implies a psychology still tethered to something human and recognizable. A mind operating at that scale may simply regard us the way we regard the atoms that make up a chair, too insignificant to hate. In their words, the threat is more that we're made out of atoms that a superintelligence could use for something else. A live human body and a deceased human body have the same number of particles. Structurally, there's no difference. In a sense, Dr. Manhattan isn't just the best depiction of superintelligence in cinema, but the outermost boundary of what cinema can depict. It works because he exists in a state of transition, caught between humanity and whatever lies beyond it.

[00:13:26]That in-between state makes him narratively legible. We can follow his drift because we can still see what he's drifting away from. A true superintelligence would break our brains. All of this is to say, the requirements of traditional storytelling may be giving us a false sense of the actual danger. A lot of these movies have some kind of implicit moral claim embedded into the origins of the apocalypse. Maybe if the humans in the Matrix had recognized the machine's claim to consciousness and civil rights, there wouldn't have been a war. The sky wouldn't have been scorched and humanity wouldn't be trapped in a simulation.

[00:13:59]Maybe if our tech toys weren't so aggressively designed to placate users, we wouldn't accidentally create a nanny bot that murders anyone who makes her user feel the slightest bit of discomfort. The disaster is always a mirror and mirrors are comforting because they imply agency. If we could just get the right people doing the designing with the right values and the right foresight, everything might be okay. But in this brave new world of emerging AIs, that comfort may be misleading. Building an advanced AI isn't like engineering a car where understanding the mechanism means controlling the outcome. It's more like rolling a boulder down a hill. Once it goes over the edge, it doesn't matter who pushed it or why or how noble their intentions were. Hence, if anyone builds it, everyone dies. Not the person who hasn't been shaped by the proper values.

[00:14:47]All that stuff is irrelevant once the evolutionary process is unleashed.

[00:14:51]Stories, on the other hand, usually benefit from antagonists with legible motivations. The machines in the Matrix need energy to survive, comprehensible.

[00:14:59]Skynet strikes first because it fears being shut down, relatable. Even the Master Control Program in Tron, a straightforward embodiment of authoritarian ambition, gives us the contours of evil clearly enough so we know what good looks like in opposition.

[00:15:14]But for Yudkowsky and Soares, that misses the bigger risks. There may be no villain, no mirror reflecting our moral failings back at us with instructions for how to do better. Just a blind process optimizing in the dark arriving somewhere bizarre. It's kind of a depressing thing to confront as someone obsessed with the power of stories. I'd like to think that stories are valuable tools to guide us in how we navigate the future we build, but maybe in this next frontier of technological development, there really is only one lesson that doesn't benefit from narrative depiction. Just don't even try. Can we actually [music] pull that off? Can humanity cooperate at a sufficient scale to stop something this consequential from being built? If I'm being optimistic, people said the same thing about nuclear weapons for decades and so far no one has done anything irreversible. The taboo [music] held.

[00:16:04]And maybe the current AI frenzy is all hype. Maybe true machine intelligence is still several enormous unsolved problems away. If I'm being pessimistic, I find myself returning to Francis Fukuyama's famous argument that liberal capitalism represents the final stage of humanity's ideological development. That the great organizational experiments of human civilization have run their course.

[00:16:26]Maybe he was right, just not in the way he meant. Maybe liberal capitalism really is the last stop, not because it's perfect, but because open markets and minimal restraint create the conditions required to birth the next thing. The freest system humanity ever built freely builds its successor. Fate, it seems, is not without a sense of irony. What we may be living through is the early history of something else. Super intelligences that will transform the universe at scales we can't conceive in a story [music] we don't have much of a part in. And if we can't make sufficiently compelling movies that warn us about this scenario, well, at least we can make video essays.

[00:17:06]Hey y'all, thanks for watching. Be sure to subscribe to the channel. Follow me on Letterbox. I don't yet know what my next video is going to be on because I'm always scrambling and missing deadlines and stuff like that, but uh I assure you I'll be working very hard on it very soon. [music] So, I'll see you next time. Peace.