Here's why security measures won't work on superintelligence

Added: 2026-05-17

[00:00:00]The authors make one last comparison to computer security. It's impossible to make computers that are fully secure, at least while still being useful. You can make computers more secure, but they'll still eventually fall to the best hackers funded by the biggest governments. As renowned security professional Bruce Schneij writes, "Modern systems have so many components and connections, some of them not even known by the systems designers, implementers, or users, that insecurities always remain." The authors give the classic example of a buffer overflow attack. A program asks for a username. Attackers type in a longer name than the program reserved space for in memory. The system by default stores the extra letters to the next location in memory. When the computer tries to use that next piece of memory, it can go haywire. If that happens when the computer is doing something important, say deciding what code to run next, the attackers can choose the right bizarre combination of characters and soon take control of the entire system. Testing usernames at random won't catch this attack. Only a few inputs work out of the hundreds of trillions of possibilities, almost all of which will just crash the computer. If the programmers could fully understand their system well enough to stop it from ever crashing or behaving unexpectedly, maybe they could stop all of these attacks.

[00:01:17]But in the real world with complex systems, an attacker that knows the system better than they do will always be able to find a vulnerability that slipped past the programs. Yowski and Sori say that the lesson here isn't that very smart AIs could hack the world's computers, although they could. It's that if you're relying on the constraints that you know enough to come up with to stop an entity that knows the game better than you, you're always doomed to lose. Constraining an unaligned super intelligence, they argue, is just like computer security.

[00:01:46]Novices think they can win. Experts know that they can only delay losing. If you want to try to keep an AI from causing trouble, you might try putting restrictions on the AI to try to make it slow down or prevent it from making itself smarter or any number of things.

[00:02:02]But the moment your restrictions get in the way of something the AI is trying to do, it will look for a way around the rules, at least by default. And no matter how hard you try, there's just no way you'll be able to catch every vulnerability the AI might exploit. This doesn't work even for computers that we control every part of, much less the real world. with all its complexity. So, if the AI is smart enough, your restrictions will probably barely slow it down.