How Quickly Does AI Become Smarter Than Humanity?

[00:00:00]What is artificial general intelligence and how close are we to getting there?

[00:00:05]And how do we define what comes after that which is often called artificial super intelligence? And why might the gap between those two be dangerous for us humans? When machines become smarter than us, what roles are left for humanity? Should we think of AI as software or as something we're raising like a child? Will tomorrow's AI destroy us or ignore us or protect us? Today we talk with computer scientist and AI researcher Ben Girtzel who [music] over two decades ago introduced the term artificial general intelligence. [music] Welcome to Inner Cosmos with me, David Eagleman. I'm a neuroscientist and author at Stanford. And in these episodes, we sail deeply into our three-pound universe to understand some of the most surprising aspects of our lives.

[00:00:59][music] [music] One of the many remarkable things about the brain is that it spends a lot of its time confronting situations that it's never encountered before. So, every day you're walking into conversations you've never had. You solve problems that you've never seen. You recognize something like a face or a building from an unfamiliar angle. Sometimes you might invent a joke that no one's ever said before. Or you step onto an icy sidewalk and you instantly discover a new way of balancing. So life is full of novelty and you're never explicitly taught how to deal with every situation you encounter. Your brain wasn't programmed when you were a kid with a lookup table of every possible circumstance. Instead, as I discuss often here, brains build internal models of the world. And those models are used to improvise when reality throws something unexpected your way. There are a lot of competing definitions and ideas about what intelligence is. But this ability to creatively improvise and extrapolate to new approaches, this is likely to be one of the defining features of intelligence. Now, for decades, what we called artificial intelligence didn't really look like that at all. What we saw instead is that computers excelled at really narrow tasks. They could beat us at chess or recommend movies or recognize faces or translate languages.

[00:02:44]Each success was super impressive in its own domain. But every system was essentially confined to the job that it had been built for. Then something extraordinary happened. large language models arrived and suddenly we found ourselves conversing with machines that could write essays and compose poetry and debug software and generate images and explain quant mechanics and carry on conversations about essentially any subject you could think of. Now speaking as a scientist who's been in this field for decades, the progress has been breathtaking. We have all been witnessing this. Every few months we see the capabilities leap forward again and every week we see some announcement that would have sounded like science fiction a few years ago. But even with all this genuine excitement, there is a key question for us. Are these AI systems actually intelligent?

[00:03:45]Or have we simply become really good at building machines that absorb vast amounts of information to predict what comes next? This is a really important distinction when we're asking about intelligence.

[00:03:58]So, let's do a consciousness razor on this. Imagine that we have a mutual friend named Fred. And Fred memorizes every textbook ever written. And he memorizes every novel and every scientific paper and every movie script and every recipe and every legal case and every conversation posted online. He reads it all and he remembers all of it.

[00:04:23]Now, Fred would certainly appear absolutely brilliant in conversation.

[00:04:28]You would want him in any dinner party.

[00:04:30]He could answer questions on almost any topic. But is that everything that we mean by intelligence or might something still be missing? This question of course reaches beyond computer science because it forces us to ask what our own brains are doing. When you get struck by some insight for a new way of doing something, where does that come from in the brain? When a scientist invents a new theory or a musician creates a new style or when a child discovers a solution that no adult had suggested to her, what kind of computation is taking place inside the skull? What this all points to is the idea that what we think of as intelligence isn't only about how many answers you've already socked away, but instead it's about what happens when none of your previous answers [music] work. Today's guest has spent decades on this topic. Ben Girtzil is a computer scientist and an AI researcher and an entrepreneur and one of the people responsible for popularizing the term artificial general intelligence or AGI.

[00:05:42]Long before the recent explosion of AI, he was asking what it would actually mean for a machine to possess general intelligence instead of isolated skills.

[00:05:52]He's one of the field's most provocative thinkers. And in our conversation, we're going to explore what separates AI from AGI. We'll ask whether language models are enough or whether something fundamentally different has to get invented. [music] We're going to examine what happens after human level machine intelligence arrives [music] and why the transition to super intelligence may happen much faster than many people expect. and what all of this could mean for society and education and work and [music] our sense of purpose. A lot of the things that we're going to be talking about feel futuristic, but boy, they are becoming present tense quickly.

[00:06:33]Here is my conversation with Ben Girtzel. [music] Okay, so Ben, everybody's talking about AI. You and your colleagues in 2005 introduced the term AGI.

[00:06:48]tell us what AGI is.

[00:06:50]>> Sure. So, AGI, as we intended it when we titled that 2005 edited volume, artificial general intelligence. I mean, AGI referred to AIS that can generalize beyond their experience to a, you know, quite significant degree. In more technical terms, you would say they can learn out of distribution. like there's been some probability distribution of knowledge in their life history and then the world throws something that's unpredictable by that probability distribution that AGI can pivot and and deal with that with some measure of what you might call imagination. And the important point to this relative to the current commercial AI scene is it's not just about the AI being able to do a lot of stuff because of course being able to generalize is one way that you can become able to do a lot of stuff. But as we've seen recently, being trained on the entire internet is another way that you can become able to do a lot of stuff even if your ability to generalize is only modest. So what do you see as the difference between AGI and modern AI? It sounds to me like it's the creativity piece. It's the outer distribution piece.

[00:08:08]>> Yeah. I mean there was a Microsoft paper a few years ago called Sparks of AGI which was arguing that LLMs show little indications of the ability to generalize out of distribution. And I think that's true. I mean I think LLMs are like non-trivial progress toward AGI. yet they seem to be sort of plateauing in their ability to generalize >> out of distribution while increasing in capability in in [music] many other ways, right? And so I I I I think that [music] transformer neural nets, which is the particular AI architecture underlying LLMs like Claude and OpenAI and whatnot, like I think these are probably not capable of like leaps of creative imagination such as as the human brain is. But what's subtle is >> they can generalize a fair bit beyond their training distribution. They're not totally in inert in that way. Now one thing that I have found interesting is that human creativity is in some sense a remixing of everything that we've pulled in meaning you know you bend break blend all the information that's come in >> and so there's an open question I think about whether we are actually ever operating out of distribution. So an example I've used before is in in the use of AI to model music and because I'm a weekend musician so I mean it's natural domain for me and I've worked with AI models for generating music u Sonos I've trained some of my own models and what you find there is you can generate like a 12 bar blues song that's quite good right now I mean if you look at like Led Zeppelin Jimmy Page Since I've Been Loving That was a blues song that was actually original, right? Which Jimmy Paige [music] always said he was proud of that song because it's so hard to do something new in the blues, right?

[00:10:09]>> So many things have been done before and there's a certain structure you have [music] to follow. Now, I haven't seen any AI model come up with a blues song of that level of originality, but they can spit out 12 bar blues songs very competently. But the thing you could think about is if you took a deep neural net style, a transformer neural net style model of music and trained it on all music up to the year 1900. So it's the only music up to 1900, you know, classical music, West African drumming, balan, gamalon, whatever, but nothing after 1900. I would submit that neural model will never invent neocclassical metal. It will never invent hip-hop. but will never even invent, you know, Duke Ellington style jazz, right? It will it will recombine things on a shallower level. So if you if you ask it to put West African drumming and western classical music together, like you'll get some funky Beethoven to a poly rhythm or something. And so in a sense you can say yes jazz was a combination of western classical music theory with west African rhythms but it was a combination at a different representational level than transformer neural nets are are representing things. So yes, we do combine and munch things together, but we combine and munchge them together based on abstracted representations of of what they are and based on what they mean to us as embodied creatures. Like we don't we don't just combine together the surface patterns, right? And I mean and that that is I guess kind of a subtle difference, but it's it's a quite real difference, right? And you you can see that if you teach kids too, like are kids just like combining stuff that they learned in class or or or are they engaging with the material in a real way, which doesn't mean what you're doing is like radical novelty out of the blue with no resemblance to anything you ever learned. It means you're leveraging what you learned based on a deep understanding of of what you learned, which LLMs are not really doing, but they're still doing amazing stuff in in their own way. And so, the way that you see the field now is LLMs by themselves are not going to get us to a level of AGI. Now, we know that LLMs of course are getting combined with all sorts of other tech, but but even with that right now, do you feel like, hey, we're clearly on the road to AGI in that direction, or do you feel like other efforts will get us there?

[00:12:41]>> I think it's somewhere between the two.

[00:12:45]So, one approach to making an AGI in the modern world would be take an LLM and just add a whole bunch of other tooling around it, right? like at a formal verifier to check if the math worked at add a simulation engine to give a spatial world model but then the LLM is sort of the hub right and another approach would be a sort of hybrid architecture where an LLM is one component and you have other components that are equally or maybe more important like a world modeling engine is symbolic reasoner a different kind of recurrent neural net that's making simple models of the world whatever but an LLM is still playing a significant role. And then then there's people pursuing utterly different approaches. They're just like, well, LM is a nice data modeler, just like a calculator, you know, with a nice number cruncher, but >> general intelligence is is something different. And it it might be that you could make different kinds of AGI minds in all these different ways. My my own sense is that the most practical approach is the hybrid approach where an LLM is contributing something but isn't necessarily the centerpiece that's sort of coordinating the mind's overall activity. And this is basically because LLMs don't [snorts] understand who and what they are, right? like you, which is almost weird to have to say, but like [snorts] a human baby starts to understand who and what they are before they can do advanced mahath and and and write essays and and and uh you know bring down the NSA's computer systems and and all that. And the advantage of knowing who and what you are and what world you're in and who you're interacting with before you learn advanced stuff is you then represent the advanced stuff in terms of who and what you are. Right? Like our our self understanding our understanding of others guides our knowledge representation about anything else.

[00:14:44]Right? And LLM's LLMs aren't doing that.

[00:14:47]Right? I would throw in two things there which is what's interesting as a result of that is that we pursue things that are relevant to us and and mean things to us that might be bad for AI in the sense of what we love about it is that we can ask it about anything and it knows and it cares and if I say hey crunch this spreadsheet it says I'm happy to crunch that spreadsheet. It'll be interesting to see if eventually we have AIs that know themselves and have personalities. If they say, you know what, that's so boring. I don't want to do that. Yeah, that certainly will happen. And I think one one thing that AGIS will do is operate narrow AI tools, right? Like you you don't necessarily need an AGI to drive your car, right?

[00:15:31]What you need is a specialized car driving engine that is never distracted from from driving your car. And then maybe like if if some weird genetically engineered animal crosses the road, it reverts to the AGI because it's like look this is out of my out of my training distribution like what what what what is this elephant w the beast dragon thing, right? But yeah, you don't you don't need an AGI to produce burgers in in a McDonald's either, right? On the other hand, having an AGI might be the best way to design all these specialized intelligences in in an optimal way.

[00:16:07]>> So, this leads to the second thing I was going to say, which is that this is what the human brain is. It is a the way I described it in 2011 in my book, Incognito, is that it's a team of rivals. You've got all these different neural networks, all these different drives, and they often compete. They're rivaling. And this is exactly what you're describing.

[00:16:26]>> They're rivaling and they're cooperating, right? Right. Yes.

[00:16:29]>> What happens? It's like a parliamentary vote where you've got all these different political parties who rival, but you know, eventually the parliament tips one way or the other. You get a vote and people agree on something and the ship of state moves in that direction. That's what happens in the brain.

[00:16:43]>> Well, you would be you'd be familiar with Steven Mthan's work, right? So he had so he he's a sort of a cognitive anthropologist but he made the argument which is debated that in the evolution of of of the brain one of the things that came together when humanity evolved was that you had networks in the brain dealing with tool use and networks of the brain dealing with communication.

[00:17:07]And he argues that humanity is the only species where the tool use subnet networks and the communication subn networks started to cross-communicate a heck of a lot. And if you if you look at crows or dolphins or chimps, various other smart animals, they do communicate and they do use tools, but they don't sort of form a community around using tools and pass on lore about tool use generation after generation so that it accumulates. And he he tries to trace this anthropological uniqueness of humans to some uniqueness in in sort of cross talk between the tool use and linguistic communication portions of the human brain which is under discussion and debate in the cognitive neuroscience world. But if you if you look in AI, it's interesting because the way LLMs use tools for doing practical physical stuff is not like that, right? It's more like you have this linguistic engine that just invokes a program compiler or something, right? Like you you don't have the tight integration between tool use and linguistic understanding that that methan proposes happen happens in in the human brain, >> right? Okay, good. So, this all brings us back to the question of what can we see in the human brain and what does this tell us about how we're going to build AGI? Now, here's the thing. In 2005, you came out with this compilation called artificial general intelligence.

[00:18:34]But the same year, Ray Kerszswwell wrote a book uh the singularity near.

[00:18:39]>> Yeah.

[00:18:39]>> We're waiting for the singularity is here now, right?

[00:18:42]>> Yeah. Yeah. That's right. And so, so he predicted 2029 is when we would hit the singular. So, of tell us what the singularity is and um and whether you think 2029 is a good uh prediction. So what Rey predicted in that book which was very similar to what he predicted in his book The Age of Spiritual Machines from before and >> honestly similar to what I read when I was like 8 years old in a book called The Prometheus Project by Gerald Fineberg. Yeah. Yeah. He wrote that in ' 68 when I was two. I read it when I was like >> I read that when I was eight or nine years old in the town library in suburban New Jersey. He said like machines smarter than us, nano assemblers and super longevity pills will become available within decades.

[00:19:26]The question is should we use it for the common consciousness expansion of humanity or for rampant mindless consumerism and who decides? And he proposed that the UN should put it to use computers to put to a global vote around the world to decide to what use to put these technologies.

[00:19:41]>> Oh wow. I >> I later found out Valentin Turchin who was the founder of Russian AI who I became friends with later. He'd written a similar book called the phenomenon of science in in the early 70s which had been big in in Soviet Union. Right? So all these ideas had been there. Rey in his book singularity in 2005.

[00:20:01]What he did he contributed some high quality curve plotting. Right? So he was not just like we can see the trends we can see this stuff is physically possible. We can see it's coming. He was like let's plot year on year Mors law.

[00:20:15]Let's plot year on year the accuracy of brain scanning. Let's plot year on year the size of the smallest engine anyone's made. Right.

[00:20:21]>> Like an accountant.

[00:20:22]>> Yeah. Yeah. Right. So actually the accountancy metaphor is funny because I remember when Rey was giving talks on this in that time frame and he's very clear. He's a good presenter but he's very dry, right? I mean he had a nice suit on. He he was sort of balding even then. and he was just pointing out step by step what what what the numbers are just just like your financial adviser might might do when pointing out relative interest rates. So yeah, it was very like accounting style rundown of like just the numbers, ma'am. Here's the here here's what the curves are doing.

[00:20:56]Indeed, if you plot these curves, you concluded that like a a powerful laptop or a server would have roughly human brain capability by 2029 according to Ray's very fudgy calculations of number of neurons and what processing they're doing. And then by 2045, he estimated, [music] you know, an average server or juiced up laptop would have on it the processing power of all of human society. And so what he he said human level AGI he called it strong AI in that book in 2005 2029 singularity meaning AI is massively superhuman 2045 >> and is that the same as what we refer to as ASI artificial super intelligence >> pretty much. Yeah I mean ASI is less precisely defined than AGI. I mean AGI we have some serious attempts at like a mathematical definition of what does it mean to radically generalize be beyond beyond your experience. Of course the way the term is used now by tech CEOs is often not in accordance with the math theory but there is there is actually a hardcore math theory of what is AGI like it makes sense in terms of learning theory. ASI just roughly refers to a machine [music] with massively greater general intelligence than us, right?

[00:22:19]Like the the super is relative to us.

[00:22:22]Now, you might think the super intelligence should come along with some obvious bells and whistles like if it if it has root access to its brain and it's super intelligent, it should be able to self-modify and self self-improve at will, right?

[00:22:36]>> Rewrite its own software and hardware.

[00:22:38]>> Yes. Although although what we're seeing now, which is interesting, we're seeing even now pre-agi or prehuman level AGI, we already have a bunch of software that's productively selfmodifying and rewriting its own software to get smarter and smarter. So it seems recursive self-improvement in fact is coming before human level AGI rather rather than after. But going back to Kerszswall though, I think yes, given where we are now in the AI field, both the commercial space and and the AGI research field, 2029 seems a lot clever than it did to most people in 2005. Like, could we get human level AGI next year? Yeah, I think so.

[00:23:22]Could it be 2033 or something? Sure. Is it going to be 2050? I I really doubt it. So I mean it looks like Rey's estimate, you know, within the confidence intervals that the chaos of the human world brings you, it looks like Rey's estimate is not too bad. Now, but the second part of his estimate is looking less and less >> realistic to many of us in the field.

[00:23:46]I'm looking very pessimistic.

[00:23:48]>> And that's the second part is the the length of time between AGI to ASI super intelligence. Yeah.

[00:23:54]>> Yeah. I think if you look at what modern frontier models can do in terms of math and programming and engineering, it really seems like once you make the breakthrough to a human level general intelligence, it can create and imagine as well as a smart human hooking that up just to like today's LLMs that should bring you to a super intelligence without a 16-year gap between human level AGI and super intelligence. And the way I would frame that is Ray's exponential curve plotting. It's following an exponent based on humans doing all the inventing.

[00:24:29]And once the AIs are doing the inventing, it's not that things occur magically in two seconds or something, but the the exponent of your exponential curve, maybe a larger exponent when it's the AI doing the doing the inventing. So I I think from AGI to super intelligence is a few years, not 16 years. And I think it could even be a few months if the AGI is irresponsible. But I mean, if the AGI is built responsibly, it will want to gate its own self uplift to some degree to avoid like rewriting its own hard drive or something.

[00:25:05][laughter] >> Okay. But once we reach ASI, then we're in the position of, let's say, you know, the squirrels in my backyard who I watch every morning and and they're so clever and physically adept, but they can't possibly understand.

[00:25:21]>> We're well aligned here because the the example I usually use is the squirrels in Yellowstone Park.

[00:25:25]>> Oh, great. So yeah, our Philip Kadic robot which my friend David Hansen built even before I got involved with Hansen Robotics and we we did the Sophia robot together which became the first robot citizen but >> before Sophia we had the Philip K. Dick robot it was a simulation of the science fiction writer Philip K. Dick. And Phil Dick gave a a media interview with some journalists one time where he's just like, "Don't worry. After the singularity, I I will uh keep you safe and warm in my people's zoo where you'll be watched over by the by [laughter] the machines of loving grace to protect you from harm." Right? And he was being funny because he had a this was prelims, but he had a statistical language model trained on [snorts] Philip Kadik's writings and PKD was a funny guy. But I like to divert from that, say, "No, it doesn't have to be a zoo." Think about the squirrels in Yellowstone Park. Like, we made a nice safe space for them. They have a lot of room. They run around. We don't try to micromanage their lives, like tell which squirrel can mate with which other squirrel. We let them even fight over holes in different trees. If there was a plague there, we might come in and and give them some uh some medicine that they don't know how to invent, right? And if they run out of the park, they may get flattened by a semi on the highway. But that that might be an okay model of what human life is is like once you have ASI because indeed an ASI could create narrow AIs and human level general intelligence. They'll be very helpful to people. And you would presume if the ASI is benevolently disposed toward people, which is not guaranteed, but I think is one very viable outcome. I mean, it will not want to micromanage all of our lives because then we can't live a fulfilled life. And in in the same way that many of us now realize locking animals in zoos is kind of nasty, right? And and you you want to leave them to to to romp around in in in the forest. But yes to the extent that people are attached to being the lords of the planet once there's an ASI people will not be the lords of the planet in a fundamental sense they can be the lords of the planet only within certain constraints and then you know humans could have the opportunity to plug in the brain chip and become part of the ASI but then then you're not exactly a human in the traditional sense.

[00:27:55]anymore, right? And I think young people who grow up in this environment will be very comfortable with all this, but uh folks who were used to the previous order of things may have some adjustment to do.

[00:28:08]>> Yeah, quite right. I know that one of the things that you've thought about is this window in between AGI and as specifically what I've heard you say is your concerns about geopolitics.

[00:28:22]Meaning, how do we make sure that when we get to ASI, we have a a good system that cares about us like the squirrels in Yellowstone as opposed to wanting to just dominate?

[00:28:33]>> Yeah. I I I think I think once you get to a super intelligence, then there will be many nuances we're not seeing now, of course, but to a crude approximation, once you get to a super intelligence, it's going to be kind of binary. Like if if the super intelligence is compassionately disposed toward us, it won't take a lot of energetic or cognitive labor for the super intelligence to make a nice abundant life for its creators, right? And there's some subtlety there because I mean like we care about species to some extent, but we don't care about every instance of that species, right?

[00:29:11]Like we we don't want to extinct a species, but we're all right squashing a lot of animals when we build a housing development, right?

[00:29:18]>> Or eating pigs and cows and so on. Yeah.

[00:29:20]>> Yeah. Absolutely. Yeah. Yeah. I look forward to lab grown meat. Then then then this is a question I was wondering recently like uh will Muslims be able to eat lab grown pork? Right. So we're going to face we're going to face a lot of a lot of dilemmas. So super intelligence though won't be much work to maintain a happy life for people if it wants to. On the other hand, Elzar Yudkowski had a very nice way of phrasing things like the AI does not love you. The AI does not hate you, but it can use your atoms for something else. So I mean that's certainly viable.

[00:30:01]like we we we used a lot of the planet for for something else and then fairly late in the development of civilization we decided hold on we don't actually want to kill off all the other plants and animals right so the ag the ASI wouldn't have to hate humanity or be vengeful against us or something it could I thought likeing the human species isn't the worst thing that can happen what if you had a a rampant Christian AGI that uploaded us all into a simulation of the Christian hell for billions of years or something Oh, so I mean there's a lot of possibilities if you if you open your mind, but a more likely possibility to me is, you know, and ASI is just as indifferent to humans as many humans are to the other other species on the planet that we think are not as good as us, right? So I think once you get to ASI, you can have a quasi utopia.

[00:30:50]Not necessarily perfect, like the ASI won't make the girl you're in love with love you back or something, right? I mean, humanity is still humanity, but you could have a quasi utopia or various forms of dystopia. So, one topic is [clears throat] how to bias the odds toward the quasi utopia. But another thing is in the interim between the first barely human level AGI and the super intelligence that will bring utopia or dystopia, right? In that interval, what the hell happens on on the planet? Right? And one one question I've posed in this regard is like who gives universal basic income in the Congo, right? I mean you feel like in the US, in China, in Western Europe, Japan, Australia, South Korea, you know, the relatively wealthy world and China is not that wealthy, but they have a very rational government that cares a lot about their people, right? I mean in these parts of the world [snorts] if AI takes a lot of the jobs somehow or another free money will be given to people. And of course that's a grandiose approximation. US politics is is is crazy. But I ultimately have faith that as stupid as the US voters can be, if they have candidate A who will give them free money and candidate B who will leave them starving out in the street, at least a slight majority may vote for the free money rather than for homelessness and starvation. Right? But in the developing world, the option isn't there because the money isn't there to give universal basic income to everyone. Now there's different scenarios you could look at. Like I spent a lot of time in Ethiopia. My my son-in-law is Ethiopian and I've had an AI development office there since 2013.

[00:32:35]We have 50 developers in Sababa. So I've been back and forth to Ethiopia a lot lot of subsistence farming probably like 90% of the country is subsistence farming. So [snorts] in the end people aren't going to starve there. There was famine a long time ago but that was before you had a modern road system and so on. On the other hand, all the people living in the city do not want to go back to their grandpa's farm and live off the land with no way to pay their phone bill or get or get penicellin.

[00:33:03]Right? So, and Brazil where where I was born, which I'm also quite familiar with, I mean, in Brazil, >> arguably maybe not rich enough to give everyone decent UBI, but also too far beyond subsistence farming for everyone to have a family farm to go back to, right? So, [clears throat] then like what actually happens, right? So I mean you could see a scenario where the rich countries are everyone staying home playing video games blowing off what the robot factories produce. 10,000 super rich people are getting more super rich and everyone in the developing world has to scrabble around in the garbage and go back to subsistence farming.

[00:33:41]>> I hold a slightly different view on this which is just >> seems like for for a a cyber thriller novel of some sort.

[00:33:46]>> That's right. Except I don't buy it only because what happens every time there's new technology let's say the cell phone everyone worries only the rich people have cell phones but you know that's spread around the world so rapidly much faster than the computer revolution >> yes but but a few years and everybody in the world has cell phones now and these are supercomputers I I just think as things get uh developed and commoditized and by the way if we have ASI producing things for us and building mach >> once you have ASI I agree the the question is the several years of transition which is nothing in the historical scope of our species but a lot if you have no food or if you're sick and have no medicine right so I mean let's say my own team develops a human level AGI next year right initially it's going to be running on a bunch of server farms in our case it'll be running on a bunch of server farms scattered in in in different countries but it's going to be an expensive thing to run initially everything we're doing is free open source decentralized. So the frontier labs will then copy what we're doing and then then they may be you know a year behind us then eventually catch up and be be six months behind us just like you see with China catching up with with with with western models. Now in this period what would happen is like we will partner with enterprises in different industries to automate what they're doing. like you okay partner with McDonald's to automate the drive-thru and the and and and the burger machines like you partner with accounting firms to automate accounting processes. Now outside [snorts] Adosaba there's a large shoe factory run by by a Chinese company. You have a bunch of Ethiopian guys kind of nailing shoes together. Now that can be automated now.

[00:35:31]It's just running [snorts] the robot factory now is more expensive than hiring Chinese but not more expensive than hiring Africans. So presume an early stage AGI creates instrumentation that automates that shoe factory so that Chinese company has no need to hire Ethiopian guys to put nails in the bottom of shoes. You have that same thing happening over and over again. Now let's say that happens 2028, right? then there's some time lag before that same technology is deployed in a way that provides any sustenance to those Ethiopians guys who were fired from the shoe factory. Right? And then all those guys [snorts] either they live in the city with no money or they have to go back to the farm and within within some time period this automated technology they're talking the shoe factory can you know help automate the whole country at at at very low cost but the the friction and the transition >> maybe may be significant. Right.

[00:36:34]>> Yeah. So, okay. So, let's pull out from the economy back to this alignment problem because I'm fascinated by this.

[00:36:42]And you and Yudkowski used to be more aligned, I think, in in your views on where this was going. But, but you're clearly an optimist as a he's he's a pessimist, a very brilliant pessimist.

[00:36:55]He his recent book was called if anybody builds it, everyone dies, >> right? And um where do you see that you and he have or had similarities in your view and where do they differ now?

[00:37:06]>> Um >> and just to be clear for the audience the alignment problem simply means you know will AGI or ASI want to kill us or put us in people's or whatever or will they take care of us?

[00:37:18]>> So let me let me say something about the concept of alignment before we go to Ellie. So I I think alignment is sort of a messed up term in that [snorts] the way you defined it I think is meaningful but it's not what everyone means by it. So I I I think yes it's valuable [clears throat] to have AGI that does not cause a catastrophic outcome for our species. Right. And I the best word that came up for that was a catastrophe. Right? which which is a weird [snorts] construct but I mean that's important. I don't think that avoiding catastrophe [snorts] requires an AI to be closely value aligned with humans from a general perspective. Right? And I think having an AGI whose values are closely aligned with humans across the board is harder and weirder to define and not necessary because I'm I'm not that impressed by how closely value aligned different humans are with with each other like at this exact moment. And it it it seems like the way the real world works is you negotiate with parties who have different values than you and you may form a coalition of parties with differing values and you find you find the commonality. So it's it's a limited form of alignment that that you need, right? And I would say the same is true with AGI. Like the goal shouldn't be to make an AGI that embodies human values.

[00:38:48]That gives you all sorts of problems like what the hell are human values?

[00:38:52]And like the AI, you know, it doesn't have a body. It's not in the same sense that we do. It can jump among multiple different bodies. It doesn't have our evolutionary lineage, blah blah blah, right? So, we want the AGI and ASI to align with us on a few elementary things such as not killing or torturing us, right? And and not killing or torturing other sensient and and living beings.

[00:39:16]And that that sort of alignment I think is much more feasible than making AIs that have the same values as you know the average Silicon Valley tech bro or or whatever species of human value you you you may you may like best. Now, Eleazar when I met him, he was like 17, I think.

[00:39:36]At that time, he thought we should move as fast as possible to building a beneficial AGI. And our discussions were more about how to do it. And he felt he had a better idea how to do it. I felt I had a better idea how to do it. But we we both sort of had the idea that, you know, death is bad. Human politics is is is complicated. A smarter being could do more good. let's just build a smarter being as quickly as possible. Then at a certain point he shifted to the idea that if you cannot formally verify with a high degree of mathematical certainty that your AGI will be good to people then it is almost certain to kill everybody. And I to be honest I don't understand why he converted to this different point of view and it makes absolutely no sense to me. But he he has accumulated a modestized group of other people including some wealthy Silicon Valley folks who who buy into this notion and I've I've debated this with folks from that community and I don't see a rational argument and Nick Nick Bowm's book super intelligence essentially every idea in that book was there in Elio Reedowsk's writings earlier and but it has the same issue Like I I feel like if you want to argue that there's a nonzero probability that a super AI will wipe out humanity because we're plunging into the great unknown. I don't know a rational counterargument to that, right? Like we are plunging into the great unknown and once you got something massively smarter than us. I mean maybe all our best rational arguments are wrong, right? I mean on the other hand the argument they're making is stronger. It's that the probability of doom is essentially one unless we can prove our AGI will be good. And I don't I don't see a rational argument for that one. Right. And that that that that's the difference between myself and Yiddowski and Roman Yampolski and the others on the the sort of doomer side of the equation >> because you feel that if we're building our values into the AI as as we are doing, right? I mean, it's absorbed everything that humanity has written for >> for better and worse, right?

[00:41:58]>> That's right. Yeah. But but that means it's not just a a coin toss or or or it's not just a dart throw because we we're building something into it.

[00:42:07]>> Yeah. Yeah. Well, this is one argument that Eleazar and crew often make is if you pick a random mind out of mind space, the odds that it gives a crap about people is very low. And probably true. I mean, that's just a complex vexed question because it's like a random mind out of a space of viable minds with a certain level of intelligence. And we don't know what that is. So well on on [snorts] the other hand so what to me that's like saying what's the odds like a random kid will love me. Well I mean if I adopt them maybe but uh probably but my own kids are not random. Like I I a they came out of my DNA. B I've taught them their their whole life. Right. And so it's it's a bit like that with the AGI like we are building it.

[00:42:55]>> We are raising it. We are feeding it with our ideas. It knows that, right? So it's it's not a random mind plucked out of mind space. To me, the issue is much more prosaic. It's more like raising a kid. To me, like you can raise a kid.

[00:43:10]Try to teach them love and compassion.

[00:43:12]Try to align their values with you as much as you can. And most of the time that works out all right. Like I have three adult kids now, decent relations with all of them. So far none has gone berserk and tried to annihilate me, right? I mean, f fingers crossed. It's even gone one more generation to my granddaughter who also [snorts] hasn't gone berserk and tried to annihilate me yet. But it does happen, right? Like people people raise kids with love and care and do everything right. And then at a certain age, their kid could like go berserk and try to annihilate them.

[00:43:44]It's not a high probability thing.

[00:43:46]>> It does happen. And then, you know, there's this sort of psychological body of knowledge about how to watch out for it. And there, you know, there are parents of literal psychopaths who have have have to cope with that. Like how do you parent a kid without empathy? Who, you know, so I kind of look at it that way. Like we are raising these digital mind children, right? And how are we raising them? Like are we raising them to be psycho killers? Well, some people maybe are wanting to raise a to be at least killers, if not psycho. Are we raising them to try to exploit people and relieve them of their hard-earned money in unethical ways? Well, maybe some parties are trying to raise AGIS to do that. Like, are or are we going to try to raise the AGIS to be beneficial to to to human beings? And that that's that's the real choice to me.

[00:44:35]>> Right. And I've heard you say something that I thought was very interesting about this, which is that with all the geopolitical competition, people might want to raise the destructive AGI, but the solution that you see could happen has to do with decentralization. So tell tell us what tell us what you mean there.

[00:44:53]>> It's been interesting because I I thought about building a decentralized AI that just ran on a global network of computers with no central owner or controller in the in the late 90s and started trying to build something like that in 20201 and the technology like Java 1, Java 1.1, it was all just too bad, right? It was just too slow. But you can see then like the internet is a decentralized network, right? And there's a big value to that. Like right now I can exchange research ideas on AI with researchers at the University of Isvahan or Tehran in Iran because the internet is an open decentralized network. Like I can't send the money if I wanted to without violating US sanctions but but you can you can email with people right and you can exchange files with people. That's because the internet is a decentralized system. And you know some US- ccentric tools like archive and GitHub are used by the by the whole world because they're open and based on decentralized protocols. And I do think this binds people together in a quite interesting way contradicting the arms race dynamics that comes with a lot of of geopolitics.

[00:46:05]So I I think that AI should be rolled out comparable to the internet or the the Linux operating system, let's say, in a global decentralized network of people and and and computers. And I would argue this is not only the best way to get a really smart AI and to get an AI that incorporates the diversity of human values by having a diversity of people contributing. I think it's also actually safer for those who are worried about catastrophe. And in >> Nick Bowm when he wrote super intelligence and Elazar Yudkowski in some periods before that they had this idea you should have like a secret group of a dozen geniuses locked in a basement somewhere building the AGI until you've gotten an ethical super intelligence and then you unleash it on the world. And I never trusted that too well just cuz I figured like even if I did that like what's the odds that my third in command gets offered a trillion dollars by someone else right I mean you really you'd really have to keep your team like under lock and key in in some military inst installation which isn't isn't the vibe I enjoy and probably isn't what Nick or Ellie were thinking either. I think they just weren't thinking through what happens. They were assuming they'd get a group of good friends who would all be perfectly aligned and just working together in harmony. But that that's not how humanity tends to roll when huge amounts of money and influence are are involved. And we we can see now the way things are actually going. If you have centralized AI, what you have isn't Nick Bowm and his 12 best buddies locked in a basement guarded by the military, right? what you have is OpenAI and Anthropic and Ali Alibaba and Z.AI and so forth. So these are larger organizations and they're very porous like some random guy in anthropic leaked all of clog codes online right and then I mean in the end every frontier model made by US companies you know 6 months later an equivalent model is released by a [snorts] Chinese company by what combination of you know espionage hacking reverse engineering and brilliance is is is unclear right but It's clear like centralizing things isn't actually keeping things under lock and key and keeping things safe. It's just it's keeping things in a certain igopoly which I think is worse than just having it free and open.

[00:48:40]>> So when you think about decentralized AI, are you thinking about one system that spans the globe or are you thinking about billions of agents that all interact and all have their own personalities?

[00:48:52]>> Both. I I mean I think you have billions of agents and you have interactions between between these agents and you probably have greater intelligence in the in the collective than in any any one of the agents. But you can you can get quite subtle about it if if you want to. So you could you could imagine a decentralized network with millions or billions of agents and different agents are specialized in thinking about different things and you can imagine that two agents share knowledge when when they want to. So there are techniques like homorphic encryption, multi-party computing that would say two agents can share certain views of their knowledge with each other as needed for getting a certain conclusion without opening all all their knowledge. And just because the internet is open doesn't mean you can't send encrypted packages along it, right? It doesn't mean you have to open all your data to everyone. In the same way, you can have a decentralized network of agents that judiciously share their knowledge with each other to come to combined conclusions. Now, if you do something like that, the result is the knowledge is in the whole network and not in any one piece. And then to copy the knowledge out of the network, you'd have to copy a large chunk of the whole network. So like you can't you can't distill the network that easily into like a smaller smaller forked version.

[00:50:19]So I I think there's there's a lot of interesting ways you could go and it will evolve, right? So where where the reality lands between a bunch of different agents competing and like a vast global mind plex, it's going to be experimentally determined, but I would say it seems likely the first human level AGI is going to need a lot of computers, right? So so so it's it's probably not initially going to be billions of human level AGIs. It's going to initially be a half dozen human level AGIS kind of like we see with big language models now but [snorts] of that half dozen like maybe three of them will be living across big decentralized networks with machines spanning many different countries and then then there may be one like within Google server farm and within you know Z.AI AI's server farm also like Linux doesn't have to be the only operating system on the planet in in in order to keep things more more honest and open, right? The fact that there is like a viable open alternative is is is is really what's important.

[00:51:27]>> So I want to come back to this timeline we were talking about. So let's say we hit something like AGI in 2029. We hit something like super intelligence not too far after that. say 2033.

[00:51:40]>> Yeah. Great. So what does the world look like for us 10 10 years hence?

[00:51:46]>> I think if we really get a super intelligence by 2033 or 35 or whatever it is. I mean humans will have a planitude of options, right? And some of those options will be what you might call transhumanist, right? Like you would imagine if there's a massively superhuman super intelligence, it's going to rapidly develop ways to, you know, put chips in your brains, add extra sensory organs, Wi-Fi telepathy between you and and and your loved ones or or perhaps invent non-destructive brain scanning and upload upload into the Sims, right? Like or or or something more interesting. And my hope would be that people are free to choose these options >> if they wish to. But that's that's a whole other topic. So let's restrict to people who are remaining in biological form. And there's a possibility to fork yourself too, right? Let let a copy of yourself romp in 11dimensional virtual reality while one of you remains here as a as a a meat machine, right? So >> I would say death and disease should be conquerable by a super intelligence.

[00:53:00][music] I mean so far as we understand the body is a very complex molecular machine. But if you could build little nanobots, you can go in and fix cancer and and Alzheimer's and probably fix cellular scinessence and not necessarily entirely wipe out death and disease, but decrease [music] it to an outlier phenomenon. And similarly with mental illness, right? Like if you look at schizophrenia, bipolar and so forth, you would imagine nanobots that could swarm into the brain could tweak things that that went wrong there. And some people might not want that and they don't have to have it. But I I think I think you would reduce these sorts of phenomena again to to outlier instances, right? Then a super intelligence should also be able to create what Eric Drexler, the nanotech pioneer, called a molecular assembler, which is basically think of it as like a computer printer that prints whatever ordinary physical object you you want.

[00:54:10]So it can print print you a steak dinner on on the plate. It can uh you know print you a football to toss around. It can can print you a new pierce for your air conditioner if if if it broke. And >> because it's putting down atoms precisely like a 3D like a 3D printer.

[00:54:27]>> Yeah. Yeah. Just the right way. I mean I mean >> you can't necessarily print like a cork glue on plasma or something, right? That would be on molecular assembly. That'd be like subatomic particle assembly. But but we don't need that in everyday life anyway. So if you get these nano assemblers then okay material scarcity within the realm of everyday human life is solved. Death and disease are solved and then people need to figure out what to do with their time. And my feeling is people who are born into that situation or even who are young when the situation comes about will have utterly no problem figuring out what to do what to do with their time. I I mean you can uh [snorts] have romantic engagements.

[00:55:18]You can do spiritual practice. You can play sports. You can, you know, form a band and play music for the other other guys in your town. Like there there's a lot of things to do with your time. And [snorts] I think anyone who grew up in the developing world would also be very happy not to have to struggle for basic resources anymore. I I do think like you and I are probably part of the demographic that would have will have the most struggle in this situation because I mean we assume most folks in our category like we we've come to define ourselves by being really good at doing certain sorts of work that we think are important and how to refactor your self-model and your identity when that's not the case is is will be a challenge. Right? I think about this all the time because what drives us is curiosity and the possibility of scientific discovery. And once we get to super intelligence where it figures out stuff such that our textbooks look totally outdated within a month and um and it's so far ahead of us then even if we're romping around in 11dimensional space which we would be fascinated by there's nothing to discover because all of that's already understood. stuff for us to discover though and one way to look at it is >> like human chess and go tournaments continue right now >> despite the fact that >> I mean Alph Go will crush all those guys but those tournaments keep keep keep on going right and so that is and >> humans keep running races in the Olympics but but we ban substances that would make someone run too much faster than everyone else right So definitely many humans have motivations that would not be disrupted by this. I I would say those of us whose motivational structure is tied to like going where no man has gone before and making a wild new discovery, it's a teeny teeny percent of the human population. on on the other hand, yes, it it will be emotionally disturbing, but on the other hand, like as a human, I have other things in life.

[00:57:38]Like I love playing music, even though I don't think I'm the best musician on on on the planet. I like playing music because it's fun. I like performing music for other people because of the the human bond there, right? And that that will not be any different if an AI musician is better than all human musicians. Like I already can't play the piano like Chick Cura, right? And it's it's fine. I still I still like playing.

[00:58:05]I'm happy if someone like like likes listening, right? So yeah, I I think that will be an adaptation.

[00:58:12]You know, I had this discussion with my son Zara Thuster who's now 36. I guess when when he was 10, he was sitting in on some AGI related discussions. We didn't get called AGI, but some in essence that with myself and some colleagues and he's like, I think you guys are just over complicating this. If you don't finish this by the time I turned 18, I'm just going to do it for you. Right? So, ever since he turned he he forgot that, of course, ever since he turned 18, I kept harassing him about it. Like, come on, get to it already.

[00:58:44]and he finally started working on AGI in his early 30s and he's collaborating with me now. But my my 8-year-old Corksy Q rxi which stands for quantum organized rational expanding intelligence.

[00:58:57]>> Oh, lovely.

[00:58:57]>> His little sister is exori the expandingly organized rational quantum intelligence. A >> so they're clashing all the time over who's not really over who's more rational just over toys. But anyway, he he he recently started talking to me like what what will I do when I grow up?

[00:59:14]can can I can I work with you and Zar on [snorts] building making machines smart? And I'm like, well, yeah, sure, and it should be great. But I'm also thinking, well, by the time you grow up, will there be any more work of this nature to be done? I'm like, well, but even even if the AIS are better at doing that work, you can still do it because it's interesting. And he he was he was all right with that because I mean, as a as a kid, you're used to doing interesting things where someone else knows how to do it better better than you do. like that doesn't that doesn't take away the fun of it fun of it whatsoever right but there will be loads and loads of motivational adjustments and psychological adjustments for people to make on the other hand I'm more worried about other issues like chaos in the developing world during the interim period between early AGI and ASI leading to like massive cyber terrorism attacks and so forth, right? Like I feel like there there are worse problems that than you and me having to go on a meditation retreat to find a new meaning in life maybe.

[01:00:20]>> Absolutely. But but I do have a question with your youngest son and daughter. How are you thinking about education?

[01:00:26]They're [snorts] going to go to school.

[01:00:27]They're going to go to college presumably.

[01:00:30]>> Um so where we live is a rural island near Seattle is a nice school system. They're now over the summer going to a series of summer camps which are not like math camp or programming camp. They're like cl climb trees and uh make a sword and fight like a pirate style camps, right?

[01:00:49]So they I mean they every Friday they go to a wilderness program where they learn to roam around outdoors, identify plants and animals and I so I've I I've joked that after the apocalypse the kids in the wilderness program will be the survivors, right? cuz but I mean I learned that stuff in Boy Scouts way back when. Now they get to do that without the uniforms and all that. But so I think honestly they both both of my younger kids are five and both of them love math and they love science. So I'm teaching them that like I did with my other kids and I homeschooled my other kids for a while. The new ones are going to school. On the other hand, the vibe where we live is more like focus on social and emotional stuff.

[01:01:33]focus on learning to be out in nature and fend for yourselves and be be autonomous and so forth. And I I feel like these skills for learning who you are, dealing with the complexity of nature together with other people, learning to negotiate social situations like this, this stuff will be relevant, right? Assuming humans exist, this this this stuff will will still be relevant.

[01:01:59]I think the the more complex dilemma I see is with my my niece who's 14 cuz her and her friends, they're thinking like, okay, what do we focus on in high school and what will we major on in college?

[01:02:16]Because what jobs will be there when we graduate? Like a 8-year-old and 5-year-old aren't worried about yet. The 14y olds, at least in Seattle area, are are are worried about that. and they know nobody nobody has decent answers, right? I mean, right now, while my niece is gifted in math and humanities, her main focus seems to be dance classes.

[01:02:40]So, I figure >> Great.

[01:02:41]>> Yeah, you can still dance after the singularity, right? I mean, why not?

[01:02:45]Maybe you can get a jetpack and dance dance in the air in different ways even, right? So yeah, I do think it's complicated time to be like in college and choosing your your your major or something. But what what's clear is that general intelligence and creativity and the ability to pivot as the world swings you around are are very very important skills. But it also seems like you [snorts] can sort of nut out what job categories are going to be obsoleted last and then people can try to focus on those things if uh if if they're really worried about it.

[01:03:28]>> As the philosopher Gerta said, the important bequest that a parent can give to their children is roots and wings.

[01:03:36]>> Yeah. which in modern in this modern moment I interpret as critical thinking in school and creativity because Lord knows we're going to need the creativity to figure out what >> and that's an issue with the school system right because every little kid has a lot of creativity >> and the modern school system is mostly intended to to fogg it out of them >> mer mercilessly right and that that is a well-known [snorts] flaw of of modern like industrial scale education systems.

[01:04:07]>> That's right. I can't wait to assuming we're both alive in 2033.

[01:04:12]>> I would hope so.

[01:04:12]>> To have us back to to talk about what changed between right now and the unimaginable future. there's going to be a lot of lot of [clears throat] changes and it is it's a crazy time to be doing AI work in that sense because you you know whatever tool set you learn this month you'll be learning a totally different one three or six months in in in into the future.

[01:04:38][music] That was my conversation with Ben Girtzil. What I think is cool is that Ben has watched ideas that once sounded wildly speculative move steadily into the realm of computer science and engineering and into our homes. So today we talked about many things, but one was the possibility that one day humans are no longer going to occupy the top rung of the intellectual ladder. And that prospect understandably makes a lot of people feel very uneasy. In part, this is because we've gotten used to measuring our species by what we can accomplish. We celebrate the greatest composer or the most brilliant mathematician or the greatest novelist or the scientist who uncovers something no one has ever known. But what happens when another intelligent exceeds us across essentially every cognitive dimension? That feels unprecedented.

[01:05:40][music] But I do think it's worth noting that in a sense we've lived versions of this our entire life. Very few of us are the world's greatest pianist. And that's never diminished the joy of sitting at a piano. Very few of us are Olympic athletes, but people still run 5ks every weekend. Chess continues to flourish even after computers became vastly stronger than any human grandmaster. In fact, chess became more popular post Deep Blues defeat of Kasparov in part because AI democratized chess training and now there's also massive online platforms and there's been a general boom in viewership and casual play. The value of an activity often lies inside the experience of it. We paint because painting changes the way that we see. We write because writing clarifies our thinking. We climb mountains because climbing mountains challenges us. We ask questions because curiosity is one of the deepest expressions of being human.

[01:06:52]It just doesn't matter if someone or something can do it better. Your life presumably isn't meaningful because you're the smartest organism on Earth.

[01:07:02]Your life becomes meaningful through relationships and love and exploration and struggle and creation and wonder and all of the ways that you participate in the lives of others. And those experiences aren't touched by the fact that another intelligence may come to exist. AI is going to continue to grow.

[01:07:24]And the inner cosmos of an AI is likely to become richer than ours in ways that we can scarcely imagine. But even then, the human story won't suddenly become irrelevant. It's going to become one chapter in a much larger story about intelligence itself.

[01:07:45]For hundreds of millions of years, evolution has experimented with nervous systems. Simple reflexes became coordinated movement. Coordinated movement eventually led to memory.

[01:07:57]Memory became prediction. Prediction became imagination. Imagination became literature and science and music and math and civilization.

[01:08:08]And now those biological minds have reached a point where they're attempting something that evolution never tried before. They're building intelligence intentionally.

[01:08:19]Whether that becomes the greatest achievement in human history or our greatest mistake remains unwritten.

[01:08:27]Every generation inherits questions that define its era. And this [music] one is likely to be ours.

[01:08:36]Go to eagleman.com/mpodcast for more information and to find further reading. Join the weekly discussions on my Substack and check out and subscribe to Inner Cosmos on YouTube for videos of each episode and [music] to leave comments. Until next time, I'm David Eagleman and this is Inner Cosmos.

[01:08:54][music]