The Secret of AI’s Power | Robert Wright & Paul Bloom

Added: 2026-06-10

[00:00:05]Hello Paul.

[00:00:07]>> Hello Robert, good to see you.

[00:00:08]>> Good to see you. It's been too long.

[00:00:09]You've been raising your allegedly cute baby.

[00:00:14]We'll We'll return to that subject by the way.

[00:00:17]>> You know my baby's cute. I sent you pictures.

[00:00:19]>> I know you think your baby's cute. I mean I should just just to foreshadow this later in the podcast, possibly in the overtime section because this could get a little bitter.

[00:00:28]We're going to have a a cute baby face-off because I I now have a granddaughter who is almost a month old and Paul has a daughter who's what 3 months old who Paul thinks is is cuter [snorts] than my granddaughter and so we'll we'll I mean ultimately we'll let the viewers decide but >> some background to the viewers, Paul's saying the same thing you're saying. My granddaughter is cuter than your daughter at you know what you Send me pictures at the age that they're at so you can calibrate. So send me pictures and then I'm like adorable kid I have an adorable kid. So I send pictures and then and then I say to be nice, "Look, it's not a it's not a competition." Yes, it's all a competition. It's status.

[00:01:06]>> I see. Status is a is a zero-sum game and when it comes to babies, cuteness is the currency, you know? It's like when you and me What?

[00:01:17]>> They don't have much else.

[00:01:18]>> No, I mean they will I will be comparing my child's my grandchild's SAT scores eventually with your child's. We will do that podcast but right now it's just all about cuteness and um let's just say I'm going into the into the competition pretty confident, okay?

[00:01:35]>> We should be so lucky as to plan to be comparing our kid these kids' SAT scores.

[00:01:40]>> Especially in light of the uncertainty cast across our future by artificial intelligence. How is that for a segue, Paul?

[00:01:48]>> Someone >> I should introduce this. You're the proprietor of the famous the famous Small Potatoes newsletter and a a world-renowned psychologist a man about town uh recently gave a TED Talk, uh such as your stature, at the real TED conference, not some kind of >> TED TED The main The main stage.

[00:02:08]>> Yeah, yeah, as they call it, TED. And uh I am Robert Wright. I was not asked to uh give a TED Talk this year, but I'm not bitter. I'm I'm totally over it. I really am. I I rarely think about it anymore, Paul. Anyway, >> Your TED Talk was the one that you did give >> classic.

[00:02:23]>> It was a classic.

[00:02:24]>> It was ages >> Yeah.

[00:02:26]>> Uh it was [laughter] Oh my god, it was 19 years ago. It was the first year they put them online as videos. But anyway, uh what else? Oh, yeah.

[00:02:36]You Wait, you're saying I don't look young now, Paul?

[00:02:38]>> You still look young, but you looked so young back then.

[00:02:41]>> [laughter] >> That's the way [clears throat] I would put it.

[00:02:43]Um Oh, yeah, and I put out the Nonzero Newsletter. This is the Nonzero Podcast.

[00:02:49]Anyway, Paul was kind enough to say, "Hey, Bob, since you've got this book coming out in only a couple of weeks called The God Test about artificial intelligence, why don't we spend the whole podcast talking about your book?" Now, I'm too modest to do that, but you I mean, moreover, you you you you went the uh the extra mile and uh actually purport to have read the thing. And uh that even if only half to is uh a real commitment.

[00:03:22]>> So, my my only criticism of the book the big one is you get the book, it's PDF, you sent me the PDF yesterday, by the way.

[00:03:30]Um >> This is where I'm not sure you read the whole thing, by the way, but go ahead.

[00:03:34]>> So, I I had no clover appetite in bloom.

[00:03:37]Mhm. You know, >> [laughter] >> and I'm in the acknowledgements in this huge name dump. I benefited from conversations from the following people.

[00:03:46]Uh and there I am and everything. You know, I I I I feel a more accurate description would be I shaped your worldview this. I I I And beyond that, you wouldn't be in a shape to write a book at all if it wasn't for our conversations and my my guidance for you. Just when you do the paperback, just amend.

[00:04:02]>> Paul, weird thing happened in the production process.

[00:04:05]>> [laughter] >> And I don't know how this happened, but you know how it is being an author, you're not in control of everything.

[00:04:10]There had been the final paragraph of the acknowledgements began in finally, I'd like to devote a paragraph to my spiritual advisor, Paul Bloom.

[00:04:16]>> That's right. That's right.

[00:04:17]>> And I don't know, it's weird.

[00:04:19]>> I I can't reading. And then, you know, you talk about your wife. Very gracious and say, "Okay, I'm I'm next."

[00:04:24]>> You wanted billing like above my wife?

[00:04:27]>> I wanted I wanted Yeah, you You wife and then you you just do it in order of importance. Anyway, um look, wonderful book. Wonderful book. I enjoyed I I'm a fan of all your books. I have I have I haven't read Three Scientists and Their Gods, but I but I have read every other book you have written. I should read Three Scientists here just for completeness purposes. Um I will say a few things. One thing it is it is very nicely written. You you write you write like you talk. You're sort of a uh a pleasing sarcastic aggressiveness to it. Nice humor. This is not something written by by chat GPT. You got to say this whenever you talk about a book you say.

[00:05:02]>> And the book works as I see it on three levels. So, first it is a very state-of-the-art description of where we stand uh with AI. And despite the fact that it's a book and it first takes time to go through things, it is you know, you're talking about you know, talking about things happening in 2025 and it it is you're talking about more book you talk about. It is very current. It is very thought provoking.

[00:05:24]>> Yeah, I even Yeah, I was I was like really antagonizing the production people at the last minute with insertions and galleys, but go ahead.

[00:05:33]>> Second, as with all of your books, um uh you you spend a lot of time with engaging in beefs and uh you know, and and settling old scores.

[00:05:43]>> OH, NOW THAT NOW, We'll get to that, okay. But anyway, a false allegation, but go ahead.

[00:05:48]>> False Some Somebody just Let me tell you what I think of it. Seriously, Chinese people >> me just say, I just had lunch with Steve Pinker, okay, and we were >> the one of the people who would say >> one of the people who where you would say I'm like airing some kind of like beef with him.

[00:06:05]Uh and uh we parted on good terms. Of course, he hasn't read the chapter yet, but No.

[00:06:11]>> If you read the chapter and what you had to say about him, he he you know, he would not be having lunch.

[00:06:15]>> I told him there was one paragraph he might not like, but it's very It's fine.

[00:06:19]It's fine, man.

[00:06:20]>> No, it is it is fine. It is fine. Um there was another person who you can't even bear to say his name.

[00:06:27]>> But, I [laughter] didn't say his name.

[00:06:28]>> You just allude to him. You allude to You allude to a blogger. You say, "No, no, no, there's some blogger or like that who, you know, Can I say his name?"

[00:06:37]>> I refuse to confirm the name, but say go ahead, say a name.

[00:06:41]>> I think it is Jerry Coyne.

[00:06:43]>> Oh, I've heard that name.

[00:06:44]>> Yeah.

[00:06:45]>> That's all I can say.

[00:06:46]>> A little bit of a little bit of Dennett.

[00:06:48]A little bit of Dennett. You can say >> Well, Dennett I mention in the uh in the appendix in a totally fair way.

[00:06:54]If he were alive, he would have no objection at all. Uh that that I think.

[00:06:57]Uh but uh yeah.

[00:07:00]>> But, anyway, um and then third and the biggest thing is the the book is an argument. It is It is an argument for the real importance of AI.

[00:07:12]Um here, I I got I got I got these quotes. I I got these quotes that I I love them because of their their You describe what's happening at LLMs as You first say, "The most abruptly dramatic transformation of human experience and human society in the history of our species." And then, a few You're not like happy with that. That's the end of the To understand So, at the end, next sentence, the later say, "I think the coming of artificial intelligence marks a major threshold, not just in the species of our technology, and not just in the species in the history of our species because you said species before it's too but in the history of our planet so you you and then you and you make the case for it you make the case the word God shows up in many places you talk about this as I think an extraordinary threat to us an extraordinary opportunity which could bring upon you know this wonderful paradise and also bring together this challenge could bring together the people of the of the earth but if we fail this challenge we are toast and then and then you know and and the God that's right it's like the scene in the movie where somebody said it's the perfect storm and I for every for every podcast you should say it's the God test >> I like the way you whisper that and I just use that recording that little snippet [laughter] >> that's a clip it in I heard and you know and we get into it it it it book establishes themes um uh that were developed mostly in non-zero a lot of shout-backs to non-zero but also to some extent in your Buddhism book to some extent in the evolution of God themes about about human progress about about the the purpose of of life the purpose of evolution about you know historic broader historical trends challenges and so on no books of the light okay can I ask you a question >> okay should people is it is it should people pre-order now >> that's right should people pre-order should they buy should they put in their review in amazon.com should they we're going to talk we're going to talk by the way in overtime segment about the mechanics of >> about book promotion these days and I must say rapidly changing landscape am I wrong I mean you're so prolific that it changes less rapidly for you you write a book every like three years or something but I >> also also with my level of fame the promotion hardly matters >> no I know you just your people call their people and you show up on like Joe Rogan or something >> That's right. That's right. Me and Joe.

[00:09:39]Um okay, so so a couple of questions just as a way to get you to introduce the theme of the book. So, in in these passages of uh of uh where where you talk about the importance of superintelligence, you you talk about awe. And you say one of the points of your book is to persuade people that they should feel awe um about the what's coming with AI. And it occurs to me it's a different way to slice this. Like, you you you could talk you talk about doomers who think AI will kill us all, accelerationists who think that they're going to bring upon this post-scarcity utopia, and we should hurry things up.

[00:10:14]But you cut things differently, and it occurs to me it's an interesting cut.

[00:10:17]There are those who experience awe when they think about AI.

[00:10:21]It is people like you. It is people like Yudkowsky who thinks it's extraordinarily powerful, will kill us all. It is most of Silicon Valley.

[00:10:28]Who are the people who run Anthropic and OpenAI, just Sam Altman and so on.

[00:10:34]And then there are people who I would would cleverly describe as naw.

[00:10:38]Who would just say, it's not going to kill us all. It's not going to save the world.

[00:10:42]>> It's hype. Well, or moreover, the you know, the companies are hyping it. You you see this a lot on the left, I would say, more than is good for the left.

[00:10:50]You're seeing this kind of thing that I think amounts to a kind of denialism that will in the end come back to haunt the left if it persists. But yeah.

[00:10:59]>> Yeah, so I would include on the list, and I think these are all previous guests that you've spoken to, Gary Marcus, friend Gary Marcus, Emily Bender.

[00:11:09]Um >> Both mentioned along with you in the acknowledgements, by the way, and they're not complaining. Actually, [laughter] Emily Bender has been known to complain about me, but never mind.

[00:11:18]>> Yeah, she has his words for you, but Freddie de Boer, who I've been reading a lot in AI and is very very good >> him on the podcast, right?

[00:11:24]>> I'd have to have him on the pod. I would love I'd love to hear that.

[00:11:26]>> That's a good idea.

[00:11:28]>> You know, and he would he would go on and say, "Hey, here's a challenge for you, you know, spend a month without AI or spend a month without indoor plumbing.

[00:11:36]And as you're sort of in your backyard, think about which is important.

[00:11:40]So, he would he would he thinks it's just a hype. And then So, here's the here's the way to frame this. This is the way to sort of get you talking a little bit about what would you say to these very intelligent naysayers?

[00:11:51]What What are they missing?

[00:11:54]>> That That is a question I very much want to answer. Uh I want to quickly say that the word awe, like the the doomers are uh feel it in the in a in a kind of archaic but but earlier, you know, in earlier sense of the term, awe in the sense of terror. That was the overwhelming connotation of it, you know, couple hundred years ago.

[00:12:15]Uh and then it was actually because of its religious usage to refer to the divine. The divine inspired terror.

[00:12:23]That uh it was kind of in in concert with the evolution of the conception of God, I think, that it came to have more positive connotations. All in the sense of sublime whatever. Anyway, yeah, I go into that a little, uh but but that's not the question you asked. The uh Here's the thing. And this is what I've really uh tried to do in the book. One of what One of the things that I most hope I succeed at is explaining to non-technical people from early on why this has got advanced so far, so fast. And I think if you don't acknowledge that it's been at least fairly rapid advance of capability, you're really not paying attention. But moreover, why we can be sure the advance is going to continue.

[00:13:15]And here I draw on what is the closest thing to a narrative framing of the book, which is that as it happens, uh in the 1980s, I interviewed the person now, uh, commonly referred to as the Godfather of AI, Geoffrey Hinton.

[00:13:33]He's won not only the Turing Award, uh, for but the the Nobel Prize for what he did because of its relevance to physics.

[00:13:42]Um, and at the at the time I was writing about these two schools of thought, one this maverick school involving neural networks that was kind of on the margins, and then the mainstream approach, which is kind of the Gary Marcus approach. Um, as it happens, the the maverick approach succeeded, and now Hinton is this god in the field. He himself is a doomer, though. He's terrified now, which he he wasn't back when I talked to him, but anyway, the thing is, I now understand something that I didn't understand at the time.

[00:14:14]And I'm honestly not sure how clearly he he understood the full potential of this fact I'm about to describe, but uh, he he was a lot closer than I was. I mean, we were nowhere near large language, uh, models back then. As I note in the book, he did write a paper in the in the mid '80s that that kind of anticipated that, but even that was after I talked to him. And, uh, so anyway, here's the thing.

[00:14:45]Is, um, and and maybe I should uh I didn't didn't set this up, but I think maybe I'll try to find this thing I quote from from the 1956 Dartmouth Conference at which the term, uh, artificial intelligence was coined. Uh, okay. Uh, the the the the the the founding, you know, the founders of AI, uh, John McCarthy, Marvin Minsky, um, and so on. I think maybe Claude Shannon, who did information theory, was there.

[00:15:16]Anyway, here here's the quote from the proposal, uh, that led to this conference in the summer of uh, uh of of of [clears throat] '56, 1956.

[00:15:30]Uh they said uh so you know, artificial intelligence is taken to be uh you know, making machine behave in ways it would be called intelligent of human results of it behaving. Here's the key passage.

[00:15:42]That turned out to be deeply misleading so far as what would actually happen.

[00:15:48]Quote, the study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. Okay, the key thing is it turned out with this neural network approach, what Hinton was calling even when I talked to him in the '80s, massive parallelism, even though it wasn't massive yet, he was explaining that once we had a lot of cheap microprocessors, this would this would become very powerful. The key thing is with this, you do not have to, as they put it, uh precisely describe the dynamics of human cognition and instantiate them in the machine.

[00:16:33]The machine does the building of the cognitive mechanisms that perform the functions that the human mind performs. You just feed in a bunch of human data and and train human generated uh data uh or human uh perceived data, like in in in the case of auto, you know, self-driving. Uh you know, but but anyway, you you you feed in input and output that that, you know, information like as it would be processed by a human brain and and you train the machine to predict uh how how the human brain is going to respond. Most famously, just, you know, next word prediction, which is the first phase of training of a large uh language model, where here's the first part of the sentence, what would be a plausible next word?

[00:17:24]Um, and it turns out that if you just teach a neural network to do that, um, it will build these these mechanisms that we now understand do uh, have a certain parallel with the mechanisms in the human mind. But, we didn't have to understand anything about the human mind to get the machine to create these mechanisms. That's >> I'll just jump in and say I you you you're going on this, but but to say that your book contains a really good discussion of how it's a mistake to dismiss this as auto complete.

[00:17:59]To dismiss this as sort of, you know, it just seems makes it seem incredibly stupid. Like, what Google does when you type in a few words and you say, "No, it's far more sophisticated than that."

[00:18:06]And actually, the mechanism you're describing could lead to an output that's never been manifested in the in the input it received. So, giving it sort of creativity. So, anyway, I just want to say It's a pointer to the book to learn more about this.

[00:18:18]>> And not not only that. So, yeah, I mean, the the whole stochastic parrot thing, I mean, I'm sorry. That's just such a deeply misleading uh metaphor to begin with. Parrots don't uh, they don't process the meaning of the words they repeat.

[00:18:33]And we now know, and this is a good example of the machine kind of inventing something significant. Like, it has a system for representing the meaning of the words, and we now know what that is.

[00:18:45]I don't want to give it complete credit for that because we did say to the machine, "Look, you got to represent each word, well, strictly speaking, each token, as a long sequence of numbers separated by commas, that is to say, um, a vector."

[00:18:59]Uh, still, we didn't say, "And and figure out a way that that this this sequence of numbers will reflect the meaning of the word." But, we now understand in what system how the machine actually does wind up breaking words down into the kind of the components of their meaning and representing those with numbers. I I I I won't go into that further, but it's for that reason that and this is I think a key point still I think not totally appreciated or not everyone in the field would agree with this characterization. Many would, but even they don't articulate it almost ever and I don't think people understand it, which is that the training of a large language model is is like yes, it is in some ways a process of learning, but in important ways it is like a process of evolution. It is like it is like collapsing all of human like natural selection into a few months at least in the sense that it really does come up with specific mechanisms, cognitive mechanisms, and perceptual mechanisms that in the case of vision training um image recognition that it took evolution much much longer to come up with. And evolution like built them into us.

[00:20:16]Natural selection built them into our genes. So, it's not it's not just you know the old phrase uh ontogeny replic- recapitulates phylogeny. What What that mean? You know, ontogeny unfolding of an organism recapitulates evolution. It's a disputed claim, but the point is, you know, it begins with the the acknowledgement that if you view embryonic development Right, like if a child they start out looking kind of like reptiles and they look more like, you know, and it almost looks like sequence of stages of evolution. Well, leave all that aside. My point is that the training of a large language model is like both phylogeny and ontogeny at the same time.

[00:20:56]I don't mean that it reverse engineers exactly the mechanisms in the human brain, but I do think and again some people in the field might argue with this, but I think not that many, honestly. I do think it reverse engineers mechanisms that are functionally comparable to the mechanisms in the human brain that evolution engineered in. And this is why we know it's going to get super powerful because in any realm at all, driving a car, doing the dishes, whatever, uh it is possible, and we haven't done it yet in a lot of cases, or the job you're doing now, the reason Mark Zuckerberg is now monitoring the keystrokes of his employees is he's doing this with them so that he can replicate uh the thought process they used to do their job, and then he can fire them, okay? Because you can take any any data set in any perceptual realm, uh you know, speech, audio, written, vision, sound, smell, any, and uh once you've got the data, you can get these things to kind of reverse engineer functionality comparable to our to our own functionality. That's my claim. And >> you kind of answered in two ways. So, first, you sort of described AI as in "Look, shouldn't you feel awed at this intellectual accomplishment that Jeff Hinton and other people came up with?"

[00:22:24]Like that's an that like an amazing accomplishment. I get that. And then you move to more as a sort of meat of the concern, which is "Yeah, but what difference does it make?" Like, you know, right now right now I use chat and cloud for a lot of things. It's very useful. But I still use a washing machine.

[00:22:40]Honestly, the washing machine's more important. If I didn't have a washing machine, I'd have to wash my clothes by hand, and it'd take hours and hours and hours. I'd rather have to go back to Google and give up that give up AI I'd rather give up chat than give up my washing machine. But you're saying that yeah, but what's happening is there's a process that's going to get better and better and better and better.

[00:22:59]Um it's sort of an argument I've actually heard from Sam Harris who who talked he has sort of AI doomer talk once and he says why do we think imagine you could put things on a single scale of intelligence, you know, it's it's I think he dismisses wrong but imagine you can simple as that right?

[00:23:16]So you get a machine that's 80 IQ, 90 IQ, 100. Why do you think it's going to stop at where humans are? Why do you think there's sort of a limit to us? So the idea that I think you you tell me if I'm wrong is that right now these machines are impressive in many ways unimpressive but but the direction that they're going is they're just going to get better and better than us and maybe become as if they are gods.

[00:23:40]>> Right. And you know, now you might say well they can only become as smart as us not smarter but first of all we're already seeing exceptions. I mean it was only like last week there was a big hubbub about I think GPT latest GPT model uh solving you know proving one of these great unsolved math problems. Uh and you know, you're seeing stuff like that and there's a couple of reasons for that like first of all yeah, it's you know, it's true that in terms of training data uh you know, it would seem that in some respects the ceiling on the data is just what people can do right? Uh but on the other hand you know, a given a single AI can be completely capacious in the extent of its knowledge. It can be not just like the world's leading expert in one thing but everything.

[00:24:34]Also works a lot faster.

[00:24:36]And finally keep in mind these things can collaborate intellectually the way people can. We've already established that and that is you know, most of the actual progress in in you know that having looked in this having looked into how various ideas took shape.

[00:24:57]You know, there's often one person who gets credit for it, but you look back and it's like as Isaac Newton said, I stand on the shoulders of giants. It's always a collaborative process, whether intergenerationally or intragenerationally, that leads to the great ideas.

[00:25:14]And now you're going to have I'm not saying you have this quite yet, but you're going to have this happening at a very fast pace. This kind of intellectual synergy. I think now people might say, wait a second, uh you know, we maybe we won't have any more breakthroughs on par with like the transformer or multimodal training or chain of thought reasoning, which are are kind of the last big three, I would say.

[00:25:44]I I mean, I think even if you didn't, um we'd go a very long ways, but also you probably will. I mean, they keep figuring stuff out.

[00:25:55]>> So, we've talked before where I sort of relayed Freddie de Boer's challenge, which he made to Scott Alexander. He is he is very much of a this is nothing burger. This is just, you know, and hype. And his challenge is let's to Scott Alexander saying, look, you say it's going to transform the world, so let's put some money on it. Let's make specific predictions about the stock market and the job rate and so on.

[00:26:18]And when I raised that to you, your answer, which I found pretty persuasive, is you think it will have most likely a huge change. And you can't predict the directionality of it. So, it's kind of a wash. If you were confident it would make um it would make GDP magnify by a factor of 100, that's one thing. If you were confident it would drive the economy to to dust, that's another thing. But since you don't know which one it is, it's you can't make money off of it.

[00:26:43]>> Yeah, and and and there's the time the timing issue, too. There's a There's a line from The Big Short, I think, where somebody says, uh I wasn't wrong, I was just early." And the guy says, "Early is wrong."

[00:26:56]>> It's like he shorted the market too soon, right?

[00:26:58]>> This guy, I forget his name. He was played by, um, I should forget the actor. Handsome guy actor. Um, and he's and he basically predicted that the housing market would crash, but he had to keep waiting. And his investors were freaking out and abandoning him. He had to keep waiting for it to happen. And you can imagine a lot of that both the doomers and accelerationists in a sort of waiting pattern. But let me ask you this just to get a feeling for what we're talking about. Um, right now what we're doing is 2026.

[00:27:26]We're I'm I'm sitting here, you know, I'm going to take my daughter out for a walk to the park soon. I have a job. I have, you know, this is life. And it's not that different from 2016.

[00:27:37]>> Mhm.

[00:27:37]>> In between there was a pandemic. This stuff is just a glitch and then only that. The world is different in many ways, but you know, we had phones. We have phones now. We have phones. It's staring in front of a screen.

[00:27:48]Do you think that 10 years from now, it'll still be life is kind of pretty much the same or do you think it'll be radically different? That we won't whatever we're doing now spending our time for pleasure, for work, for relationships, for we won't be doing that. We'll be in we'll be in pods off off to Alpha Centauri or we'll be slaves of AI or we'll be dead or we'll be in a you know, we'll we'll be in in a Do you think things will be very different 10 years from now?

[00:28:18]>> Um, I I do. I'm not I'm not sure Alpha Centauri is where the pods will be, but no, I'm not sure there will be pods, but but I I I think yes, I will say first, people are adaptable in a couple of senses. First of all, they find a way to get by even after change. And B, they kind of forget how things used to be so that they almost underestimate how different the world was. It's like you you notice it sometimes with movies that maybe only 20 years old, but you go, "Oh, this whole plot couldn't happen now."

[00:28:49]>> Yeah.

[00:28:50]>> Like they'd be able to, you know, >> Yeah.

[00:28:52]>> Uh they they they they'd have Google Maps or they >> There was a Seinfeld Chinese restaurant thing which if they had cell phones would have been immediately resolved because they didn't >> Oh, there there's tons of these uh and and so but will will the world be uh more more dramatically different than has ever been the case after 10 years? I would say not only yes but by a long shot. That's that's my prediction. Yes.

[00:29:21]>> That's a great prediction. So So if 10 years from now I'm sitting here, you're sitting there and we're just talking and you know that that is not what you would expect. That's a falsification of the claim. But if instead we're, you know, a difference we can't even imagine.

[00:29:36]Yeah. Because that's what radical change looks like.

[00:29:38]>> Yeah.

[00:29:39]>> You know, I I think I think if you took somebody from I don't know 19 1926 and then you you said predict the crazy future. They may say all sorts of things and be like jetpacks and so on but they wouldn't say and men and women men could marry men and women could marry women and everybody would find that. They wouldn't say it because they they would they couldn't imagine what that difference would be.

[00:29:59]>> Right. That's actually a good example.

[00:30:02]Cuz I was at The New Republic when Andrew Sullivan wrote his cover story for The New Republic which was a huge intellectual event about gay marriage, the case for gay marriage and I was like, "What are you talking about?" I I just hadn't heard anybody bring the subject up and and that's so that's a good example. Um >> I'm old enough that as a philosophical example of something that's true by definition, a man cannot have a baby.

[00:30:28]It's just an example of something which is a simple fact that's true by definition. Now you're getting a lot of trouble in sort of social media sites for saying that a man cannot have a baby because they would say a trans man can have a baby.

[00:30:39]>> What what >> you wouldn't have predicted that.

[00:30:40]>> Yeah. Yeah.

[00:30:42]Yeah. yeah, no. This is all true. Well, well, look, you know, that's why I mean, if you read my book, >> [laughter] >> I leave the future open to a very broad range of possibilities ranging from heaven to hell to something in between.

[00:30:59]Actually, there's something worse than hell, which is maybe worse. I don't I don't know actually, but human extinction. I think that's a real possibility. I don't I'm not a I'm not predicting it the way Yudkowsky does, but I do like march through the logic of Yudkowsky's and I I like to think in a way I go further because they they when they talk among themselves, they just make these sweeping declarations and even when they write books, it doesn't get all that much further than that and I try to put make it a little more fine-grained and and and and just the point is it's actually very hard to dismiss at least the possibility that things could get out of control in the way they worry about.

[00:31:47]>> Now, one of the very nice sections of your book, you know, you point out that Yudkowsky has, you know, famously strong views of it being the author of the book, co-author of the book, if if anybody builds it, we'll all die. That's what the title is. And and you point out that he is not a particularly effective communicator of this and I've heard him being interviewed and he has this impatience with people who don't quickly get his point and his examples are not accessible and so on.

[00:32:12]And so what you try to do, I think I think and I see an evolution of your own views here, which I think you were once a lot more skeptical about the Yudkowsky line, but I think you you said, well, let me make this plausible and let me give you express this in a way that's more understandable and more convincing.

[00:32:28]>> Yeah, and if you want to see how far I've come and even how far he's come, on YouTube, there's a conversation I had with him 15 years ago and he was not yet a full-on doomer. And I was very skeptical of his uh claims. I I got >> He started He started off as an accelerationist, I think.

[00:32:46]>> Yeah. No, he got his his original money from Peter Thiel. The singularity was going to be wonderful. Uh something I never understood. It's like if it's a singularity, like if things if stuff starts changing so fast, I mean, the term singularity it its use in in physics uh refers to a situation where you have no idea what goes on beyond a certain point. The beyond the event horizon the laws you're familiar with break down, and you don't know what's going to happen. And the first use of singularity in this context by John von Neumann made explicit uh kind of reference to that idea of the laws of human affairs breaking down. And so, I was always like, "What Why are you optimistic unless you're like religious?" Unless It's like a religious faith. And they say they're not. They say they're the opposite of that. And uh so, I never got that. But anyway, Eliezer, to his credit changed his views early and has proved prescient in in in predicting that these machines would have certain kinds of worrisome tendencies, which it turns out they have. Toward deception, toward power seeking, toward, you know, and so on.

[00:34:04]Uh toward self-preservation. Like resisting, like resorting to extreme measures to keep from getting shut down.

[00:34:10]Now, these are in laboratory. These are in experiments.

[00:34:13]M- Mostly, these are in experimental conditions, and people say, "Well, they're the conditions are too contrived." I don't know. Um I I I think it's pretty clear that we should not be sanguine about this.

[00:34:25]>> Yeah, I mean, the the skeptical response to some of these demonstrations is nicely captured in this cartoon I've seen a few times where the first panel is somebody going up to like a a computer and say and say hey, say I'm conscious.

[00:34:40]And the second panel is the computer goes, I'm conscious. Oh my God.

[00:34:45]And so to some extent, um these a lot of the behavior of these things, for instance, the the the Microsoft uh machine that uh with that was interviewed by Kevin Roose a few years back and then claimed to fall in love with him and demanded he leave his wife and and that his wife his love for his wife was not sincere. He really loves loves her. Um And and but to some extent it plays this crazy science fiction insane machine which it had been exposed to. And so a lot of the discourse, I think I think the cleverest studies often involve those that go into the guts of the machine and say >> Oh, yeah.

[00:35:24]>> explore systems what how it's exerting self-control, what it really believes in some sense of what it really believes.

[00:35:32]>> Uh what some of the things you're finding, but it's important to remember that there's still a lot they don't know. I mean, to some extent these are black boxes and that's worth pondering. And now, I take your point uh first of all, um you know, when when uh I think it was the code name for for that particular model of the AI was Sydney Sydney for some reason. And Microsoft put it out uh in connection with the with with being, I guess. But anyway, look, first of all, yeah, it's kind of a common vein of thinking about what's happening in some of these situations is that basically it the model gets in a particular narrative groove. Like it's been trained on text where there are these certain kinds of people. Like they'll try to steal you from your wife, which they did in this case. Um And you know, and we should say he worked long and hard, you know, with his prompts to steer into one of these weird grooves.

[00:36:40]>> Yeah.

[00:36:40]>> That said, you know, there's going to be a lot of people using these things and not only that, but remember that, you know, with open-source AIs, people can fine-tune the models, actually change their weights, change their character in various directions, and so there's just My main my main point is we have no idea what's going to happen.

[00:37:08]You know, there's such a range of possibilities. If you're ruling out the catastrophic ones, I really don't think you're paying attention. And we need to get together and think about this, and we have to do that as a global community.

[00:37:23]>> So that that brings us back to Peter Thiel. So you're discussing Peter Thiel's views on all of this, and at one point he talks a lot about the Antichrist.

[00:37:31]At one point you say, I think correctly, that when he thinks about the Antichrist, he might well be thinking about you.

[00:37:38]Maybe not you in particular, but >> I know I don't think he I don't as far as I know he's not aware of me. I say he's thinking about people like me.

[00:37:45]>> You you represent in his view you represent the views of the Antichrist.

[00:37:49]>> Yeah.

[00:37:49]>> That's a I like that passage.

[00:37:51]So so I'm I'll answer my own question, which is this is because you see world government as a viable as response to to the AI potential threat. And you came at this I mean I'm not sure you came at this fully honestly, cuz that's something you had always wanted anyway.

[00:38:11]And this is in some way how this this God test builds up from the themes of your earlier books, which is your argument is that that you're correct me when I if I'm wrong, but one part of it is there's a general trend towards increasing consolidation and cooperation. But the second thing is just on a practical level to solve serious threats, um we need to form a world government of some sort. And um and that AI counts as a serious threat.

[00:38:35]So here's yet another argument for the outcome you want. And the outcome that Thiel I think literally views as the plan of the Antichrist.

[00:38:43]>> No, he does equate world government, I think even he the term global governance with the Antichrist.

[00:38:50]Now, I don't use the term world government in terms of what I'm advocating. Um the uh partly because and and this is really important. And by the way, there's a little news flash. We might as well get slightly topical.

[00:39:07]Uh the Trump administration has has actually issued this delayed uh and and subsequently amended executive order that will as a practical matter mean that the federal government uh reviews the most powerful large language models produced by American companies. Uh strictly speaking, it's voluntary. Uh but the the big makers are are are not going to try to defy it, I think.

[00:39:31]>> Yeah.

[00:39:32]>> And what it points to, I mean, this is the dilemma we face. And I'll get back to And this is why I use the term global governance not world government, is because on the one hand, we do need some kind of systematized institutional oversight of these things, or things will get out of control. I'll elaborate on that.

[00:39:56]Um on the other hand, it's it's you know, leaving aside whether the institutions are governmental, they may not be. And maybe that's better. But the uh on the other hand, there is real danger in a concentration of power in the control of these things. And the Trump administration so perfectly illustrates that. Like right now, it has Mythos.

[00:40:17]We don't. Uh if ever there was an administration I would not trust to use it responsibly, like that's it. And and you know, so mythos gives it the power to I mean, of course the government has a lot of technical surveillance power it's not supposed to use anyway, but this mythos could give it whole new ways to do surveillance without being detected and to break into systems and so on and it's like do you trust the Trump administration?

[00:40:43]Now, I don't. On the other hand, even so I got to admit we need something like this. Uh and so that that is the fundamental dilemma with AI.

[00:40:55]Too much control centralized control is dangerous.

[00:41:00]Uh too too little oversight is dangerous. And both could lead to catastrophes like a global police state at one end and utter chaos, collapse, extinction at the other. So um Now as for when you said well I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I but uh but I would say it's uh I would object to quite the way you phrased it because it's the same argument I've always made.

[00:41:43]AI is is a manifestation of a technological trend that was all along the basis for my arguing for more international governance. Perfect example is bio weapons. You can go back and find me like in the 1990s in the New Republic in a cover story called be afraid, be very afraid. That was Andrew Sullivan's phrase, very good. He came up with title. Uh about weapons of mass destruction like pointing out that like a truly contagious bioweapon it you know, you can't control that at the national level. You need international governance. And so so the argument I've been making all along is that various technologies, information technologies, weapons technology, all points of technologies make relations among nations more non-zero-sum in that they give the nations an interest in a common self-interest in cooperating uh to govern these things. And AI is just much much much more of that pretty suddenly.

[00:42:43]And bioweapons is perfect cuz one thing A, one thing AI can do is help people make bioweapons, so it exacerbates that challenge. And B, a lot of the other things uh AI can do like mythos hopping from, you know, uh data center to data center and and taking over the world or whatever, which of course it can't do, but it suggests that you know, rogue rogue AIs like that could be problematic. You know, that's kind of a biological weapons is a good metaphor for that, right? So uh so the bioweapons thing is a good way to frame uh the the perils posed by AI. And there And there are many more. I mean, surveillance, again, uh and surveillance is is the reason you want to be careful about how much regulation you introduce to to address the the other problems. But yeah, Peter Thiel is uh among those uh who are allergic to the idea of I apparently just about any international governance. He doesn't like the World Trade Organization. He doesn't, you know, um and yeah.

[00:43:50]>> It's true.

[00:43:50]>> You sound very reasonable, just like the Antichrist probably would be on his quest for the [laughter] for world government. Um I guess I would just make a distinction between the normative and the descriptive. I mean, you're putting a really good case for why it's a perfectly rational response for the countries of the world to get together to deal with the AI threat. And it's something that you and I have been pushing for a lot and it makes a lot of sense. As a matter of fact, and I'm not sure you'll disagree with this, at least in the at least in the United States, the narrative has almost been the opposite, which is that the the development of AI has led to sort of a a exaggerated and xenophobic reaction to China. Which is we must we must beat the Chinese so they don't get there first.

[00:44:32]Somebody's going to have this, it better be us and not them.

[00:44:36]And so it's been moving in the opposite direction than the direction which I think you plausibly enough say it should go in.

[00:44:41]>> Yeah, I mean it it it's I I think the race a race dynamic is the last thing we need. And in fact, uh you know, because as I said, this stuff is just already moving faster than I think we're going to be able to wisely respond to, right? We're we're And remember, our government is in a shambles in terms of its ability to respond wisely to anything. It's almost dysfunctional, right? Uh because the polarization is so extreme. Uh which is one reason I, you know, toward the end of the book I start talking about the psychology of tribalism because we can't afford the domestic polarization, let alone the international conflict uh in in I think the coming age. We're going to have to get together and and and think about this calmly. But the uh one one >> I am sufficiently short I am sufficiently uh anti-Trump, sufficiently disgusted and disdained by by by what him and his council is doing that I prefer dysfunction.

[00:45:46]I prefer Well, blocked partisanship, messy and everything, as opposed to an efficient, well-running system where he gets a lot of power to do what he wants.

[00:45:54]>> In a way, what I'm saying is AI is going to render that position untenable.

[00:46:00]In other words, right now, yeah, a completely, uh, dysfunctional government uh, beats a government in which he would be able to do everything he wants to do.

[00:46:10]But, but I'm just saying, uh, in the age of AI, either of those two extremes is going to be hell.

[00:46:19]Okay? Like, uh, some crazy guy centralizing power because the if he really secures control of the technology of AI, um, lights out kind of. I mean, in one sense or another, either oppressive authoritarianism or worse.

[00:46:36]But, at the other end, I'm I'm just telling you, if we have no governance of this technology, uh, I could be wrong, but I mean, at a minimum, you're going to have to have some sort of organic emergence of normative solutions through the magic of the free market in synergy with the wisdom of human beings or something. Maybe, but and like I said, non-governmental There is such a thing as non- de facto non-governmental regulation in a way. And I mean, I guess if if it's regulation, it's kind of governmental, but the the there are ways to try to have it both ways, but it's going to be a very It's going to be like threading the needle.

[00:47:18]>> Yeah.

[00:47:21]Let me step back to like ask a different aspect of the big question of your book, and maybe the only thing which I didn't fully [clears throat and snorts] see where you were going with, and you can clarify things with me.

[00:47:32]>> Uh-huh.

[00:47:32]>> Which is There's a couple of big ideas in your book, and I didn't know how to make them go together. So, one is the idea, um, of, uh, of that we are there are forces driving us to become increasingly over- interconnected as individuals. This is something you did in the legendary TED Talk a while ago.

[00:47:50]>> Legendary.

[00:47:52]>> You have this nice quote from, uh, from Orwell's 1984 where somebody says, I guess it's Smith." He says, "Um can you not understand when Winston that the individual is only a cell?

[00:48:03]The weariness of the cell is the vigor of the organism." I get that and I kind of like that.

[00:48:08]>> Yeah, that's the totalitarian talking to Winston Smith, I think. Yeah.

[00:48:11]>> So, it isn't going to There's There's this growth.

[00:48:14]And then there's the idea of AI itself being a super intelligent that grows, communicates with other AI, and so on.

[00:48:21]I did not have connected two ideas. Like because the original idea we are parts of this broader system. And I can see that being a being working for say social media.

[00:48:30]>> Uh-huh.

[00:48:30]>> But I'm not part of ChatGPT. And ChatGPT isn't part of me. Is it the idea there's two independent systems growing and spreading or are we interconnected somehow?

[00:48:41]>> I think maybe should that be the cliffhanger that that gets people to cross the paywall, Paul?

[00:48:46]And and become >> will.

[00:48:49]>> and become paid subscribers either to Small Potatoes or to Nonzero.

[00:48:54]Uh uh or to uh And by the way, you know, through the magic of the uh Nonzero Network paid subscribers to either can uh get a 50% discount on a one-year subscription to the other.

[00:49:10]Uh there's a I know there's a way to to find that on our website and uh well, you can just put in a link. You You very kindly uh always send out a post when we do one of these. And you can just You can just put a link to the network uh page on that so your subscribers can do the same. In any event, uh you you, you know, paid subscribers to either can can uh watch or listen to the rest of this. And uh there's a link in our show notes and and and in in Paul's version of this. And uh the Yeah, and and of course we're going to have the cute baby face-off.

[00:49:50]>> The cute baby face-off.

[00:49:51]>> And >> And and we're going to talk to I I had this uh uh pod this sorry this Substack called Moneyball for uh publishers and Substack writers where I explored the mechanics of how do you promote your book.

[00:50:05]>> Mhm, mhm, yeah, yeah, yeah, yeah, yeah.

[00:50:07]>> And and I come off very skeptical. I say there is no secret sauce. There is there is no secret sauce. And and I talk a bit about that. So I wanted to to use you as a case study. You are about to promote a book. Um How do we do this? What's what or or is there no answer?

[00:50:22]>> Yeah. Well, one of the keys is you say to people right before your paywall, "Even if you don't follow us into the paywall, hey give me a break, do me a favor, pre-order on Amazon." That's one thing.

[00:50:34]Uh I mean >> is that is something you could try. That is something yes that is.

[00:50:38]>> But I would not do that because you know, crass self-promotion is something that just I'm allergic to. So I would not say something like, "Please, please, please, please, please pre-order my book." Like right before the paywall.

[00:50:51]Like, "Okay, fine, you're not a paid subscriber, could you at least pre-order my book?" I would never say something like that.

[00:50:56]>> Too much dignity, I think.

[00:50:58]>> [laughter] >> It just It just kind of I exude it, right? The You can smell the dignity.

[00:51:04]Um so anyway, subtitle is if the if the god doesn't do it, artificial intelligence and our coming cosmic reckoning. I mean, is that going too far, Paul? Was I No, wait Well, wait, answers after the paywall. Paul may We We may see a little flip here where Paul goes rogue on me and uh as they say in Pulp Fiction, I'm going to get I'm going to get medieval with your ass now or whatever it is.

[00:51:31]What's the line? I'm going to go mid >> Anyway, that's kind of kind of line. It's an unfortunate way to end this otherwise amicable conversation.

[00:51:38]>> [laughter] >> Well, I'm I'm not promising it. That's the suspense. Will Paul go medieval? We don't know. It's a mystery. Okay, but thanks everybody who's followed us this far. And Paul, thank you for suggesting this. Did I point out that you suggested this exercise in crass self-promotion on my part? I would I would thank you for that. I I >> I really enjoyed the book. I I think I think people I think >> nice. You really have. You've brought out the things that I want to bring out.

[00:52:03]You're saving the fierce and painful criticism for after the paywall. Exactly what I would have asked you to do.

[00:52:11]>> Glad.

[00:52:11]>> That's the kind of guy you are. And you know, by the way, everyone says you're a nice guy. Do you know that?

[00:52:17]>> No.

[00:52:18]>> Everyone. Everyone. It's kind of annoying, frankly.

[00:52:22]Cuz I'm not sure everyone says that about me. Do they?

[00:52:25]>> Let's discuss that in the paywall.

[00:52:27]>> Okay. All right. Thanks, folks. Here comes the overtime.