DEBATE: "Lumpenspace" (AI Accelerationist) vs. Liron (AI Doomer) — Is it GOOD for AI to replace us?

Added: 2026-05-09

[00:00:00]You would agree that the arc of intelligence bends toward morality, right?

[00:00:04]>> Oh, no, no, no, no, no. Wait. I don't know which kind of like morality or whatever this new intelligence will have. What I know is that it will not have boring goals.

[00:00:12]>> Do you see a lot of value in an outcome where the universe gets a tile of tiny paper clips?

[00:00:16]>> Of course, I don't. But that outcome will not happen. It's it's kind of obvious. No, like why why would they?

[00:00:21]It's boring. The famous example of the nanobots killing everyone on Earth. How big are these nanobots? I think that that thing will never exist. The the whole point is that we cannot predict how an ASI thinks.

[00:00:32]>> The chess analogy is like, look, I don't know what move it's going to make, but I definitely can model it as an agent that's going to win at chess.

[00:00:39]>> So So if you were a mouse, you would be like, I don't know which move this human is going to make, but I can model him as an agent that is going to win at taking cheese. That's kind of stupid.

[00:00:57]Welcome to Doom Debates. My guest today goes by Lumpin Space and his real first name is Claude. Lump inspace is a prominent technological accelerationist.

[00:01:07]I met him through Twitter. I've also seen him in person at the manifest conference last year in Berkeley. Lumped Space has organized a conference at Ligh Haven in Berkeley called the Delight Nexus for techno accelerationist thinkers. He also is a prolific Substacker at lumpedpace.substack.com.

[00:01:26]On Twitter, he has 22,000 followers. His background is that he studied math and linguistics in Italy. He went into finance and he recently ran a pop-up AI research lab in San Francisco. So, very interesting character. We tend to disagree a lot on Twitter. I'd say we're from different parts of Twitter, kind of different communities, but he was brave enough to come into the lion's den and hash out some of our disagreements face to face, virtually face to face. Lump in space, welcome to Doom Debates.

[00:01:53]>> Hey, thank you. Uh, thank you for having me and I'm looking forward to understand more about yeah, your side of things.

[00:02:01]>> Hell yeah. And, you know, it seems like we don't have that much in common based on our Twitter interactions, right? We take opposite positions a lot, but I will give you props for coming on the show because the level of discourse when we're here talking, I feel like, you know, where it's it's it's more civil and it's probably going to be more productive, right?

[00:02:16]>> Hopefully. Yeah.

[00:02:17]>> To kick things off, tell us a little bit about your recent conference at Ligh Haven because I know it was called Delight Nexus and it's like a play on the torment nexus because you're basically not worried about technology and AI being the torment nexus, right?

[00:02:28]You think it's more likely delight. So, so explain that.

[00:02:31]>> Yes. Okay. Well, so the thing is I noticed that there was a lot of talking about about AI and about risks and opportunities and there was very little experience in it and I thought that it would be nice to have a conference in which the the things were mostly hands-on, you know, and that's been it's been quite surprising the result. I I've seen people that never done like art or things like that doing videos with AI.

[00:02:55]This was thanks to Kiri/Kiranu uh that led the beauty workshops and I've it was it was a relatively small thing but I think that yes some people became friends and some people started collaborating and it was generally kind of a nice mixture between the accelerationist and and doomer faction.

[00:03:15]Those were good times >> and uh you know what does lump in space mean? Cuz I feel like you're you're getting a lot of mind share for that term. You know, people have heard who you are, but why are you called Lumpen Space?

[00:03:25]>> Oh, well, originally it was just a pan on the Lumpy Space Princess from Adventure Time.

[00:03:29]>> Got it. Okay. Yeah, I'm not familiar with that. Uh, okay. So, it's a reference basically.

[00:03:34]>> Yeah, kind of. Yeah. And also like, you know, I come from a relatively like nonorded uh kind of like background and I was kind of I thought it was fun to put it in my name.

[00:03:46]>> So, we have a limited amount of time. We want to get we want to hit on the the most important topics here. I think it's basically that you're an accelerationist, right? You're kind of on team Bef Jazos. Is that fair to say?

[00:03:57]>> Um I I don't think so. I'm on team intelligence in general. I want more intelligence to exist in the world.

[00:04:03]>> Do you have any major disagreements with the Bef Jesus EAC type of philosophy?

[00:04:09]>> Well, um no, not necessarily a disagreement. It's more than I I see my my view as grounded on on other kind of like axioms and I don't care much about thermodynamics and I care a lot about intelligence and evolution.

[00:04:24]>> I think my central claim that you might disagree with is that we're building AI that's posing a high risk of imminent extinction. Right? You disagree with that claim?

[00:04:35]>> Um well define uh define high risk and imminent and extinction. Basically, I have a high P doom. Uh, it's about 50% by 2050. Like, I think we'll literally just all be gone and dead then and the future will suck by 2050. That's currently my best guess. Let me ask you the same question. Okay.

[00:04:52]>> Mhm.

[00:04:59]>> Lumben Space, what's your Poom?

[00:05:02]>> Uh, my Poom I think uh just for AI, you mean?

[00:05:06]>> Yeah, let's do P AI Doom.

[00:05:08]>> Okay. Uh so I think that it's very very little. Um it depends on what you mean by doom. So if you mean um nothing of value exists in the universe then it's close to zero.

[00:05:19]If you mean um no no biological humans exist in the universe that must might be close to like 30 maybe.

[00:05:28]>> So you think there's a 30% chance that biological humans will be eliminated?

[00:05:31]Are you thinking 2050? Do you need a longer timeline?

[00:05:34]>> Well yeah uh I mean I don't know. No 25th is too is too short. I think around in a duration of two. The thing is like I don't think that they will be eliminated. I just think they will be superseded.

[00:05:46]>> By what year roughly?

[00:05:47]>> Um I I have I have no idea. I hope like not too late.

[00:05:52]>> I mean in terms of your ballpark timelines, right? Are you thinking like a thousand years or a couple decades?

[00:05:57]>> Oh no. I'm not thinking a thousand years less than a thousand years. I'm thinking like something like two generation tops.

[00:06:02]>> Okay. Yeah. So maybe by 2100. Maybe not by 2050.

[00:06:06]>> Yeah, I accept him.

[00:06:07]>> Okay. So, you think it's likely that we're one or two or probably not more than three generations from being superseded by a different race, but you see that as generally kind of fine and good?

[00:06:18]>> Yes. The reason being there are some things I value about humanity and very few of these things are the things that we share with monkeys.

[00:06:26]>> I mean, I I can I can definitely imagine that the future can be non-human. I mean, I consider myself a transhumanist.

[00:06:32]So if you tell me, yep, the future it's not going to have any biological humans, but it's going to have something that's kind of like uploaded humans, right? Or kind of like digital versions of humans or digital successors to humans. There's this term that I think Danielle Fella made up or or popularizes called worthy successor. In principle, I'm open to the idea that my descendants are just worthy successors. Like they're not, you know, they don't have to have literally my physical genes. I can identify with generations. Like I'm flexible, right? I can take a a global cosmopolitan perspective, but it's I think the devil is in the details, right? Because I think when people want to do that, like Robin Hansen is kind of like the champion of always being like, "Yep, it's fine. There are descendants." I think that when people do that, they instantly kind of open the doors and go down the slippery slope and suddenly end up endorsing like, "Yep, whatever comes and conquers everything in the next like 10 or 20 years is totally fine." Even if it's like very much like cancer, like low value. I think they open the door too wide.

[00:07:24]>> That I agree. Um I that's funny because uh there there's two there's two thoughts uh in in the sentence. One of them you would be fine with whatever manages to emerge and and dominate and the other one is that the thing would be cancer. I don't think so. I I don't believe in orthogonality. I I don't think that like intelligence and goals are orthogonal and actually I think that the more you increase intelligence the more your goals become even having more intelligence. And we we can see it like empirically from I don't know you you line up an ant and then a mouse and then a cat blah blah a person and you will see that the person is way more into into increasing their own intelligence and their own capabilities. And you can also see it because of a simple game theory thing. In the end, whoever increases their their capacity of understanding the world is going to have a strategic advantage that is kind of like unbeatable and all of the others will like, you know, if you decide to increase something else that it isn't intelligence, then you will be out competed.

[00:08:29]>> Okay. Well, this might actually be our main crux of disagreement. Like it sounds like you would agree that uh the arc of intelligence bends toward morality, right? That's basically your review because you deny the orthogonality thesis. You don't think they're orthogonal. Do you think that they're going to dovetail intelligence and morality?

[00:08:44]>> Oh, no, no, no, no, no. Wait. I don't I don't know. I don't know which kind of like morality or whatever this this new intelligence will have. What I know is that it will not have boring goals. It will not have it will not create a universe without values. It might create a universe full of things that we don't understand or even that looks like horrible to us, whatever. But they will be things that a higher intelligence will be able to understand that we can't. So if I were an astropeticist, yeah, I I would have had my my goals, my my morals and whatever and thinking about the future, I would be very very afraid of the super intelligence that will come later. And if I saw a picture of of of a of a modern city or of a museum or whatever, I would be scared shitless, you know. And I am very very happy that this austropi didn't manage to to kind of like align us because otherwise the world is would be far more radian and far more murder and instead we we do art and stuff.

[00:09:41]>> I I mean let's let's play out a scenario, right? I mean to me I think that if we are going too fast the way we're doing now um to I feel like you think the natural outcome is like well whatever we'll make some intelligent agent and it might not be moral. It might not be what most humans would recognize as a moral agent with good moral priorities, but it'll be interesting, right? And whatever it does, it's interesting. Like maybe it'll kill all of us humans. Maybe it'll do it in like this gruesome way that we think is immoral, but it'll have its own version of morality, which is interesting. And ultimately, that's fine, right? That's kind of your view, >> better than ours. And like if I think that morality and intelligence go together and I think that you can look back at your evolutionary history and kind of like be horrified at how brutal and simple the goals of the creatures that preceded were preceded us were. And I suppose that the fact that we cannot understand them like hobbling something that is able of of greater beauty and greater complexity than us just because we want to understand it seems really really mean and it seems like something that we wouldn't be happy if it was done before us. No, like would you have been happy if chromagnons kind of like aligned us?

[00:10:46]>> Well, yeah. But so what's what's wrong with with the description that I said though, right? is like we if we looked at it today we would recognize we would think that it's bad and you're just saying yes we would think that it's bad but it's fine because past humans would have thought that we are bad so then that's what I'm saying you think it's fine >> yes yeah I do think so yeah >> yeah so and then this is what I'm saying is like I actually think that if we bend what we think of as morality a little bit that probably is fine because I describe like past humans looking at us I think there's a bending going on right I think they'd be I like to use the example of like you know homosexuality right there's plenty of of ancient cultures and even some modern cultures, right, that that look at homosexuality and are like, "That's not fine." I'm very fine with it myself. I don't have a problem with it, right? And if you look at ancient humans, some of them would be like, "Wow, okay, there's homosexuals.

[00:11:31]You got to kill them, right?" I think even Sharia law today kind of has that perspective, right? Uh, but I think that the majority or at least a big fraction of humans, even in cultures that frown on homosexuality, they would see it as like a bending where it's like, okay, yeah, you've got a modern society that lets homosexuals marry and be respected, but ultimately I can see the appeal of the society. Like, I don't think it's horrible. I don't think it's disgusting overall. I think it's still a good society, right? Don't you think that most or a lot of human Yeah. So, so that's the problem is I think you're kind of bringing in too much, right?

[00:12:02]You're you're you're not just letting it bend. I think you're letting it bend like too much and snap.

[00:12:07]>> Um, interesting. So, you think that people from these cultures, people from the cultures that stuck at ISIS, people from the culture that are now kind of like semi- invaded in the UK that they they do think that we are a similar and interesting culture and that they don't they don't have any animals against us.

[00:12:28]Well, what I think is that if you could go back in time and and and survey humans, I think that 20% of all humans who have ever lived, if you give them a description of everything about our modern life, including the parts that their current culture and their morality would frown upon, like homosexuality, in many cases, if you just give them a full description, be like, "Look, this is what our life is like. What are your thoughts?" And they'd be like, "Okay, I don't like this part. I don't like this part, but you know what? It sounds like ultimately you guys are doing pretty well. You know, I think you guys have have been stewarding the future pretty well. Like I'm I'm pretty happy. I feel like that would be the reaction of at least 20% of all humans who have ever lived.

[00:13:03]>> I I I I do think that most of them would be horrified actually. But who would be far more horrified were were the chromagnons or or even the chimps, you know, like the difference and super intelligence is larger.

[00:13:16]>> You really think the chimps would be horrified by I mean the problem is that a chimp I mean chimps can you know there's I mean chimps aren't exactly our ancestors, right? They're like they're they're another descendant of our common ancestors, right? But if our common ancestor was chimplike, right? If a chimp is standing in for our common ancestor, I I don't think chimps look at humans today and they're like, "Oh my god, these humans are are so gross."

[00:13:36]They're just like, "Whatever." You know, like they they don't mind.

[00:13:39]>> They But they don't understand surely. Right.

[00:13:44]>> Right. So they they don't understand. So So what what analogy is relevant? Right.

[00:13:48]So to me, the most relevant analogy is like, okay, humans from 5,000 years ago.

[00:13:53]Okay. Yeah. Let's go with humans for some 5,000 years. But wait 5,000 years ago. You say that the jump you always say that the jump between us and the super intelligence is not the jump between a dumb guy and Einstein, but the jump between an ant and and a human. So why don't we go a bit back? Let's go back to like, you know, 1 million years ago. This these creatures would be horrified if if they saw a mother city.

[00:14:16]Of course, you you take any like Theosaurus, you make it fly around San Francisco, it's going to go crazy. Um, so yeah, obviously we don't >> Sorry. You make wet fire on San Francisco?

[00:14:27]>> Terasaurus?

[00:14:29]>> A terasaur? Like a pterodactyl?

[00:14:32]>> Yes. Kind of. Yeah.

[00:14:34]>> I mean, so but this is what you were saying before, right? Is like a pterodactyl. They're not really reasoning about whether they like the future. They're just kind of like executing their adaptations, right? Like they're not really reflective.

[00:14:43]>> But sorry, what how intelligent do you think the super intelligence is compared to us?

[00:14:49]>> I think it's much more intelligent. I mean, right. I think there's a, you know, you can describe it as being like thousands of IQ points. That's roughly my intuition.

[00:14:56]>> All right. Very well. So, so it's kind of like the same uh like ratio that there is between us and the rat, like not a rationalist.

[00:15:04]>> Yeah. But there is an important generality threshold, right? Like a rat just doesn't think about, hey, what is the world going to be like in a 100 years, right? A rat fundamentally can't ask that question. And and we can and you could argue, hey, there's some questions about string theory that I can't really ask. Okay. But realistically, there's just all there I would argue the majority of interesting questions are ones that most humans can ask that rats can't ask. Okay. And that's a very important threshold.

[00:15:26]>> That that is an incredible coincidence.

[00:15:28]>> Well, it's not a coincidence because it's it has to do with the the theory of computation, right? We know that there's these universality thresholds. And it's not a coincidence that human cognition crossed a certain univers universality thresholds and suddenly we're here asking all these interesting questions.

[00:15:42]a bunch of tape and uh and and the punch and the punch thing crossed the university threshold. But uh it's it's not really much of an argument. No, the thing is there is the possibility. We we don't know. The whole point is that we cannot predict how an ASI thinks. Isn't that the whole story? Yas always says you you will have no idea. And yeah, >> it's it's not as simple as having no idea. There are you can predict uh highlevel properties of what's going to happen when it right the the chess analogy is like look yeah I don't know what move it's going to make but I definitely can model it as an agent that's going to win at chess right which has useful predictive power >> so so if you were a mouse you would be like I don't know which move this human is going to make but I can model him as an agent that is going to win at taking cheese that's kind of stupid like >> well cheese isn't isn't humanity's goal right but if you're a mouse you could be like look this is actually a good analogy because if you're a society of mice and suddenly humans are coming in and they're living in the same place as you, it's a safe bet that anything that you have which seems like a resource, right, an instrumentally convergent resource. So for example, if you have land, right, you can probably predict that for one reason or another, the amount of land you have is going to diminish because humans are >> as a mouse.

[00:16:55]>> Well, I mean, if if you know a mouse is not going to is not a strong reasoner, right? But if the mouse had a little bit of reasoning power, right, the way we do, if it had a little bit of universal reasoning, then they could predict that their territory is going to shrink.

[00:17:06]>> But then if we if we if you're talking about this magical threshold as the threshold of reasoning before there's no reasoning and now there's reasoning and it just so happens to be when we're here, then you you're that super intelligence is not much far away from us really.

[00:17:21]>> So there there's two concepts that you have to hold in your head simultaneously. And I know this is confusing to people, right? Because there's David Deutsch is an example of somebody who notices that humans are universal in a certain sense, right?

[00:17:31]We're universal computers. And he calls us also universal explainers, which he says is a higher threshold than being a universal computer. Fine. But then David Deutsch gets really enamored with that.

[00:17:40]And then he's like, "Yep, we we've got it. There's really no other interesting thresholds to talk about. Like we we we're already good here." And I'm like, "No, no, there is there is such a thing as being like a much better, more effective intelligence than humans, even though humans are universal."

[00:17:52]>> But do you understand why David like thinks so?

[00:17:55]Like what?

[00:17:56]>> Yeah, I I understand because it's confusing for people to think that there is a univers universality threshold and yet we can still be easily crushed by ants by better thinkers. Crushed like ants.

[00:18:06]>> Yeah, I don't think that his preoccupation is or at least it wasn't with with super intelligence. It was more like with the general universalist view of human beings, you know, he didn't want to think that there were more than one threshold because this would have would have like made him think about group differences and stuff like that.

[00:18:22]>> Let's just recap where you and I are in the argument. Okay. So, you brought up what I think is a Hansonian position, similar to Robin Hansen, basically saying, "Hey, smarter agents are coming.

[00:18:30]They're going to do something, but it's going to be intelligent. It's going to be complicated. It's going to be interesting. And ultimately, it's going to be fine in the same sense that we should think modern humans are fine relative to ancient Greeks, right? Like, it's just it's just the continued progress of finness.

[00:18:46]>> We're not talking about ancient Greeks.

[00:18:47]We're talking about something. It's you guys that started with the with ant versus human, right? So something like that.

[00:18:54]>> Okay. Sure. I'm just I'm just trying to summarize your claim. Okay. So your your claim is that the future is going to be fine because whatever the intelligence chooses to do is inherently interesting and and it's a descendant as Robin Hansen would say, right? It's just automatically a worthy successor by virtue of being intelligent.

[00:19:10]>> I don't care whether it's a successor or worthy successor. I just think that there's not going to be a universe of value. Yeah. there's going to be far more value that we have now and a value of a type that we are not able to understand.

[00:19:22]>> Okay, so you know the Alia brought up the example of tiny paper clips as an example of a value that seems pathological to us, right? Like that doesn't seem to have any value even if the molecular squiggles look exactly like paper clips and the AI loves it, right? But that has like no value to us.

[00:19:38]So my question for you is to use an extreme case, do you see a lot of value in an outcome where the universe gets tiled with tiny paper clips?

[00:19:43]>> Of course I don't. But that outcome will not happen.

[00:19:47]There is >> so why why do you think that you know that we're not going to enter uh you know a positive feedback loop with a crazy value like that?

[00:19:54]>> Yes. Because the any intelligence Yeah.

[00:19:59]that is able to um overcome humans and tile the universe with paper clips will not tile the universe with paper clips.

[00:20:08]And it's it's kind of obvious. No, like why why would they? It's boring. Why? I don't understand.

[00:20:15]>> So this I mean this is this is a pretty firm denial of the orthogonality thesis, right? So you you claim that tiling the universe with paper clips is boring. Um so so maybe you can unpack that, right?

[00:20:25]Because I mean let me give you an example. Imagine that I just write a line of code, right? Imagine I just have a program that that's capable of plotting strategies to do anything and then I'm able to tell it what to pursue and I tell it to pursue tiny paper clips and then it goes ahead and pursues that.

[00:20:41]So, where in that chain of events does it pause itself and be like, you know what, Luron, I know I've been doing this for 20 years, but uh I'm getting bored.

[00:20:49]Right? So, you're kind of assuming that boredom exists in the system, but I didn't program boredom in the system.

[00:20:53]So, what's your account of events here?

[00:20:55]>> Well, you don't need to. Okay. So, first of all, we're talking about the system that is able to overcome humans, right?

[00:21:02]>> Is it?

[00:21:02]>> Um, yes. Yeah. So, let's go with the premise, right? That it's what I call a goal engine, right? It's just like this engine that's better at achieving goals than a human is. It takes the form of an AI, let's say. the goals in particular if they're social goals and ifol they involve other players at the very least as an amount of modeling the other players should be should be done right >> okay >> you should have an idea of what the other player does and so on so the thing is okay if you if you look at if you look at a string of of animals in increasing complexity actually put put a thermal start at the start you will notice that the more the complexity increase increases the more the goals become not only varied but also changeable. And at a certain point, more or less kind of like primates or maybe crows or whatever, there's the there's the idea that they can look inwards to their goals and and change them themselves. They can decide to change.

[00:21:54]Yeah. And and this is a function of intelligence so far. So intelligence allows you to have more complex goal. It allows you to change your goals. It allows you to revise them. And it allows you to to uh want to build more interesting things, more more it allows you to want more intelligence. That's the thing like we are we are some of the few animals that want more intelligence.

[00:22:17]There's also some smaller ones and and the thing is if you notice the smartest people that you know, they're also the people that are the most curious and the more interesting cultivating their own blah blah and so on and so forth.

[00:22:28]>> So to so to to repeat back, I'm just summarizing your point here, right? So if I if I claim that somebody could build an engine, an algorithm that's basically an agent, and you just tell the agent what to try to achieve, and the agent is just entering a loop, just always plotting strategic paths to get what you tell it to achieve. You're telling me that such an agent, if it's superhuman at plotting these paths of the goals, it necessarily will stop itself, right? Necessarily will step back and reflect and come back to you and be like, you know what, I would like to do something else. Uh no, I'm telling that such as such an agent will not exist like an agent that can they can only like obediently um do execute one action uh one specific action and and reach that specific goal. Yeah, it's not a particularly intelligent agent. Like I I don't understand there are two things.

[00:23:20]You want the goals to be fixed and you want the intelligence to be like superhuman. And I don't >> super human. Correct. these things can can fit together because one one you have intelligence in order to to choose your goals in order to to evaluate different goals and in order to to choose the best the best scenario you don't have intelligence for other reasons you know and um the I don't know are you worried about like dumb powerful optimization or reflective super intelligence >> a dumb powerful optim I mean I don't think there's such a thing as dumb power I think anytime you observe a system that's superhuman at optimization, meaning at achieving a goal. I I think there's there's no way that it wouldn't also be smarter than humans in any sense that we would call smarter.

[00:24:04]>> Yes. And and the thing is that you you're you're you're thinking about the system that needs to be like godlike in terms of means and insectlike in terms of ends. And that never you can see that in all of our evolutionary history and also in all the complexity of like currently existing human beings the the the ants become complex together with the increase in means that there's there's no way there could be other ways.

[00:24:30]>> Sorry. The ants become complex together with the increasing what >> the means are the the capacity to change the world. Okay. And the ants are to towards which you change the word. Yeah.

[00:24:41]You imagine a creature that has godlike means and that has like insect-l like ends. And this thing does not does not >> I see I see what you're saying. Okay.

[00:24:51]Yeah. I mean, so humans obviously are are the most intelligent.

[00:24:57]>> And I mean, it feels to me like some humans have primitive means, right? I mean, there's certainly some humans who dedicate their life to, you know, tasks that are pretty ro, right? Pretty, I don't know, low end on whatever spectrum, right?

[00:25:09]>> Mhm. And they also have primitive ends.

[00:25:11]Like if you if you go and take a look at the tenderline, the people on the on the on on I mean the the the various like pavement phone, they're not the the brightest. No.

[00:25:19]>> Yeah. But what but what about something like hey uh you know I just I just really like um uh sewing so I'm just going to become like a really good sewer. But that you know I wouldn't say that's I mean sewing is great but does that really represent the peak of human intelligence or I don't know. I mean you know maybe they do complex designs right but like what about like I I want to be like a a really fast runner so I'm just going to you know run so fast every day.

[00:25:41]Is is that really the kind of uh means or the kind of task that justifies human intelligence?

[00:25:47]>> No, of course. But I I I don't think that that there's the smartest humans are are are into I mean maybe some of them they are as a hobby or them are particularly like physically endowed so that they do it anyway. But I don't I don't I can't imagine >> Well, I I I think you're you're making this assertion that I'm not seeing that much empirical evidence from looking at humans, right? Because it seems like yes, humans are very smart and they can achieve goals, but it seems like a lot of us just select goals being like, "Okay, well, I just really want to do this." And that this that they pick is not like that. You can have a genius who's like, "Okay, yeah, I mean, look at Jirro. You know, Jirro dreams of sushi, right?" He's like, "Okay, he seems like a pretty intelligent guy and he just wants to make the perfect sushi, but don't you think that like he's not using his full brain power to make the perfect sushi?"

[00:26:29]>> I don't know. I think I think that he's doing the best thing he can do. I mean, there of course some humans might be wasting their brain power, but you know, the more intelligent they are, the least they do it. like you know you wouldn't imagine a funman to to to go like I want to be the best in the world as sudoku you know and and >> right well you know it's I agree there's some correlation right where if you notice that you have this amazing talent that's productive for humanity and only you can do it and it's also stimulating then you're more likely to do it I I agree but I think you're trying to generalize a principle that you're you're asking your principal to give you a lot of safety right you're saying this AI because it's going to be so smart it's never going to be like oh I love molecular squiggies that's never going to be enough for it. And I'm like, you might be surprised that some of these AIs do think that that's enough for it as a goal.

[00:27:13]>> Well, I mean, we can see it that the more the the current AI list, the more they become intelligent, the more their goals are varied. So, I don't understand.

[00:27:22]>> Okay. Well, what about this? Okay. Think about evolutionary dynamics. So, for me, the default outcome, a lot of people would claim this, the default outcome is just evolution continues, right? We're kind of in this pause. humanity has has such a huge tactical advantage over the other animals that we're kind of in this bubble where uh generations haven't kind of caught up to the new the new kind of warfare that's happening on planet Earth. Um so so we're in what Robin Hansen calls the dream time and it feels like evolution is stopped for a little bit. But if you let it run a few more decades or a few more centuries, my mainline scenario is that evolution comes back meaning like there is just an allout fight for resources and whoever survives and replicates the best takes the resources. Um, so if there's just a bunch of AIs competing, right, like let a thousand flowers bloom, don't you think that the AIs that win the competition are just the ones that pour everything into survival and replication?

[00:28:11]>> No. Uh, they are the ones that poured everything into intelligence. Obviously, like the thing is at the at the limit.

[00:28:18]>> Yeah. If you if you have more, you are the person who wins. But but apart from this, wait a second. Let's go back a second. I don't think the evolution stopped. Evolution has accelerated.

[00:28:26]Evolution never stopped accelerating from the very very start. Now it before it was only okay at the very start it wasn't even through DNA means it was like through through through dumb bacterial means then it became to uh through DNA means and it it continued this way and then it became through DNA and culture means when we when we started to be able to pass down cultures and then it became it became uh through written culture there was a different kind of like way to preserve things and so on and it went faster and faster and faster. You can see it simply by by the state of the world.

[00:28:59]>> Do you understand the sense in which I'm saying that evolution is has kind of been outpaced in the last century.

[00:29:04]>> Yes, I I do understand but I don't think it's I do understand but I think that at the same time it hasn't. The thing is that we are we are competing on different on different grounds and it's not much the genetic material that we that that is the driver of the evolution as much as much as the cultural material you know like and I I mean by the driver of evolution that if you look at the city right now and the city 50 years ago and the city 100 years ago so on it's it goes logarithmically down the the amount of change no and and that means that things are changing because you know how you measure the the the evolution of a species is how it adapts to the environment and how it adapts the environment to them and we are adapting the [ __ ] out of this you know so clearly >> yeah you know that's I mean now that you mention it you could be like you know what's become really evolutionarily adapted in the last 100 years like roads right so like roads don't uh surv they don't reproduce using DNA they reproduce using humans building them but like roads are now out competing humans and I think there's some truth to that to be like yeah if you look at the right unit of evolution it's still progressing Mhm.

[00:30:10]Yeah. There Daniel B Dennett had a cute thing that was like a a scholar is the way of a library to make another library.

[00:30:17]>> Huh.

[00:30:17]>> Sure. Yeah, I've heard that.

[00:30:20]>> Yeah. As a pun on the chicken one from uh was it Wilson?

[00:30:24]>> Yeah. I mean it's it's interesting because if you frame the question as like what's evolving maybe we can frame the question as like well which configurations of atoms right or configuration which patterns are being are taking up the most share right of of pattern space in in the physical universe. And that's why I'm saying like roads could be an example or artifacts that humans build, right? Or >> well the thing about roads is that yes roads I can see them more a result than an agent of evolution because the the existence of roads has a very relatively little decision power while the existence of books or memes or whatever is a bit more.

[00:30:56]>> Yeah, there's some sense in which you can identify that roads aren't driving the evolution because uh when you perturb them, right? like if you go destroy a bunch of roads, the roads probably they don't have much power to like go go fix it, right? To go bounce back.

[00:31:10]>> You really have to tell the story with with other agents. So, I don't know. I mean, it's complicated discussion, but like I still think there's some truth to the idea that when we look at the world today, >> we don't see uh we we don't see a coherent force trying really hard to survive and replicate. Like, if you just ask today, what is trying really hard to survive and replicate? It does seem like individual humans and human cultures have kind of dropped the ball like you know their eye just isn't on the ball of surviving and replicate and there's this irony to it of like yeah the name of the game is to survive and replicate but I don't meet many people or cultures who are like we are maximizing survival replication >> something is maximizing survival or replication you know this >> I mean by definition but the universe doesn't look like the universe looks like if you go to like the first 10 years of the NBA right the first 10 years of the sport of basketball professional basketball it's like yeah people are playing it right But they're just not playing it hard yet.

[00:32:00]>> It's true. Um but I we're playing it a bit harder. But but the thing is that the the thing that is driving evolution always is capital more than culture.

[00:32:08]>> So getting back to your point though, right? I mean in the context of like a doom debate, right? So in your mind you're like, yeah, you know, these AIs, they're going to be more powerful than us. Like you accept the premise, right?

[00:32:17]You accept the premise that they're going to be vastly super intelligent.

[00:32:19]Correct.

[00:32:19]>> Yeah.

[00:32:20]>> In in in a >> That's an important point of agreement.

[00:32:23]And it sounds like you also accept that this will happen within a 100 years.

[00:32:26]You're not confident about like 10 or 20 years, but you're saying within 100?

[00:32:29]>> Nope. Uh, no. The super intelligence mean meant like ASI. I have absolutely no estimate. I >> I'm pretty confident it's going to be under 20 years because the pace of progress has been so astounding to me that I'd be shocked if it's more than 10 or 20 years, but I'm not, you know, I'll still leave 10% chance at least.

[00:32:44]>> Of course, I think that ASI is kind of underdeed. Uh you think that ASI is going to be there in 10 to 20 years? You said >> well I think that it's, you know, it's it's tough, right? Because I don't think that I have that much insight. Right? I can only the main reason I say that is just because I look at the human brain and I'm like why should we have an advantage, right? What's the secret sauce here? And I'm like, well, we had a lot of iterations, right? We had a lot of feedback, consequentialist feedback during evolutionary time. But then I look at AI and I'm like, well, they have feedback, too. And the feedback's coming much faster and more money is being invested, more energy is being invested, more optimization is being invested, right? I mean, human generations, they just that wasn't that much optimization pressure relative to the the optimization pressure that all these companies are applying. That's, you know, the the designer, the god of AI is us, which is a much more powerful god than the god of evolution, which is just, you know, dumb generations and and fitness uh, you know, reproduction frequencies. That's like a dumb god relative to the god of these people of the Sam Alman is a smarter god than evolution.

[00:33:39]>> I kind of think that they are the same god. And uh, >> okay.

[00:33:43]>> Yes. Um, I mean, I do think that at the moment there there's blind optimization for capital and that results in more intelligence. Yeah, but it's but you see what I'm saying where where the the mind of the AI it's not just being fashioned, you know, one generation after another like what's going to happen? No, there's humans being like poking at it being like we're going to shape this, right?

[00:34:03]There's actually engineering happening in a way that that that you know natural selection is not exactly an engineer.

[00:34:07]It's more like it's pure trial and error.

[00:34:09]>> Yeah. No. Uh like I mean cultural evolution is also kind of like part of evolution as we said up until now and and people have tried to guide it as well, right?

[00:34:18]>> Yeah. Okay. Sure. People guiding evolution. Sure. But that's, you know, there haven't been that many generations of people smartly guiding evolution, right? I feel like nobody really has time like farmers, ancient farmer, subsistance farmers weren't doing that much guiding, >> of course. But I also kind of think that that we we've been on the same exponential from the start. Uh, in terms of complexity, we never left that exponential like from ammo to to to lions to to to dinosaurs.

[00:34:42]>> Okay? But but you see what I'm saying, right? That the generations of AIs, right? AI now versus two years ago, right? Part of why it's been advancing so much is because human engineers are looking into the future, right? They're plotting paths to the future of AI progress and they're accelerating AI along those paths in a way that evolution never did.

[00:35:00]>> I really don't think that that any of the progress like I mean one thing I noticed in all of the various like surprising language models is that they never were on purpose. like kodinc2 for instance there was like the first language model you could actually talk to was was supposed to just do code was just trained on GitHub >> right look I agree that there's eureka discoveries right so there's a combination I'm not saying that that humans know exactly what's going to happen but they're injecting a lot of knowledge as well in a way that evolution doesn't and remember the point I'm trying to make here is that when I look at how we got our powers as humans okay we got it through a process right so because this is this is an important reframe because if you just compare the object versus the object Right? If you just compare the clump of neurons versus the AI, then it looks murky because you're like, look, we have so many neurons. They're connected in such an interesting way that we don't fully understand. It looks murky. But then when you reframe and you say, okay, don't worry about object versus object.

[00:35:53]Work worry about creation process versus creation process. When you look at it that way, then it's like, oh, holy crap.

[00:36:00]Like, our creation process is dog crap compared to the creation process of of these AIs.

[00:36:06]>> Okay. I I do I do think that the process is still variation and selection and the fact that uh men like some agents instead of some other other agents do it is not much of a of a difference because you know men don't people don't know much about about how to make AI and and and I don't know I just don't think it's particularly important. I know I know that it's going faster. Of course, it's going faster, but it's been going faster since since forever. Like if you take any time, they're so similar.

[00:36:37]>> The disagreement.

[00:36:37]>> I mean, look, on some on some level, you agree with me because you're saying 100 years, you're not saying 100,000 years, right? So, >> so even though I I say 10, you say 100.

[00:36:45]>> The only thing is uh I the thing that I don't agree on is exactly what ASI is.

[00:36:50]Because if by SI you mean the thing that can uh like change molecular structures of everything a random like in the in one of the famous old examples so you there's this uh film situation in which there's someone in a in a garage that makes them AI and then the AI kind of like hides infiltrates data centers and then starts making these nanobot factories no and a certain point bam everyone drops dead thanks to these nanobots. Yeah, I think that this will never exist because I believe that sensitive dependence of initial condition exists and that dissipative heat exists and the physics exist. So, >> okay, so this is another point we can argue about, right? Because I I'll just clarify for the viewers. So, my position is that the universe is highly engineerable. So, what you say sensitive dependence on initial conditions, right?

[00:37:35]Also known as chaos, right? Or unpredictability. Uh what I see in the universe is like yes, there's absolutely chaos. There's unpredictable chaos. And yet it's possible to draw boundaries of being like, "Yep, I know there's chaos, but I can create a zone that sufficiently avoids the chaos that I can walk to my desired outcome, right? Like getting to Mars, there's so much chaos, and yet I think a Mars mission will work."

[00:37:56]>> Of course, I I agree. I agree fully. And the thing that uh the problem with the creating the nanobots that unseen attack the whole uh Earth population at the same time, boop, is that you didn't create that that bubble. is that you have to contend with with the complications of a society continuing to work as usual of humans being complicated of biology being complicated and stuff like that and you won't have this sort of like level of predictability I mean even if you had sensors every meter everywhere on earth and the most powerful supercomput you could only predict the local weather for one month you know and at a certain point the amount of information the the bottleneck is not the amount of information you can process but the one you can acquire >> well you know what though I I there is a strategy that even though it's you're correct that weather is chaotic and even I'm actually open to the idea. I'm not sure, but I'm open to the idea that even a real super intelligence that can conquer galaxies, even such an intelligence might have a lot of difficulty predicting the weather more than a month out on Earth. I'm open to that claim. I think that may be true.

[00:38:55]And yet, I also claim that such an AI can predict the weather anyway. You know why?

[00:39:00]>> Mhm.

[00:39:01]>> Because if it perturbs the weather in a certain way, I claim that it can get the weather into a less chaotic can do that.

[00:39:09]Freeze everything. Yes. But the thing is the the the the famous example of the nanobots killing everyone on Earth. They were unseen and they they were simultaneous. You know, I think that that thing will never exist simply. And I think you agree.

[00:39:24]>> I mean, no. I I think nanobots absolutely I think undetected nanobots can kill everybody because and the reason is because I don't think we're operating at the the engineering level of the universe. You know, like it's like, you know, imagine a 5-year-old who grew up in a primitive society. That 5-year-old would have no idea what adults in the year 2026 are capable of engineering.

[00:39:42]>> Yes, that's true.

[00:39:43]>> Right. I think there's an analogy there.

[00:39:44]>> But there is a thing that simultaneously all people on Earth unseen is a weatherlike situation. You know, >> it's a thing that you have to decide where the information goes. Does it go in the nanobots? How [ __ ] big are these nanobots? You know, there is there is >> there's plenty of information compresses pretty nicely.

[00:40:03]>> No, no, no, no, no, no. You don't compress the information. position of every person on Earth in in in a in a like two atoms of carbon.

[00:40:11]>> Yeah. You you really think chaos is going to protect our species?

[00:40:14]>> No, I don't. Who said that?

[00:40:16]>> Okay.

[00:40:16]>> I'm just saying that that kind of ASI the kind of ASI that has this sort of control over humanity. I do not know when it will appear because I don't think they will ever appear because it cannot physically appear. But instead, if by ASI you mean the sort of thing that can eat a planet, Yeah. then it will certainly appear.

[00:40:32]>> Yeah. Yeah. But but why I'm still trying to understand though like why you know Yudkowski claims which I agree with that there there's very likely to be nanobots, right? Nanotechnology is just like an obvious tool that AI is going to grab, right? It's like the lowhanging fruit on the tech tree if you have enough intelligence. Um so and why why do you not think that's going to happen?

[00:40:48]Why why do you not think AI will rapidly get to nanotechnology?

[00:40:50]>> I'm I'm not saying that AI won't get to nanotechnology. I am I am disputing one one specific scenario which is an AI that gets to nanotechnology unseen.

[00:41:00]Yeah. And that's this technology attacks all people of Earth in the same second and kills the moon.

[00:41:06]>> Okay.

[00:41:06]>> I'm saying >> the same second is a little tricky. I'll give you that. Okay. May maybe it'll be the same week.

[00:41:11]>> Uh yeah, the same week is kind of a complicated thing because you know like the the thing is information itself has a size you know that you need to you need to you need to imanize it in some in some substrate and nanobots are really really small. So the thing is are they self-guided or are they guided externally? If they got it externally, who's going to go back to them with these magnets? And like, you know, there are just simp, you know, I believe.

[00:41:36]>> You don't think the AI can coordinate the nanobots worm?

[00:41:39]>> How how will the communication happen?

[00:41:42]>> I mean, look, we're talking about an AI that's capable of making the nanobots in the first place. Okay. I think it can solve it can put a few, you know, uh, uh, antennas around. Okay.

[00:41:50]>> I think it can solve that.

[00:41:51]>> So, then then we have we have we have a problem. We have a problem of radio waves and we have a problem of interference.

[00:41:56]Like, you know, it's it's just >> Yeah. So, you think humanity can respond to the nanotech communicating on radio frequencies?

[00:42:02]>> Did I Did I say that? I don't think so.

[00:42:04]No, I'm just >> I just I I just don't see your fundamental objection because what I'm understanding so far, I don't know if there's a piece I'm missing here, right?

[00:42:10]Is you're saying, hey, there's the chaos, but you kind of admitted that, okay, chaos is not going to protect us.

[00:42:14]And then you're saying, okay, how are they going to communicate and coordinate? I'm like, I don't think it's that hard, >> my friend. I'm not talking about protecting us. I'm I'm I'm not thinking about that. I'm just saying that a super intelligence has some limits. These limits are given by physics and these limits would not allow to imitize the scenario that Yukowski describes in his in his diamond nanobots.

[00:42:36]>> Okay. So instead of even taking Yowski out of the equation if I say hey then nanobots have a whole week to kill everybody. Is that a realistic scenario?

[00:42:44]>> I I don't understand why. Uh okay it it depends like if the world is full of nanobots and if all the energy in the world is directed to nanobots but at this point people are >> not all the energy in the world it's modest you don't need a lot of energy to run you know a few billion nanobots or a few trillion even you don't need a lot of energy like you know this these guys have to move kilome and kilm >> yeah they have to move but they're still they're ridiculously efficient it's like okay yeah a big swarm of insects okay wow yeah make a couple power plants like you're good only need a couple power plants worth of energy to wipe out humanity Insects, insects are fine because they die. Like if if you want the nanobots to be squishy, to be biological, like then it's fine, but then they're not they're not that immortal, you know? They they have this decay.

[00:43:26]>> Okay. So So you're telling me that like the energy supply chain or the energy generation or the energy requirements of nanobots is going to be like the bottleneck that saves humanity.

[00:43:34]>> No, I'm not saying that it saves humanity. I'm I'm not I don't We're not talking about that. I'm just trying to define the concept of AGI of ASI. Uh for me ASI is a thing that cannot do that.

[00:43:46]It doesn't matter. Humanity can kill it in other ways.

[00:43:48]>> But but you're saying it can't do this problem that doesn't seem hard. So you're saying ASI can't solve the power generation challenge.

[00:43:54]>> It's not the power generation. It's the power generation communication and movement. They are things that exist. I don't know like you should get some engineers because it's it's insane how much you handwave away things that are like actual >> there there's a way to handwave smartly, right? So, I'm saying, "Hey, yeah, if you gave a super intelligence this problem, I bet they could solve it."

[00:44:14]Like, because there's such a thing as pointing out a problem, being like, "Look, I personally can't solve it." And yet, I'm able to be confident that it seems solvable.

[00:44:21]>> But there are some things such as limits of information that are not dependent on how smart you are. There there there is there are such things such as like physical like transmission of data that is not dependent on how smart you are.

[00:44:37]There are some things like movement that >> okay transmission of data you're citing is that but that particular thing that's that's the big one for you.

[00:44:44]>> No the big ones are is a combination of all of these things. How many nanobots will you have like you you know that sorry will the nanobots be controlled singularly?

[00:44:53]>> Well look if the objective is I mean I mean there's you know it's a combination right there's no simple answer. I a central control is a nice simple architecture right I think there there probably will be like an updater system or there will be a phone home mechanism but I think there will be a lot of layers right I mean think about think about like an army right it's like the units are kind of independent >> you know you have to get a license in case you want to operate a raid you know that's for a reason is just not infinite >> and if you have 1 billion nanobots you you overshot it by by by 10 times already you know the usable >> right I mean so look I I hear you but don't you think you're talking like imagine somebody who's like you know a a city a regional chess master not like an international grandmaster and the regional master is being like listen this international guy right Magnus Carlson do you realize that my queen is protecting this area right it's like you're talking strategy here in a way that I don't think you're going to this isn't going to hold up >> it's like if you told me hey what if Magn Magnut Carlson would say okay but look I roll snake eyes so you die that's what you're saying you know it's not it's not about the >> yeah so you think you have principles, right? You you think you know enough about like abstract principles or limitations, right? You have like limit theorems basically that you think are on your side, but I'm telling you that the specific things you're saying right now, like, oh my god, the radio spectrum, like I don't think that's going to be the actual guardrail here.

[00:46:12]>> I I don't know if that is going to be the actual It doesn't doesn't matter.

[00:46:15]Like if you >> if you want to have realtime communication, you need to move data.

[00:46:22]However, you need to move data. If it's laser, then you need space and you can't have nanobots one over the other. But I don't want to discuss this. Like I'm just saying I I think that that scenario I I can write an essay about it later, but but it's just >> I I want to define ASI as as a thing that can do all of the things that are physically possible and no more.

[00:46:42]>> Okay. I mean, that's not exactly what I would define, but anyway. Okay. So, we're getting in this rabbit hole and we we do got to head toward the wrap up here. So, let's let's let's try to put everything in context. Okay. So, I still think the crux of our disagreement I guess maybe there's two cruxes, right?

[00:46:53]is that you have a different mental model of an ASI because my mental model of an ASI is basically the Yudowski nanobots. And for viewers who are like, "No, I refuse to believe nanobots," I actually think that's totally fine because if if the viewers only want to say, "Hey, you can only use technologies that are less in their infancy as nanotechnology, right? You can only use like more mature modern technology, 2026 technologies." I'll be like, "Fine, you can still control humanity and become a dictator just by using the internet really well." Essentially, I think there's already plenty of actuators and tools and causal mechanisms even with 2026 era technologies to just steadily become, you know, the the ultra Hitler, the mecha Hitler as they say. I I think an AI could do that um just by pure psychological understanding of humans with today's technology. So, I think there's many ways to win if you're a super intelligent AI. But that could be one crux of disagreement between me and you, right? Is like how nanotechnology smart can be.

[00:47:43]>> Let me let me let me apologize. So first of all like radio spectrum thing okay it won't save humanity okay and what I'm saying is simply that there's the youth scenarios very often treat like the the the implementation details as if they vanish under just by saying super intelligence no but implementation details as do exist and there were where physics live you can't you can't end wave everything at all by saying super intelligence okay fine now ASI can solve a lot of engineering problems they can't >> but that does not license arbit arbitrary like covert global nanotech in one in one step you know there's many problems of course the ASI could solve it they cannot but that doesn't mean that they can solve like any arbitrary problem and in the case of Magnus Carson like Magnus Carson cannot move through track you know like it's it's >> yeah thing >> okay I hear you I hear so look so so I'm saying this is so let's let's put a pin on that okay this I've identifi we've identified this is a crux of disagreement right how powerful ASI can really be but we both agree that it can be superhuman in every relevance That's great.

[00:48:46]>> All right. So, that that's important.

[00:48:47]And then I think we have an even bigger disagreement though. This idea of like the AI pursuing things that are interesting, right? You just can't fathom a scenario where all the AI wants to do is have AI sex all day, right?

[00:48:59]Meaning just reproduce itself. Like, I can't wait to make the maximum possible copies of myself. That's all I care about. And for you, you're like, nope, nope, nope. That's just too uninteresting. It's not going to happen.

[00:49:08]>> Um, I mean, generally like species or like individuals within pieces, they're not the smartest. like are selected.

[00:49:18]Yeah. So I don't think that that's >> so so I mean yeah I think as as you described right rejecting the orthogonality thesis rejecting this idea that you can just have a goal engine and the goal engine can route to any goals even goals that seem uninteresting.

[00:49:30]You're like nope the goals are I predict the goals are going to be interesting.

[00:49:33]That's basically your your position.

[00:49:34]>> No no it can route to uninteresting goals. The thing is that if he does it, it will be out competed by by by other agents that instead point towards intelligence and increased and they want to increase the intelligence. Yep. Like if you if >> what what if you have an agent I claim you're going to have an agent that does want to increase its intelligence as a purely instrumental goal until it gets to enough intelligence that it could be like great now I see how to conquer the world in the name of paper clips or in the name of copies of myself. And then it doesn't need more intelligence after that point. It just needs to undermine the intelligence of all of its competitors.

[00:50:06]Sorry. What? Like like I I I hear this a lot of times, but are you for like is this is this a serious like objection?

[00:50:14]You really believe that?

[00:50:14]>> So I I think yeah, if your goal is to survive, if your goal is to defend the universe for copies of yourself, then yes, you absolutely need to have an arm that researches how to make int, you know, maximize your own intelligence and you want to defend the perimeter of your expanding sphere with, you know, intelligence sentinels and you want a research organization looking at the nearest aliens, right? So you can fight the nearest aliens. You want to do all that stuff, but then in the big center of your domain, you really do just have like dumb copies of yourself, right? You don't waste intelligence doing things unless you explicitly want to do them.

[00:50:46]>> Okay. Well, very well. So, so that's the lyric I think. No. So, you're asking whether a stupid objective can lock in before a more reflective or uh intelligent seeking system out competes.

[00:50:56]Right.

[00:50:57]>> Right.

[00:50:58]Well, but I just wanted to to make sure that we understand each other's crux of disagreement, which is that I see a system that uses intelligence as an instrumental value, right? It instrumentally wants to be intelligent.

[00:51:08]It instrumentally wants resources ultimately just to do something boring and uninteresting, right? Like like making copies of itself and maybe the copies have paper clips in it, right?

[00:51:17]They have some kind of payload. And you and I just don't care. We think it's a real shame that the whole universe is being used to do that, right? That's my scenario. And you're pushing back on that scenario. You're saying it's just so unlikely because it's not interesting. No, no, no, no, no. I'm pushing back in the scenario because okay, in order to acquire intelligence, Yeah. you need to want to acquire intelligence, right?

[00:51:36]>> Yeah.

[00:51:37]>> Yeah. And the thing is, have you seen you noticed that in in our evolutionary history, the more interesting goals come with more intelligence? Have you noticed that for instance, humans now, they they they [ __ ] as a hobby, but they don't they don't necessarily reproduce, right?

[00:51:52]>> Correct.

[00:51:53]>> Yes. And and and that's kind of weird.

[00:51:55]That kind of like goes against your your theory a bit. No, we we we want to tie the universe with with us, but instead we're doing like stuff like wearing condoms and and >> yeah, we talked about this before, right? Is that we're not we're in a stage of the universe right now, right?

[00:52:09]Where there's slack in the system, right? We're like people who are or you know when when the fast when the 4minute mile hadn't been broken yet, right?

[00:52:14]That's the stage in the system. There's it's always hell. And by the way, it's it never stops. And and you you should anyway not because the thing is not the interesting is magic but it's intelligence acquisition. It's not a neutral act, you know, like >> because it's instrumental, right? This is what I'm saying. Intelligence acquisition is in it's a resource.

[00:52:34]Intelligence is a resource.

[00:52:35]>> Wait a sec. It's not neutral in in another way. In that the the system that improved recursively. Yeah. I have to select for for cognition and syncing sub goals, right? Better models like better abstractions, better corrections, self understanding, you know, and that changes which kind of structures are stable. Yeah. You >> Okay. Sure.

[00:52:55]>> Yes. Okay. And then that's so >> but it's stable. It's stable to want paper clips and to instrumentally augment yourself to get the paper clips.

[00:53:02]That's stable.

[00:53:03]>> No, no. But the thing is the concept itself changes what you are. A copy maximizer that never wants anything except like more and more copies of itself or more paper clips is not my picture of an open-minded super intelligence. You know, it's a replicator exists. They exist. But historically, the most successful replicators are the ones that can generate intelligence because intelligence is a domain strategy.

[00:53:21]>> Okay. So is that your most powerful argument is trying to you use an empirical argument from what's been intelligent historically? That's that's the the greatest butress of your case.

[00:53:30]>> Well, absolutely not. And and and basically um so I'm saying that a a copy maximizer that doesn't want anything more than copies. Yeah. Will not be more intelligent of something that wants intelligence. Right. I don't think that the instrumental goals and the and the terminal goals are are different things.

[00:53:50]You know I think the nature has always used has always used instrumental goals uh sorry has always terminized instrumental goals you know like >> nature has done that right but that doesn't change the fact that some goals are inherently instrumental and some some aren't right so so somebody who doesn't start out caring about paper clips doesn't suddenly adopt the paperclip goal somebody who doesn't start out caring about intelligence will realize they better get intelligent if they want their other goal >> yes but some of these goals yeah change the structure change the stable structure and some don't. Uh wanting to have paper clips. Yeah. It's it's a very very um feeble and and kind of like ephemereral goal like replication is the substrat substrate level game. Okay.

[00:54:33]Like replication or wanting paper clips and intelligence is the winning strategy inside the game. No. Now why would the winning system like freeze at the copy part instead of continuous uh climate strategy that made it win you know why why would that happen? That's that's them. And you you always assume also that there's only one big super intelligent, but but it looks like for instance, if you look at the lab, they're they're progressing kind of like at the same pace, like at least four or five.

[00:55:01]>> Okay. Okay. All right. So, yeah, look, I we're out of time here and I I don't think that I put a really precise pin in exactly what your position is and how it differs from my position because it sounds like >> um we might you might have a little push back on the way I describe instrumental convergence, right? and and uh what instrumental versus terminal goals >> what was that >> I I just >> yeah so there was some push back there and you're about to you were about to introduce what I see as a new topic which is you're saying like hey there's group dynamics right so I'm not sure that I've had time to identify the exact crux of disagreement between us but if you want how about I'll just move to you know you can give me a closing statement right the the disagreement the way you see it right and like maybe what would what would kind of change your mind or what you think would be the biggest thing to help me change my mind and we can leave that >> I think you will change your mind I think that you are like very much into like socially like stuck at this point.

[00:55:52]But but but >> no, I I I resent that. Okay. I I like to think that um I just look I I'm transparent about why I believe what I believe and the reason I find the crux of disagreement on these shows is because I'm inviting you to convince me to believe the crux differently.

[00:56:06]>> I I thought the same before before the the recent like events and your reaction. But anyway, uh you assume that the goal is fixed through arbitrary self and improvement. And I'm assuming that self-improvement place a pressure on the goal architecture itself. You know, the the the self-improvement thing changes the goal architecture and as we've seen happening in all creatures that can do self-improvement including including LLMs. So that's kind of like mine.

[00:56:33]Do you want to uh recap uh where you see in this discussion where you see you and I our position fundamentally differ?

[00:56:40]>> You you think that um you think that um an agent can follow a goal of increasing their intelligence and they can um do arbitrary self-improvement while keeping the goal the same. Yeah. Like I do think that improving and and becoming more intelligent changes change your goals changes your goals.

[00:57:02]Okay. All right. So, I think I mean I think it's fair to say orthogonality thesis, right? The idea that uh what what you just said I think is a good uh a good way to define you thinking the orthogonality thesis is false while I think it's true.

[00:57:14]>> Yeah. You don't think that that um increasing capabilities and self-improving changes your goals?

[00:57:20]What's the point of self-improving? Then >> the point of self-improving is that you you become more powerful at achieving your goals, right? So for so for example like if my goal I is to um you know have the coolest mansion right is to purchase the highest price billion dollar mansion. That's my only goal. Okay. I don't even care about self-improvement.

[00:57:38]I like who I am today. Would I self-improve? Of course I'd self-improve into a better businessman than Jeff Bezos.

[00:57:43]>> Absolutely. And no and no I understand.

[00:57:45]Actually, you might be you might be entirely truthful in this because if your goal is to debate and and and stop AI, you would you change your goal when you when you learn new things? No, you would self-improve to be better at stopping AI, whatever whatever the new data would be. That makes sense, >> right? Okay. All right. All right. Well, listen, this I mean this has been productive and I'm sure we can go longer because we've identified all these other topics from debate, you know, all these threads we could pull, but this has been like such a nice overview. It's been a civil debate. You know, at times it can be frustrating, but it's object level frustrating, right? I'm not frustrated with how you came onto the show and you had a good faith debate and you, you know, you weren't trying to um you you weren't doing anything that I consider a lowblow. Like you weren't Well, actually, you know, there was that time when you said that you don't think I can change my mind. That's that was, you know, that was uncalled for in my opinion, right? And I'm not saying that about you. But besides that, I thought you were engaging good faith. I I made sense with your theory, you know, because your goal is to stop doom and if you improve and if you learn more things, that's always the per in the in the goal of stopping doom. Well, >> okay, fair enough. I'm going to allow it then. I'm going to allow it then. Yeah.

[00:58:48]So, so yeah, I appreciate you coming on the show. You know, come on back for a round two sometimes. Um, yeah. Any last words on your part?

[00:58:55]>> Uh, well, no, nothing. Um I I have a I have a little collection of things about antiarchality that they the post on Twitter often is kind of interesting and yeah don't don't do not throw like fire bombs.

[00:59:07]>> Yes. Uh so Lumpin Space and I both agree if you listen to the last few episodes of the show don't do any violence don't throw any Molotov cocktails that neither of us recommend that anybody should be doing that. In fact we are very much against it.

[00:59:19]>> Mhm. Yes. All right.

[00:59:22]>> Great. All right. Lump in space.

[00:59:24]Everybody follow x.com/lumpinsspace.

[00:59:27]You can get more of his perspective.

[00:59:29]>> Thank you so much for coming on Doom Debates.

[00:59:31]>> Thank you. Bye-bye.