AI alignment research requires pursuing unconventional, high-risk approaches that mainstream methods overlook, as demonstrated by AE Studio's gradient routing technique which routes dangerous capabilities into separate experts during pre-training, allowing them to be ablated while preserving safe model capabilities; this approach addresses the fundamental challenge that current safety training is post-training and vulnerable to jailbreaks, requiring instead to embed safety into the model's architecture itself.
Install our extension to search inside any video instantly.
Neglected Alignment Ideas, Factory CTO, and Forgotten Intent
Added:Okay, good morning. It is Thursday, June 18, 2026 and we have had a very eventful uh 24 hours. Nathan, good morning.
>> Good morning, Picash. How are you?
>> I am very good. And uh what has struck your attention?
Well, I think the whole AI world, from what I saw in the timeline last night, was just blowing up over MidJourney Medical's new uh Revelation and the beautiful launch video that they used to show off some of their plans. Uh I imagine everybody who's tuning in has seen this already so I won't u you know try to belabor it too much but in short they are planning to do an ultrasound uh whole body scan that they're planning to productize in kind of a wellness sort of vibe uh where you will be for just a minute like literally one minute uh they say you'll be dunked into a tank uh of water and the water is needed because the ultrasound doesn't pass through um it doesn't propagate well through air.
So you get dropped down into this pool of water. There's a ultrasound all around you that emits ultrasound in kind of a rotating fashion, bounces off of you in, you know, the full 360 degrees and is captured and is interpreted. And maybe one of the most striking things about what they showed is just really beautiful visualizations, which obviously is right in MidJourney's wheelhouse. I I I thought this was above all a really great example of an AI company putting forward a truly different but fundamentally positive vision for the future. I so often say the scarcest resource is a positive vision for the future and this is one positive vision for the future that it seems like everybody could get behind. U and so I absolutely applaud them for that. the idea that you could get a one minute scan and that you know the there could be radical abundance in seeing inside our own bodies and even make it beautiful in a way that um I think has actually maybe more medical consequences than immediately meets the eye. Um we can get into that maybe a little bit more too. I thought it was awesome, beautiful and uh inspiring and just exactly the sort of thing that we need more of. I I would love to see more of this kind of stuff come from other people who have uh suddenly got very wealthy in the AI boom.
>> Uh what struck me was I I I had a lot of thoughts on it. Number one, I love David Holtz. Um he's he's definitely been one of my favorite people in in AI for a very long time.
It struck me that he had used the wealth that he'd obtained from Midjourney in such a as as you pointed out a positive way. And it also struck me that this is only the beginning because one of my thesises is that there's going to be a capital explosion as all of the money that's being made in SpaceX and anthropic and open AI is going to go into the hands of these fairly young uh fairly young uh very empowered very technologically sophisticated entrepreneurial uh people. and they're going to be able to deploy this capital in ways that the normal pension fund trustee would not like or would not be able to support. And to a large extent, risk-taking in the United States and globally is driven by pension funds and uh insurance or organizations which are both regulated by the state. And a lot of bets are not made for this reason. So this is one of the things that I'm very excited about that as the wealth devolves into these hands which are able to make these large bets we're going to see more of them getting made. So that's that's one one obvious thing. The second thing that struck me was uh he decided to go into medical. Medical is 18% of the US US economy is medical. Um and it's always a growing percentage because whatever wealth that we have we end up using to improve our health at the end of the day.
What struck me is that it was it's always been a very hard um area of u area to innovate in because of regulation and because of privacy issues and because of all these issues. So he decided to tackle the hard problem. uh kudos to him and as you have pointed out before the medical establishment is not very on board with this. One of the one of the things that happened during COVID for example uh at the early stages of COVID in Seattle the um the labs that were testing for flu there back tested for COVID and they found examples of COVID even before the official COVID announcements. They wanted to alert the people who were positives but were banned from doing so by the FDA because the FDA doesn't like it when you don't get consent for from patients before you test for something and they don't like it when you disclose to patients something that you haven't kind of ethically discussed and decided that this is okay for disclosure. So the FDA actually banned these labs in Seattle from disclosing to the patients. And this is one of the things that has always struck me about the US regulatory system in particular is that the FDA is kind of very watchful because they don't want people to get information about themselves and misinterpret that information and say like, oh, you know, you're at 95% risk of breast cancer. I'm going to cut off. I'm going to do a mastctomy. Right? They don't want people to have that kind of information and they don't believe the public is ready to directly receive very technical pieces of information. Uh and this is actually not very well known in the public. Everyone thinks kind of like you have your right to the information. No, the FDA is not going to allow you uh right to the information from your own body. They they don't believe in that.
So he started off by saying that they were only going to do body composition.
And body composition is really measurement of body fat, which is a big deal because when you take GLP1s now, you lose both fat and muscle. And so if you have this kind of bat body fat composition monitor, you can you can figure out how much muscle you're actually losing and whether it's actually detrimental to you to actually continue to lose weight. So this is this is a pretty important thing, but it's also something that the FDA doesn't regulate that well. uh because right now in order to do it you need to go into a bone density um you know measurement machine and that's almost a little bit like an X-ray. It shoots rays into your body. This is going to be able to do that without uh going that far. And so I think for people who are conscious of their health and want to know what their body fat is, this is going to be something they're going to use.
>> Yeah. I mean, if the government thinks that they're going to block people from using this technology, I think they're going to have a real fight on their hands. And this is going to >> probably play out in so many ways. I mean, you know, I've talked about this probably Ednauseium at this point, but in the whole cancer experience that I recently went through, fortunately, I didn't have to get off of the standard, you know, my son didn't have to get off of the standard um treatment protocol. Like, it worked for him and, you know, all the exotic stuff that we were scouting out, we never really had to to actually try to get our hands on. Um but I was already gearing up for a battle um on so many fronts.
You know, just even the the DNA testing that we did which is not standard and which fortunately we you know didn't have any real trouble getting our oncologist to support um you know that like fundamentally changed my information landscape and how I was thinking about you know how confident I could be that he was in fact cured. You know, I think we're like over 99% now given all these results. We wouldn't have been able to get to that level of confidence otherwise. And you know, in terms of talking about the hypothetical, it's like, well, what if this next test were to come back a little bit positive? The answer is just like, well, you know, we wouldn't we wouldn't treat on that anyway. You know, we would um we would we would really need to wait for gross disease.
And I just think, boy, you know, people are not going to be content with that for much longer. you know that when we have these technologies and especially this one I think is what what makes it so promising and obviously they they have to deliver right I think a little dose of kind of um uh skepticism is probably warranted you know will this ever actually happen u I don't mean to cast out on that but you know it's not insane to to wonder but assuming that they can actually deliver on their promise the fact that it takes a minute and therefore is probably going to be pretty cheap you know and I don't know what their retail price will end up settling at but presumably it is something that they can operate quite cheaply on the margin and the fact that it's so beautiful to look at you know people will be able to study this for themselves I think in a really effective way and of course there will be you know all the AI um study of it as well that I think the medical establishment is not really taking into account you know the responses have been well you know the ultrasound doesn't see this that well doesn't see that that well or you know we've we don't actually recommend whole body scans because you know there's a lot of false positives and all this kind of stuff and all of this is just feels to me like kind of fighting the last war is sort of a a scarcity mindset on multiple levels. Um, people will, if there's anything like the good future, people should have time and motivation to put into thinking about their own health. Looking at one's own body in this way, I think is just going to honestly be captivating for a lot of people, right? I mean, how much time goes into how we look already and now this is a way to look inside and potentially find stuff that is of interest. Sure. will people like have scares and you know see something that turns out to be nothing like sure that'll happen all the time but I think it really underestimates people and it also really underestimates the technology and how refined it's likely to get especially if they really are able to scale it out and create the sort of scan abundance that they describe in the vision um these sort of false positive issues I think will fade away pretty quickly we'll we'll get good at reading these scans uh both because there's nobody more motivated to really study it than the patient themselves and when it's made beautiful this way and accessible. I I think you know that that the latent potential for somebody to read their own scan and triangulate what they're seeing with other data that they have just how they're feeling. Um you know I think that will be very powerful. uh but the the refinement that the AI layers will put on top of this as well is just you just don't hear uh an understanding of how fast and how much that's likely to progress once you have big data of this sort in the reactions that you see from the sort of um you know let's say uh cautious establishment that have popped up so far. One of the things that struck me about the midjourney announcement is that this is I think what I would call really like an AI native idea in the sense that when he decided to do something he decided to look forward three or four or five years and look at things which were not yet available and would not be available in that time frame if he didn't do them and he focused on creating new data.
Uh and this is something that we've seen I think in periodic labs in a couple of other firms where the defensibility of the idea comes from the fact that they have data that no one else has and they're the ones that are creating it.
This is quite different from I think labs which are kind of operating on other people's data which is what the early foundation models and the language models really did. They were operating on kind of Reddit data and other people's data. And so the defensibility of the idea comes really from you know having your own data at this point. And David Holtz is one of the pioneers of this because when they set up Midjourney the one thing that they did which is quite different and which they stumbled upon really early was this idea of doing fast generation first and then forcing the user to pick which ones to upscale.
And so they basically forced this kind of AB testing on the quality of the images early on. And this I think helped them u really kind of get a much higher quality than they would have been had a very eventful there was a little weirdness on the sound there. Do you >> do you hear me?
>> Yes, I hear you.
>> Yeah. Okay. I'm not sure what happened, but um yeah, I mean these data bootstraps are just the first step toward a you know much brighter future really. I I think back to how this is a stylized history. I'm sure there people could um add important chapters to it, but the image understanding and the and the image text unity that we have now really started with some pretty rough stuff.
You know, the original clip was taking a huge number of photos, pairing them with their captions and just trying to get an ML system to align its understanding of images with the understanding of captions. The problem initially of course was that those captions were generally very bad in terms of you know did they even actually describe what was in the photo. There was probably if you just think of photos on the internet and the way people caption them. Some of them are very literally captioned if they're product photos or what have you. Many of them are all kinds of other things that are captioned in all kinds of other ways with jokes or memories or you know notes to one another whatever right? All this stuff is kind of um so common that the the data was just extremely extremely noisy.
And yet they were able to get enough of a signal out of that that it became one notable path to image generation.
And with each generation of model that followed, a huge part of what they were doing is refining the data and more closely aligning the, you know, using the new ability to caption images to clean up the data to refine the the process to get, you know, image and text more and more closely aligned until finally, you know, we're at this point now where we have this deep fusion.
that process just is happening over and over again in all these different places, right? It's um Jim Fan, we talked about yesterday with his open- source, you know, kit to kind of do um you know, RSI on a small fleet of robots, you know, that many lab many small labs could afford. But he's previously articulated a very similar path for robotics where all kinds of video data can be initially kind of a noisy source but as they refine you know there's probably going to be all these models that will kind of translate random videos that you might have into firstperson POV videos which then can be used to train. So there's going to be all these little tricks played out over and over again in all these different modalities.
And this I think will mean that the ultimate quality of these scans and our ability to interpret them we should expect is going to be far far beyond what you get when you go sit down at a ultrasound you know and have a technician look at it today which is already not bad but you know this will be so much better um that I think it's like you could you could sort of see who has kind of figured out that this pattern is going to repeat across all these different modalities and and who hasn't based on whether or not they're kind of taking that into account in their reactions.
>> Yeah, indeed. Um maybe in terms of looking at reactions in um AI, one of the let me just segue to one of the big pieces of news yesterday, which is Nom Shazir. Nom Shazir is one of the authors of the transformer, the original transformer paper. He's one of the leading lights in AI. He's been he's been one of the leading lights in AI for a couple of decades now. And yesterday he made this bombshell announcement that he's leaving Google. Uh and he said, "I'm excited to share that I'll be joining OpenAI and look forward to working with the exceptional team there." This is after he'd been at Google for about two years, after they purchased Character AI for $2.6 billion just to get their hands on him. And here he is about uh 18 24 months later, he's he's leaving Google again. Um Nathan, what are your reactions to this one?
>> Well, this explains why I couldn't get him on the podcast uh working through the Google uh comms team for one thing.
Um, yeah, it's a big it's big news for sure and I don't know how big a news it is at the same time.
Um, you know, macro some very significant percentage of all the people working at frontier companies right now have some history at Google Deepmind over the years, right? So it's in a way like this all came from Google and and the whole space is sort of a Google diaspora and yet you know they continue to be very serious players with I would say probably still the deepest talent pool and the broadest set of research bets. So having lost all those people and still maintaining that position, you know, there's a I think a good reason to think they will survive this too. You know, they've also got 25% of global compute.
So that's a a huge strength that is not moving.
We've seen something similar with OpenAI, right? They've lost a ton of people over time as well. many executives, you know, the the at the leadership level, the basically the whole leadership team at the time of the Sam Alman firing has turned over since then.
So many many examples of key people leaving and the companies chugging right along.
And yet it does feel like, okay, there might be something a little bit different going on here. uh or or maybe even if even if all that's true and you know obviously the company company's a lot bigger than one person what does it signal what does it mean um it's hard to feel like it doesn't mean anything u certainly you know it's a it's a vote of more confidence in his ability to do what he wants to do and he definitely seems to be one who's expecting pretty radical change on a not crazy long timeline. Um, seems like he he does feel like at a minimum we can say he feels like he's going to be more able to impact that timeline at OpenAI than he would be at Google. Um, clearly can't be an economic move, right? That's one thing I think is is definitely important to keep in mind.
Like he's been very well compensated by Google. So, uh, I can't imagine this is for a few more dollars. It's got to be for reasons of mission and his sense of his likely impact. So let me let me just share a clip um for from a podcast >> for models you care about >> in in flops global to total global uh inference >> it's >> in 2030 >> I I think just more is always going to be better like like I like if you just kind of think about okay like what fraction of world GDP will be you know uh will people decide ide to spend on on AI at that point. And then like okay, what what do the AI systems look like?
Well, maybe it's some sort of personal assistant like thing that is in your glasses and can see everything around you and has access to all your digital information and the world's digital information. And like maybe it's like you're Joe Biden and you have the earpiece in the cabinet that can advise you about anything in real time um and solve problems for you and give you helpful pointers or you could talk to it and you know it it wants to analyze like anything that it sees around you for any potential useful uh impact it it has on you. So I mean I I can imagine okay and then then say it's like your okay your personal assistant or your personal cabinet or something and that every time you spend 2x as much money on compute the thing gets like 510 IQ points smarter or something like that and okay do you would you rather spend like $10 a day and have an assistant or $20 a day and have a smarter assistant uh you know and not Not only is it an assistant in life, but an assistant in getting your job done better because now it makes you from a 10x engineer to a 100x or 10 millionx engineer. Um uh I mean okay so so okay so let's see from first principles right so so people are going to want to want to spend some some fraction of world GDP on this thing. the world GDP is almost certainly going to go way way up to like orders of magnitude higher than it is today due to the fact that we have all of these artificial engineers like working on improving things probably we we'll have solved uh unlimited energy and uh and like carbon issues by that point. So we should be able to have lots of energy.
we should be able to have millions to billions of robots like building us data centers. Um like let's see what's like the the sun is uh what 10^ the 26 watts or something like that. Uh you know I mean I I'm guessing that the that the amount of compute at the you know being used for AI to help each person will be astronomical.
You gotta love it when somebody knows the energy production of the sun offand great sign of true AGI pilled mindset >> and and that's a 20 he's like 2030 it's going to be two orders of m the GDP is going to be two orders of magnitude higher and so his his timeline has always struck me as the the kind of Dyson sphere 2040 timeline like much much faster than the current timeline that we're on and much much faster than Demis' internal timeline as well. Demis' announced timeline is like AGI 2035 roughly and he's he's been persuaded to p pull back to 2030, but he's still kind of like it's better that we do this long term, right? And so it struck it has always struck a little bit more accelerationist than um a lot of other people at Deep Miner, especially Dennis.
So, uh, and and that that clip has, you know, been in my head for, you know, since it came out, uh, that that he was on this much accelerated timeline. And so my sense is that he felt that he couldn't achieve that timeline was not achievable at Google anymore. Uh, perhaps because management had different different ideas about how fast things should go and how much money to commit.
Also the original reason he left Google uh you know pre- character AI is because he wasn't given enough compute. uh he had to share compute with YouTube and one of the things that happened at that point in time is that he would have training runs and then YouTube would have like the Super Bowl and he it would cause like electrical fluctuations that made his models like his his experiments non-re repeatable and once they were non-replicable with the with the same conditions he's like what's going on and Google was like well you're sharing compute with you know these servers that we're also using for YouTube. So, you know, we had the Super Bowl. What can you do? And that's one of the reasons why um I think I think he decided to leave in the first place. And so, it's surprising to see him leave a second time. A third time actually. I think I think this is the third time he's left Google.
>> Yeah, it's certainly not a great sign for Google.
I don't know that there's too much more to say about it other than you know the the short timeline versus midtimeline I think is a a very good frame. Um, I can't imagine he had crazy compute scarcity today. Maybe, but I mean that would be if it was really like we weren't letting no Shazir cook at Google. Um, that would be a pretty bad unforced error given the vast compute resources that they have. Um, you would think that he would be able to, you know, at least have enough to run the experiments he wants to run and prove out the techniques he he believes in.
I I it's I mean I guess anything could happen, but it's it's hard to imagine that they wouldn't uh wouldn't give him enough runway to answer his questions.
>> But yeah, I mean are they I mean one one big difference between Google and the other two companies right now is the level of commitment I guess they seem to have to recursive self-improvement. So, I could definitely see that also being um, you know, kind of a vibe or a strategy level thing that he felt just he wasn't in sync with, especially given that clip that you just said. Even if he has enough uh, you know, juice to do things today that he wants to do, is there a fa is he seeing a phase change coming that he feels like OpenAI might be on the right side of >> and Google might not? I could definitely see that happening to the degree that that's true. I sort of want to take Google's side and say let's not race into that.
>> So I don't um you know that shouldn't be a neglected point from this whole >> conversation like is it wise you know is still I think very much an unanswered question.
>> Yep. No Gnome is on the accelerationist uh pathway. It's it's it's certainly certainly seems that way. So um less less less concerned about the safety aspects and more concerned about rushing rushing ahead into the future. Um so segueing again uh to uh safety in this case u we have the government US versus anthropic government's impossible demand uncircumventment guardrails. So Trump administration officials tell Wire that if Anthropic wants to re-release Fable 5, it will need to ensure the model's guardrails can't be circumvented.
Security experts say that can't be done.
And uh this is where the impass is between between the two sides right now.
>> Once again, it's the smartest of times.
It's the stupidest of times.
>> It's uh it's about equivalent to asking for bug-free code at this point.
>> Yeah.
I mean, it really I feel like we still don't know what's really going on.
It all seems to broadly be adding up to motivated reasoning and selective enforcement and ultimately, you know, bullying and lawfare by the administration uh toward anthropic. That seems overwhelmingly likely at this point, especially given that we still haven't heard any credible explanation as to like what bad behavior was actually observed. You know, that that alone, the absence of that alone is just like so telling to me. U I >> So I always say when when dealing with government, never attribute to malice what can be attributed to incompetence.
So I think I think it's just um people people have other things to do besides like dig into like security details of AI especially when it's not affecting the market and it's not affecting even the company is not being affected that badly. They're like you know let's just deal with it later and you know you have this Iran thing you have other things to worry about. I think it's just not on the not on the like, you know, I have to pay attention basket of any uh decision maker right now to come back and fix this thing. And so they're letting the subordinates kind of handle it. The subordinates don't have any uh rulemaking power. And so it has to circulate in that layer until, you know, Sam Alman needs to release his model and he needs a uh you know, defined process.
And at that point, you know, everything gets clarified and then they get to release, I think.
>> Well, my grandmother once told me that she used to say when she was growing up, you can always trust the government. And when she told me that when I was a kid, she was like, "Can you believe I used to say that?" But that's what we used to say. And I think your theory is like not a terrible one in normal times. But I would also note that like the government has been telegraphing its malice at the same time, right? We've had like very aggressive uh commentary from Hegsth and there's been quite a bit from Trump himself, right? I mean, I I don't have all the quotes uh tip of tongue here, but it's like, you know, you can only call somebody a radical wokeist or whatever so many times before one, I think, can reasonably infer that there might be a bit of malice somewhere in this whole uh you know, the thinking behind all these various moves that they're making.
I mean, it's really scary if if what you're saying is true that like it's not a priority and it's not um you know, yeah, they got a lot to deal with. But I'm not so sure. I mean, again, that's scary if it's true. They did have this lunch yesterday or whatever it was, right? The working meeting with heads of state and the the leaders of the companies.
I also think that, you know, there's got to be some way to square this And I'm not necessarily looking forward to having the rate at which my fable queries get dropped down to opus go from five to 25%.
>> Mhm. But there has to be some gray area or there has to be some sort of number of nines that the administration would accept if they're acting in anything approaching good faith. Especially because once you get out into the kind of long tale of jailbreaks and bad behaviors, a lot of them are very subjective anyway, you know, as to what really counts as a bad behavior. Was this really that bad or, you know, it was kind of roleplaying? Are we going to we're not going to ban fiction, I don't think, anytime soon. So, like, you know, a a fictional story about a bad guy doing bad stuff with like a little light technical detail is maybe something you could say, "Yeah, it's a little bit of a jailbreak perhaps." Um, I don't think it really moves the needle on anything. So, I have to believe that there is some trade-off that they could make that would just have a lot more false positives for the time being, make everything a lot more annoying and just generally kind of dumber, but get you to enough of a reliability level where the exceptions are rare enough and are also just on the face of them benign enough that it would be basically impossible for any good faith actor to say like you haven't done enough. You know, there there's still some one ina- million uh thing here that we, you know, can't let this go forward on. They they don't do that for anything else, right? I mean, this is uh this is totally out of character for this government. So I will, you know, we'll we'll continue to watch obviously, but I'm I think that the demand in some way can be met enough that there should be acceptance of that effort. You know, they've already taken some pains.
They've already, as they've pointed out, endured ridicule for how safety focused they were.
>> Mhm.
>> They could turn that up again. And if they do, and I, you know, probably they're going to have to do something to break this log jam, then it'll be, you know, extremely telling if they don't, uh, accept that turned up safety filter and let us all get back to our new fable enhanced lives.
Um, I'm going to segue again to introducing paper replications for AR archive auto research agents. Ingest popular repos, resolve setup issues and get the core claim running. Sort papers by ease of implementation. So this is a archive um and know this ask alpha archive and they've actually done a uh autoreplication agent. So, Nathan, can you can you describe what what this means?
>> Yeah, I think it's awesome. It's kind of another example very similar to the Jimfan open source project to allow you to set up your own little micro robotics lab and optimize that over time. This is just you want to stand on the shoulders of any research giants in ML. Well, some of them have published their code, some of them haven't. There's, you know, different levels of method being disclosed in different papers. So hence the sort by, you know, easy to hard. At the top of that easy sort is going to be papers with full repositories that you can just clone and immediately start doing riffs on. Um my small contribution to the emergent misalignment paper was very much like this and this was kind of an early example of it. It was that paper came out early 2025. The work was like late 24 early 25. Mhm.
>> And already at that time joining a project getting access to the repo with the Gemini 1 million token window that was you know the new hotness at the time. It was just like oh my god I don't have to read all this research code. It can read all the research code which by the way your research code typically not super well documented not you know not structured in the same um you know very maintainable logically sensible and you know separation of concerns sort of way that more production focused engineering code tends to be. Research code bases are you know often uh considered to be just kind of a mess. Somebody put it together to get to the answer. They didn't really try to make a product out of it.
So you know it's been a real challenge for many people you know many many times to figure out what did they really do here you know can I reproduce this uh how how much will that take the you know the rewards of course in science in general are much more for coming up with something new than >> reproducing something that's already been done but the successful reproduction or failure to reproduce for whatever reason is a core mechanism in science and this is exactly the kind of thing that AI should be able to accelerate in a pretty dramatic way. So, I think we're seeing these sort of AI for science play out at a few different levels. And this one is one that relative to a science agent that goes out and makes a brand new discovery will probably fly under the radar.
>> But I think also should be understood as a huge enabling force that will democratize access that will you know allow us to separate the research that's real from that which is you know varying levels of faked from P hacked to you know altogether outright faked. and the collective sense making and u you know the the frontier of knowledge should move significantly faster I think because we can do this kind of stuff.
Now >> what what has always struck me is how uh machine learning like the processes you used in machine learning are maybe couple of generations ahead of every other scientific field. Like if you were dealing with bio for example, you're still in the like your research is going to go into a journal which other people have to subscribe to and it's going to be a peer-reviewed and it's going to come out a year and a half after you did the initial research maybe two years and it's very slow and there's a lot of like inconsistencies.
What I love about ML is that from the you know from early on these techniques to improve the research process in general have been applied widely. So I think it's exciting. I also think this is kind of a funnel almost where all other kinds of science will have to kind of go through this funnel where they have provide all of the data and allow you to figure out replication and you know look at the statistical measures and you know look at all of the processes that were done and be able to replicate every single step of them right and hopefully that kind of thing reduces this you know ep epidemic of fake science. I I don't I don't even know how much fake science there is in the world. Like every every few years like some reversal comes out especially in stuff like nutrition which is you know has been um just completely hacked by these uh larger food firms. So I I hope to see this spread into every other scientific field and so that we have more replication of all of these things.
>> Yeah, it's it's amazing. I mean it really is one of these kind of pure good you know it's it's I've been so focused on the political dysfunction and the tela like drama among the company leaders and their long personal histories. It's it's good to spend today with just a few really positive stories that seem very unambiguously to move things forward and you know inspire people to get into the game. That's another thing I I think is really um you know if there's kind of a call to action on this it is with tools like this with vibe coding in general you can do ML research you yes you can do ML research you don't really have to have a deep background in math you don't really even have to know you know how um you don't have to know how GPUs work you don't have to worry about kernels there there's just so much work that you can do at a relatively high level because the translation from ideas to implementation um especially with things like this but again just with vibe coding you know to help out as well it'd be a little bit of a stretch to say it's solved but it's it's so much closer to solved it's like 98% of the way solved compared to what it used to be in terms of a barrier to entry so I think this is a great additional signal for people who have ideas or just questions that they want to answer to get in the game and, you know, truly stand on the shoulders of giants and try to get those questions answered. I've seen a little bit of that from people who've never even coded before, but I think we could see a lot more of it coming basically now. There's no no reason to delay any further. And on that note of no reason to delay any further, let me introduce our first guest for today. Jud Rosenblat, the CEO and co-founder of AE Studio. Jud is a serial tech entrepreneur who previously founded the nationwide food delivery platform Crunch Button. But a few years ago, he realized that humanity was racing to build the most powerful technology in history without actually knowing how to keep it under human control. Instead of going the traditional route of venture capital, Jud bootstrapped AE Studio into a powerhouse product agency. They built custom software and complex AI agent workflows for enterprise clients and then funnel the profits directly into their own internal research on AI alignment and neurochnology.
Jud and his team operate on the belief that mainstream AI safety, which relies heavily on surface level behavioral filters, is fundamentally fragile. A Studio is pioneering what they call neglected approaches to alignment. By borrowing concepts from cognitive neuroscience, they are physically altering the internal architecture of models so that they possess genuine traits like empathy and selfcorrection.
Jud, welcome to the show.
>> Thanks for having me.
>> So Jud, can you give us like a short rundown of what your team has been working on recently?
>> Uh yeah, absolutely. We we basically what we do is a whole bunch of simultaneous neglected approaches to AI alignment things that are uh potentially very unlikely to work but extremely impactful if they do. So the the the fundamental thought behind that is that if you look at the history of of science a lot of the time the uh biggest breakthroughs come downstream of people with strong hunches that no one else believes in but they stick with their hunch and eventually they discover relativity or RNA vaccines etc. So we go and find people who have these hunches and then we basically take our existing AI consulting company where we have uh we have a bunch of great AI engineers and use the same structures to then sort of treat these neglected approach visionaries as the client and then build out what their idea is and accelerate whatever whatever that might be. Um and we've started to get a bit more opinionated about the things that we pursue too. Um and uh the the I I think that's that's happened as some of our work has has started to have a bigger impact. in particular. One thing that we'll be uh sharing fairly soon is extremely relevant to the uh it's it's extremely relevant to what's going on in uh politics today, which is that uh currently it is the case that uh basically whenever any new model comes out within hours or within days, Ply the Liberator jailbreaks it. And right now people are freaking out about uh about um about jailbreaks and and cyber security. Uh they're they're much bigger issues like CBRN risk etc. And the fundamental issue is that when you jailbreak a model uh it you you can do all these things and anyone can use a jailbroken model to do fairly dangerous stuff. Um the problem is that uh most of the safety training is done in post- training not in pre-training. So once the jailbroken model is there uh once the model's jailbroken you can do whatever you want um a lot of the time and so we we set out to try to solve that at an earlier stage and one of the things that we've been accelerating is an approach called gradient routing which we which basically uh winds up uh in pre-training you route different dangerous capabilities into different experts in a mixture of experts models. So you wind up having some dangerous experts that learn specifically the CBRN stuff or the cyber stuff and then you can later ablate those experts and this winds up uh so you completely remove it. So you have the regular model and then you have the the the safe model that that uh that winds up being public and this is this has been going decently well. It's like it's a it's a it's still an early stage neglected approach, but we're excited to release it fairly soon because it potentially uh it potentially solves this big issue that a lot of people are very concerned about right now today.
And and our larger thesis is that if the field had been investing more in AI alignment R&D instead of just scaling compute. If we' done this earlier on, we would have found techniques like this and you wouldn't have the issue uh right now uh with with the Trump admin and uh and anthropic because this this would be already in in Fable 5. Of of course it's not in Fable 5. It's it's only in in smaller models right now, but it seems to get better uh with scale and and we're excited for it to potentially enter uh future larger models. So that's that's one example of these neglected approaches. we we have quite a lot more and encourage people to uh to to apply uh to to share share their neglected approaches and and we will consider and and do more of them. Basically, we we just want to do as many neglected approaches as we possibly can and see what works.
>> So, I love the neglected approaches approach. regular longtime listeners will know I'm a huge AE Studio fan and the um I had a fun um experience not too long ago where I was in conversation with somebody from OpenAI and also your teammate Dooo and I was just like oh tell them about this one tell them about this one and the OpenAI person was like how is every one of these a banger so truly you know I definitely encourage folks to go through the AE studio uh archive over the last couple years there are a lot of really interesting ideas in there self other overlap. Um, also, you know, I give my, um, endorsement to on this gradient routing thing. I think one of the things that's so cool and interesting about it is how if, and correct me if I'm getting anything wrong here, but what I understand is with labeled data, you can freeze say all the experts but one or all but two um and get the knowledge kind of predictably and in a way that you you know control and have sort of engineered to flow into those experts. But then the kind of big reveal that I think is super exciting is once you've done that a bit, even unlabeled data from the same domain tends to update those same experts because the model sort of already understands that like that's where this knowledge is uh collecting and so you can kind of get the the approach to work even if you don't have fully labeled data. Right? I think that's a a very notable um you know kind of like physics is sort of maybe kind to us in that way u because it's it's a big challenge to think how would I you know label all the relevant data but if you only have to label a part of it and then the the sort of gravity well of gradient descent just like pulls all the other relevant knowledge into the same place you have a much more robust solution. I thought that was very interesting to learn about.
>> Yeah, this this absorption quality really is very cool. Um, it's it's uh what I I I should mention I'm not supposed to talk a huge amount about this because we're about to publish it and then and that's it'll be coming out supposedly sometime this week or next and then there'll be a bunch of stuff about this. But I guess our uh that the the the cool thing is yeah, we we don't fully understand exactly how this absorption quality works, but it does wind up uh once you have a certain amount of label data pulling in everything else that that you're saying.
And um we we do suspect that uh that we we need to basically like scale this up as soon as possible. Like that's that's that's kind of the the key thing because it does uh it does promise to solve like the big thing that I mean the Trump admin doesn't necessarily totally understand exactly what the issue is from their reaction so far but they're trying the cool thing is they are trying to understand it for the first time and taking it very seriously and I I think we can see that they're going to be looking for real actual solutions to this thing. Um because the I I think to a decent degree the feeling is is kind of like uh well um okay what why are you freaking out about this stuff and building all this stuff if you think it's so bad and then and then like why are you not listening to us about uh about slowing it down if you want to slow it down. Uh it just I I think they're they're they're hearing a lot of very confusing information. So, we're pretty excited to try to uh make neglected approaches rapidly transition into real actual solutions and paths forward to some of the the most pressing issues as they actually come to a head.
So one of the things that strikes me is that when you have um I would say uh curation of the uh pre-training data, you often also have impact on the capabilities front on the other end. So how do you balance this kind of capabilities versus curation aspect?
>> Uh so far we are not seeing a a major impact from that. uh and the capabilities wind up getting routed into the uh capabilities expert. So you can have the uh the the essentially like simplified version the chemical weapons expert versus the the other expert that understands more about chemistry. It does seem like there are going to be some uh decreases in capabilities though uh potentially but but um it not not sufficiently major ones that uh prevent us from launching very powerful models.
>> Yeah. And in some ways the the capability reduction sort of is the point here, right? like the the mythos to fable pipeline today is like let's lay around all these filters and maybe we'll even do some like internal um you know activation steering to make your responses not so good if they're in what we think is Frontier ML or whatever.
It's kind of all these different patches. The and probably those will survive. My guess is we'll still see defense and depth going forward regardless. But wouldn't it be amazing if you could just take mythos and then just literally pluck out the chemical weapons knowledge, pluck out the viology knowledge and then you'd be like a lot more confident even before you apply all these additional filters and you know monitors what have you that that knowledge just isn't there in the version of the model that the public is using. That would be a I think a step change you know in terms of confidence from what we have today. Can you Jud maybe tell us a little bit more about like how you understand what the Trump administration is doing?
I think you know my reaction has been fairly cynical in as much as I have felt like they seem to be seem to kind of have it out for this company. Um, I think like being wary of the industry as a whole is something I much more understand than like trying to pick, you know, one good guy or bad guy out of the current frontier companies. But I think you've articulated one of the best versions I've heard of a more sympathetic uh reading of their actions. So like what would you say is kind of the steelman for what we've seen from the Trump administration over the last 10 days or so?
>> Uh, yeah. Well, so I I think it is hard for many people who work in quote unquote AI safety to have empathy for Trump and the Trump admin. And that really is unfortunate because it's if if you don't if you don't have and a lot of our our neglected approaches actually have to do with empathy uh which which we think might wind up being fairly pivotal for solving alignment. So it's interesting to consider that if if you don't have empathy for another agent, it's hard to have an effective model of how their mind works. Um and and so uh basically my my main message to people working in AI safety is you you should try to model how their minds might be working. Uh which which which incidentally means that uh you have to like everything you've experienced prior with with all of your priors about every single like previous confrontation or thing going on in AI uh their experience of it was very very different and uh it technically understand and get told, well, okay, we we need to this is this is this is dangerous. We need to stop it. And then go ahead and and stop it and and try to ask for the thing to be stopped. and then don't get a good reception from this this guy who they've they've thought is like a uh confrontational and not uh not easy to work with in the past and and then uh they figure okay well we need to do something about it. Um, I think it's a if you understand it from that perspective, it's a it's a very reasonable response actually. And in fact, I think that most people in the AI alignment world should should be very hardened by the fact that the that that Trump saw, oh wait, something's going on and we need to do something about it.
And he took action. And this is something I've been predicting for years would happen incidentally, too, that uh that that that uh Trump is someone who can take decisive action when needed.
And that's actually something very much needed uh as AI starts to accelerate uh and and uh get to the point where things start to get crazy. Like you you want you I I think I think it's it's it's important to sort of like imagine imagine someone getting introduced to this stuff for the first time. What is their reaction like for for everyone?
It's it's a bunch of it's it's pretty crazy stuff to start to really engage with and and think about, I think. And uh also when you're dealing with with with high stakes things like uh you know international relations and war and whatnot there it's it's it's it's just another big crazy thing to deal with. Um but if you think about like friends getting introduced to all the stuff going on with AI to start with how how many conversations does it take usually for them to start to update to think about how much the world is going to change accordingly. It does take some time. Um, and and people's first reactions and first uh stabs at doing things to do with AI, people's very first neglected approaches, ideas are a lot worse than the ones they think of after they've been thinking about this for a little bit longer. So, I I I think that it is incumbent upon us to do the best we can to set the Trump admin up for success as they start to play a larger role in in in the future here.
And uh and and obviously they will. Um I I I think they can make very very good decisions and learn more about how to do that over time and if they're treated with with empathy by people who incidentally uh we did we did surveys of of hundreds of alignment researchers and effective altruists and we saw that uh less than um less than 2% of alignment researchers were politically right of center. Less than 1% of effective altruists were politically right of center. um of effective alter 40% were uh extremely progressive and another 40% were very progressive and it's it's it's also worth considering things like the the Jonathan hate research around how uh how hard it is for people to actually empathize with people of different political backgrounds and and I think that's uh a lot of what is actually going on here uh and it's hard for people to admit because you think you're making good decisions and judgments about whatever the current thing is but according to that research, you're just not uh if if your political beliefs differ from from the person you're judging. Uh like like the the studies around how basically uh things like that you theformational content of a political argument is irrelevant to what to whether someone will believe in it.
If if it is framed in terms of your preferred political party, uh you'll you'll agree with it. And if it's framed in terms of the other political party, you'll disagree. But the if the but theformational content stays the same.
uh theformational content is not what sways you. It's it's just the narrative of it. And uh it's it's hard to remember that in in every moment in every time slice of of of what's going on with with each AI thing. But I was fairly disappointed in the the AI alignment world's reaction to what happened last week because I think that the right thing to do is to be very excited that they are starting to take this stuff seriously and are able to take real action. Um and so if if we just project going forward and and also keep in mind by the way that we all have exponential slope blindness if people didn't didn't we didn't evolve to uh to to be able to unconsciously model what exponential slopes are like because we don't experience them over the course of our single human lifetimes in a meaningful way. So uh so so so that's why people didn't predict what's going on right now that it would get to this point in the first place but also everyone's overindexed on what's going on right now. So, uh, people aren't really predicting what's going to happen again in the future. Uh, if if we predict into the future, well, there are going to be much bigger, crazier things going on.
Uh, we're and and and we want uh uh in in a informed, competent uh group of people doing smarter things when that happens and and and not having unnecessary confrontations. And I think it's easy to uh for for the AI alignment people to put the blame on the Trump admin, but honestly I I really think that that the blame comes from belongs more to them honestly because uh it it's it's just uh and and it's it's hard to admit really because like in the in the local incident you might seem rationally correct but in the in the in the broader scheme of things uh and considering where we're going I think that the the better thing to do is figure out how do we get to a better future for for all AI and uh humanity in the future of consciousness.
>> So how I might be guilty of what you're saying. uh Jeffrey Ladish, uh Lauron Shapiro, who we talked to earlier this week come to mind as voices from the AI safety world that I think expressed the sentiment that you uh advocate, you know, which is like, hey, this is a good move even if it's a little bit uh you know, not as technically grounded as we might wish for at this point. It's something and and maybe it's something that we can build on.
How will you know if you're right or wrong? Like what do you think what do you think happens from here? when do we get, you know, resolution? What does that resolution look like? And then as you go forward from there, maybe give us a little bit of your expectation on what sort of crazy things you expect to see and then what you think government should be doing slash, you know, anybody with a lot of money, right, which certainly includes uh a lot of people uh at companies that are about to IPO. I think you've been fairly critical of like the OpenAI Foundation, for example, for doing stuff that's kind of too basic, you know, or or too um, you know, not sufficiently creative, not sufficiently imaginative. So, what would a Manhattan project for alignment or whatever, however you would like to um frame it, you know, what does that look like to actually meet the moment of the crazy stuff that you expect to see?
Uh well, I mean it's it's it's very hard to predict the future, but I would say that the highest impact thing that could be done to to solve stuff is just do more simultaneous neglected approaches, more ambitious R&D projects that could potentially work. And it's trivially cheap to invest in that compared to how much we're investing in in compute or uh how much you know like the there there's whatever that stat is about how the military spends more like 80 times more on drum on on drummers than Casey or something uh some some some musician musical something uh that so so uh we could everyone could be investing in alignment R&D that's that that that is the the key thing that has to happen and there are a lot of competent people who uh are are working on all sorts of random stuff but they could just be working on alignment R&D like any random person with an interesting idea because it is so easy to just build stuff today can start to work on alignment R&D we could we can uh motivate motivate and inspire the brightest people in the world to just go to work themselves on trying to to solve these problems in weird ambitious ways uh the government can can create major action to do this obviously uh they they have a lot of ability to spend money to do ambitious things there the frontier labs could be invested ing more heavily in alignment R&D. Uh and in this this in particular though the the the key thing about it is that you have to try things that are unlikely to work but high impact if they do. Um and and so the unfortunate thing is that Frontier Labs for instance tend to think too short term uh generally and and want results too fast and then are less likely to invest in the very ambitious stuff that might take longer to work. But the the and and incidentally, this is why like many companies prioritize short-term revenue and things that might look good to investors over the longer term things that actually create substantially more value for for users across the board and then create their make their company worth more money. But also uh it's in this particular case it's something that they're just not prioritizing uh they're not prioritizing that very ambitious R&D to the extent that they can. I'm glad to see that um Enthropic is doing more and more real useful stuff, but they could be doing orders of magnitude more. And uh the the the weird thing is that historically it used to take longer for your ambitious like doing the right thing in the long run to pay off, but because everything's accelerating, the the long run is coming sooner. So, uh the the ambitious R&D can pay off a whole lot sooner. It's still going to look very dubious at first. Um but but choosing to invest in it and uh makes you actually solve these very ambitious problems. Um it also allows you to solve like what are what are the near-term problems that are most urgent that we need to solve like uh for instance things like CBRN risk super urgent to solve and and we set out to do something that that could solve that that is fairly promising. um or or other immediate issues like the fact that that nobody knows how to fully detect detect and uh and and remove or or deactivate sleeper agents today. Uh like there there's all sorts of stuff that is going to potentially um be extremely relevant in the very near term and we meanwhile AI is going to get deployed in various ways which we know are going to lead to very insecure situations and we could be setting out to just ambitiously solve that stuff. We also need to figure out how to very rapidly uh deploy the innovations that are made um especially incidentally as as as other places are starting to do that um like China is moving to have much more of a cyborg state where they're adopting AI all throughout their their government at every level. And um we we we are effectively sort of in in in a war with China in the first place, though many people don't realize it right now and are constantly under various attacks from China, which are poised to get way way worse. And we need to be poised to defend that very well and then um find new innovations based on on the R&D that's going on to put oursel in a better position to defend against that type of thing. Um, and uh, and then also like it's it's interesting that so many people in the alignment world tend to be like, "Oh, we just need a pause." But this this tends to uh, now maybe that makes sense. I don't know. In any case though, the the the big question is after you get a pause, well, what then? Um, and uh, and and the the answer is well, you have to solve the alignment problem. You have to solve a large set of of unsolved uh, ambitious R&D uh, challenges that have not yet been solved. and uh and and meanwhile also you're going to have the full forces of capitalism behind trying to make algorithmic improvements that bring down the effective cost of compute at inference and so so uh potentially you have very significant risk there um unless you plan in advance to to heavily invest in in alignment R&D.
So, um, Jud, um, obviously the policy makers right now, they're in this kind of fog of war, right? They don't they don't actually have technical knowledge, so to speak.
It is a it is a new field anyway. They don't know who's telling the truth.
There's a lot of like, oh, you know, the AI safety folks are hype marketing. All of this stuff is like hype and it's just they want regulation and they want regulatory capture. There's all of these um you know conflicting narratives especially like from some of the VCs from some of the other players and they have difficulty in deciding who is correct. They've obviously chosen to uh depend on ideological kind of zones of like thought um to decide who's in the right and who's in the wrong.
um we've seen this kind of like very complex uh field get regulated over time in finance but it took over like 100 120 years to kind of you know you have the SEC and you have all of these different bodies for all of these different verticals and all of these different regulators we just don't have that much time right now right as you say there's this exponential curve blindness what can policy makers realistically do for regulation besides just telling these companies to stop, right? Like they have maybe like maybe 18 months so to speak because some people are looking at recursive self uh self-improvement in you know 24 months or so. So what realistically can they do in this time period?
>> I I I do have a lot of thoughts here but I do I I honestly think that the highest impact thing that they can do is heavily catalyze alignment R&D. So if you if if we are getting recursive self-improvement then we need to look for well what is going to survive recursive self-improvement. what what what what is going to remain invariant when AI can improve itself? What will be selected for uh when AI can make itself better and humans are not in the loop, why will it choose to keep these alignment properties? Uh because if if it if it doesn't increase its capability, it will be selected against um and and so uh they could be they could be articulating that and uh and and catalyzing huge research programs that would actually uh make this be top of mind. I mean the the thing that you just articulated the fact that recursive self-improvement is coming soon. It's interesting how few people in the AI alignment world are really grappling with that and thinking about the fact that okay well if that is coming then what techniques will actually survive recursive self-improvement and when it we're we're trying to hire as many uh ML engineers and uh AI alignment researchers as we can these days. And a question I always ask them in uh in in the interview is tell me uh explain what is AI alignment and tell me tell me AE Studios unique approach to it. And then I ask them uh I asked them to tell me what is a uh alignment R&D approach that would actually survive recursive self-improvement. And almost nobody can tell me one. I I say like even something that's very unlikely to work but high impact if it does uh but but would actually uh prove to be something that could contribute to something that would that would solve recursive self survive surv recursive self-improvement and people don't think this way to start with. You you have to you have to change the way you think about things a little bit to think about what what would actually survive recursive self-improvement um potentially. And the the the thing is like um I I think that that the Trump admin is going to be looking for solutions. they are they they're looking for solutions and wanting to say like okay well let's how do how do we actually proactively solve something and the a big issue with with um the uh AI safety field and and whatnot is that I think it's just sort of sounding the alarm without pushing forward any potential real solution. So uh you you it would be good to have if not a solution paths to possible solutions. So uh what what what the what what what I think actually can be done is the beginning of the articulation of these paths and what that realistically means is sufficient investment in the R&D necessary to do so with also some amount of of uh of um oversight that doesn't get in the way of that fundamental innovation as well. And um now that we see that the the oversight that they're willing to to stop things from from being publicly released, well, okay, uh we we we can we can have very good eval regimes that uh actually stay apprised of the latest technical developments and figure out good means of communication to relevant uh relevant current leaders in politics to uh in the admin and whatnot to to actually do this stuff. And it it seems like it seems like we are likely to move to that. Dean Ball and others point out that effectively we do have a a a regime that does this right now. It's just very opaque. And that's uh that's that's unfortunate, but um we it's if if you just project going forward, well, something something that is less opaque needs to be established. And I think we're tracking to eventually have that get set up uh hopefully in the not terribly distant future.
>> For those that are playing the AI and the AM drinking game, that's one mention of Dean Ball. So everybody can take their shot and um keep your ears peeled.
I think there might be more mentions in the not too distant future. Jud, I really appreciate you coming on and joining us this morning. And you know, bigger picture, really appreciate the neglected approaches approach. I always encourage people who have potentially wacky alignment ideas to pursue them.
Uh, you know, obviously, as you said, most of them probably won't work, but we need many more minds and we need many more different kinds of minds, I think, working on this problem. And you guys at AE Studio have been really visionary and you know laudable for putting your resources where your mouth is in terms of inviting people who haven't done this kind of thing before and whose ideas might be a little bit strange to get into the game nonetheless. And some of the results that you've produced so far show that I think that there is really a rich uh vein there to be mined. So, keep up the good work and we will hope to have you back here to help us make sense of things again before too long.
>> All right. Thanks so much. Thanks for having me.
>> Byebye. Bye for now.
>> And and we have our uh next guest coming up.
Uh we have Eno Reyes the co-founder and chief technology officer of factory an artificial intelligence research lab and platform company building systems that allow engineering teams to create and run their own autonomous software development pipelines. Instead of merely providing an assistant to help a human write individual lines of code faster, factory builds what they call software factory. This involves collection of collections of agents which factory calls droids that can autonomously handle large chunks of work from writing and testing code to reviewing security protocols and generating video enhanced documentation. With a background spanning cognitive science, deep learning for neural data and open source tooling at hugging face, ENO brings a highly pragmatic systems level worldview to artificial intelligence. The central argument is that the future of software engineering is not writing code but building and refining the deterministic systems that build the code. Eno, welcome to the show and tell us what you've been working on recently.
>> Yeah, hey, thank you. That's a that was a great intro. Um, I I think that the uh you know, one of the things that we've been thinking a lot about I think you sort of referenced which is just this idea of what does the next step after coding agents really look like? uh and I think that uh generally at factory we've sort of held this opinion that there would be something that you call a software factory and especially recently this vision has I think started to really become grounded in a more like practical reality of uh of sort of seeing where organizations are really looking to accelerate beyond just I'd say like 20% 30% 50% improve improvements the speed with which they they build and ship high quality software and as uh you know the the technology has progressed it's become clear that we're building towards this future where AI systems will from the signals from the outside world that decide what our software should be that could be anything from you know business require I mean you think about all the signals that we as humans listen to right we're reading Twitter and looking at competitors we're on industry news forums We're talking to peers. We're internally in Slack conversations, Teams conversations. We're listening to our telemetry, our data, our product analytics. And today, this like very human process of effectively absorbing and letting your brain choose to draw inferences and build prioritization along those signals. And then this is I think roughly where the formal process starts. It's like you might bring that into a PRD system or a or some sort of structured guidance by which humans say let me write down the plans and the priorities. Then you pass those to software developers. They build, they edit, they refine, they validate, test, QA, ultimately deploy monitored software which then does what? Well, it generates more signals. So there's this giant feedback loop that's extremely human-driven right now. You can imagine and in fact some people are starting to instrument the whole thing end to end with AI. And I think that the challenge that almost everyone faces is this is one a totally different problem from adopting agents and two it requires effectively a reframing of the way that your company thinks about building software. How do we set goals? What are we optimizing for? What should our software evolve into? And we think that this is just like a really big challenge and people are underestimating how massive of a challenge that's going to be. And so we're trying to start to build that initial layer that helps enable this knowing that we're not going to, you know, snap our fingers and the world's fortune 500s are going to suddenly flip to this new model building software. And then also of course thinking about the humans and like what is our role in this new system because I think it's kind of terribly uncreative to imagine that like this system appears and then actually humans just are not useful anymore. I think that that clearly is not uh the direction that things are going. So I I think I might not be able to hear you. Is that just me?
Nathan, are you muted?
So, he's got to he's got to I think reset his audio. So, one of the things that I wanted to ask you is I've seen factory on the timeline quite quite a bit. And what I've also seen is that the way that factory structures the autonomous agents produces a lot of code a lot a lot of like code reviews a lot of basically tokens.
Um how do you deal with this bottleneck of the human having to at the end of the day be responsible for all of this work?
>> Yeah. Yeah. And and this is I think a fundamental question for almost every industry which is as we start to delegate more of our responsibilities to AI systems or at least responsibilities that typically were done and executed by humans there there comes this threshold where you need to know that the work is somehow being produced at a level of reliability and quality that you can confidently back. Right? I I think that I think about this for self-driving cars, right? Like self-driving cars actually had this crazy high bar set where like fatalities on the road was this like obvious metric where even a couple years ago, self-driving cars were were trending towards not just being a little bit better, but but meaningfully better than your average human. But that was just necessary but insufficient to say like great, let's just flip every car on the road to be self-driving. I mean there's a lot of reasons for that but but ultimately there's like these metrics that I think we've developed for industries like that where we say look there's some bar where we're trying to target reliability quality and ultimately like liability minimization and I think that for software development we actually do not have a super rigorous history of being able to reliably measure what our like fatalities on the road is like you can try and measure bugs you can try and measure incidents. You can try and measure um times that you ship the product that clearly didn't work out, but all of the feedback loops and the cycles are just too short uh too long to be able to meaningfully draw a lot of inferences when it matters, which is basically like at the time of generation or like when you're reviewing something.
Um and so I think that a lot of our effort is actually right now going into trying to be really analytical and saying can we trace and and understand hey this bug was is a bug first of all that's actually hard was this a bug was it a paper cut is it a bug that turned out to be actually fine like measuring that tracing that to the source did a human write this did an AI system write it trying to understand could we have caught that issue with some deterministic checks or some nondeterministic checks and then starting to then build out this metric and saying we know that humans are here.
What is it going to take for us to get AI systems to get to like here, right?
Where they're like at the midpoint or or or less of incidents that are caused by AI and I think then humans would be very willing to sort of cosign these systems especially if they've played the role which we think they will of refining the guard rails and the guidelines by which the system operates. um at at some point it's not that different from like a a manager or like a VP sort of persona saying look I built this system I take responsibility of the system that I built is unable to achieve the outcomes that I care about and I think that's how a lot of executives are judged it's like did you drive that outcome regardless of the mechanics by which you got there so I think we're going to get to that point with a lot of software systems but as you know evident from anyone who's used these tools there's a lot of ways in which this can go wrong in subtle ways over long time periods like sloth accumulation, tech debt. Uh and I and so I think that that that this challenge is is is ultimately very difficult and not solvable by the way by just a smarter model. I think that's like probably not the right way to think about how this problem gets solved.
So before I ask a question, hopefully you can hear me now. Um, >> yeah, >> there's some background noise coming, I think, from your side that sounds kind of like something rubbing against something else. And it might be fixable with a change of mic. Um, or maybe something else. I'm not sure, but figured we'd at least try to address.
Um, >> how about let's hear you now.
>> Is this better?
>> Now I don't hear you at all. Um, you can refresh real quick if you want to rejoin and um, just make sure you have the right thing selected.
>> How about now?
>> You know what? May maybe it's me now who can't there. We do have um, still some little issues here.
>> Okay, go. Yes.
>> Yeah. How about now?
>> Oh, so much better. A million times better.
I think Can you still hear me? Yeah, I can hear you.
>> Okay, cool. Um, great. Uh, here we are.
Oh, now I can't hear. Goodness gracious.
Oh, >> that that that's just me. That's just me being uh being muted.
>> Okay. So, >> the a problem.
>> Yeah. Okay.
I guess you the last thing you said was this can't be solved by a smarter model.
I think Fable might like to have a word.
How would you interpret the Frontier code results? Because my sense of those was that for roughly a doubling of the price, you could get more than 2x the success rate. It went from like 10% with latest Opus to 20 upwards of 25% with Fable.
And it sure seemed like the kind of the motivating observation that they made was like well all these sort of sweet bench even the hard ones a lot of times you can pass the tests but the maintainer wouldn't actually merge the code because it's not that clean. It's not maintainable. It's not organized in the right way. It doesn't follow the standards. And so it was it was calling out I think a number of the things that you were emphasizing right around tech debt and kind of slop um where yeah tests pass but it we don't really like it. Um seem like Fable makes a big move in that direction.
Do you read that in the same way or do you would you tell a different story about how we should be interpreting Fable's results on Frontier Code?
>> Yeah, totally. I mean, and I think about I I think that we should actually sort of sit here and and frame what is actually happening when when we say like Fable outperforms on Frontier Code. So, Frontier Code, good great benchmark.
Like that's a I I'm really glad that people like the cognition team are like thinking through how to we how do we measure on more novel and difficult problems like the types of challenges that contemporary models are facing. And so, I think we need more of those.
There's another great benchmark called Program Bench uh that that also looks at like reverse engineering on extremely hard problems. The pass rate there is like effectively zero. Um we have internal benchmarks that we have 0% pass rates on. Uh and and I think that like generally this is great when when we introduce these new benchmarks, but if you think about what it means to score on a benchmark like I mean you can sort of read through right oh well we assessed correctness by running tests.
We used LLMs to judge correctness. We built novel verifiers specific to the problem. Basically, what that means is that when somebody spends 40 plus hours creating a verification of a single code change, we can then reliably evaluate if the model was good at working on that problem. That is like totally reasonable. But I think what it translates to is that in the real world people the challenge is often not can the model write code that works. It's basically every other aspect like can I trust that this model output code that works? Um does this model have the the deterministic feedback loops inside of the codebase to get to that correctness level. Like often times this I like I I believe that that repository the set of repositories in that benchmark are all very well tested very well-known open- source code bases right where the maintainers approved it. the level of rigor of what we would call agent readiness in open source code bases actually tends to be much higher than in enterprises. And so which makes sense.
You're basically accepting changes from the outside world from random people.
How different is that from coding agents where you're sort of like getting changes that you sort of lightly asked for and you don't even know the source.
It's kind of blackbox generation, right?
And so I think a lot of open source maintainers have gone through the the rigor and the effort to to add these deterministic verification and validation loops into their system such that when a new change comes in, you think about how did Fable get such a high score? Well, it ran the tests, it ran the llinters, it did more focused application of the type checking. It used all of these tools to hill climb its way to high success. And I think that in general the if you don't have those things, you're screwed no matter what. And what we would sort of argue is that all of these pieces are part of the puzzle. You can't just plop good model.
You can't just have agent readiness with a bad model. Like you you sort of need to go through and invest in upgrading the basis by which your company has these feedback loops. You have to upgrade the way you think about this because it's a risk thing. like humans have to say, I'm going to at this point now start accepting code changes that I haven't read. And then third, you do need great models. So, so I think that like basically opus 4.6 maybe was has been sufficient enough. I would even argue that before then we've had models that were sufficient enough to go full auto. I think that all of these other things need to catch up in order to then take advantage of these models. And basically the gains we see in models today are primarily coming from effectively like the models getting better at at getting away with not using these verification loops like humans are.
>> So one of the things that struck me is that if you've gone into a traditional software organization which and then try to switch them into a test-driven development environment, you often encounter this enormous resistance because they're just not ready for it.
And um it just takes a lot of refactoring on legacy code and that is just often for the CTO not a priority enough to commit the resources in order to do that. Do you find that you know using like factory AI your customers are actually able to engage in this kind of refactoring more wholesale? Is that is that a is that as something that becomes possible now that you have um this group of AI you know agents that can run along but then at the same time you're also pointing out that you need this kind of structure for your AI agents to work. So h how does this how does this interplay between this like legacy codebase and you know uh test-driven development and your AI agents work?
>> Yeah, totally. And I think that the general thing that we see for our customers is that the stakes have risen of what happens when you do not enter into these mindsets of I need to refactor. I need to So I would argue that 3 to 5 years ago the introduction of I don't know some sort of like endto-end test that actually checked the performance of your application and blocked PRs if you didn't hit some bar of performance. The stakes are much higher. Humans have to actually think through this problem and we are generally not the best at like when you take a hundred of humans we're not the best at having consistency in like one specific task like performance optimization right and so you know 3 to 5 years ago if a CTO were to walk into any serious enterprise software business and say actually we're going to require all of you to hit 10 of these arbitrary bars that we're setting in order to make sure the quality sticks everyone's going to revolt they're going to be like hey look we cannot consistently keep up with that level of rigor on tasks that we don't even know if it's obvious that it helps because I'm making a change to like the front end button. Me making button round versus square is just simply not going to break our performance. And that's probably true in like 99% of cases. And so back then the just the bar is just lower because it humans are pretty clever and we're good at applying things uh in in our own way and we get around these types of rules.
Today agents are not humans. They act in a very different way. Uh it's in fact a whole I would argue uh challenge of everyone as we evolve in the AI world to build a uh theory of mind for how these systems operate and start to be able to sort of like understand intuitively when an LLM will zigg versus zag uh when you ask it to do things. Um, and so I think that we don't have that same sort of permissibility of letting things slip and that makes the need for these like guard rails, these verification loops, uh, much higher. And so ultimately like what we recommend to people, we sort of have built a formula around this, right?
The first thing you should do is understand where you're at in your journey on agent readiness, right? Take take stock of the deterministic feedback loops. Then you can start to bring in automations, code review, security review, QA. These things are pretty easy. Like I I think actually a lot of companies already have some degree of either homegrown or they're using an out-of-the-box solution for for some of these workflows. You can then start to hill climb on that agent readiness. And I think that you don't need a super high degree of this in order to get productivity out of agents. In fact, I think at basically every stage agents can be useful. But what we've seen is at a larger scale, if you've got 45,000 people, uh you are going to notice that the people who are on level like very high agent ready code bases, they are just ripping. They're able to say please turn my natural language into software that works and it's consistently delivering the outcomes they expect.
versus the people who are operating on very low agent ready code bases. They are struggling and they're wasting tokens. They're spending huge amounts of money asymmetric to the rest of the org and they're not even seeing the outcomes they care about and so we feel it's a big explanatory variable on things like cost as well.
Can I ask all of that makes a lot of sense. I sort of struggle to envision where this is all going and I want to get your take on it. So obviously we've just had Curser get uh their acquisition option taken at $60 billion. Congratulations are in order for you as well with a recent fund raise that puts factory in the unicorn club. Um I'm always kind of unclear as to are people converging or are they diverging in their visions and their stories and the products they're building in the sort of role that they imagine humans playing. Um, you know, how I guess how much do you think you and the sort of blitzes and cognitions and cursors of the world are all headed to ultimately the same destination versus sufficiently different destinations that, you know, there's a a place for everybody in the future.
Yeah, this is I think this is really interesting and I can't always speak to the true northstar of other players.
Like I think that there's a lot of underlying infrastructure that's clearly the same across anyone who wants to seriously operate in the space. You need the ability to have an incredible agent harness. What's interesting is like there's I think an open question that uh that like you know do you need it to be your agent harness? We would argue I I think you do. Uh, I think not having access to model independence is extremely risky and the harnesses themselves offer quite a lot of blackboxing on certain activities that I think will make it just hard to do independent business because I do believe that we are an aggregator of models like as like a fundamental commodity this this intelligence layer and if you are not able to optimize your harness to be able to take advantage of those aggregated models then you're I think over the long run going to be in this sort of like weird place where you're sort of routing intelligence that you don't really control. you just sort of control the basically what very high level you route to versus I think at the model layer you actually do control the intelligence regardless of if like that model comes from this data center or that data center as long as you have access to the models right and so there's some of these base layers of infrastructure where even that kind of open question but ultimately everyone has a harness everyone's building sort of dev sandboxes as infra um the basics of automations and all these other things where I think that factory is sort of I'll I'll give you our uni our our take and you can tell me if this is like unique or not. Um personally my view is that in a couple of years uh we will be setting the trajectory of these feedback loops with very high level goals and we will be setting constraints and budgets and we will very much look like VCs or capital allocators, right?
And I think the the the different strategies that capital allocators take up today can give you a picture of what software orgs will look like. You'll have people who are like VCing it where they're betting on several products in a b in a basket and they're saying let me allocate compute and build guard rails around the shape thesis around what my software should evolve into and I'm going to allocate like a little bit to each of them and I'm going to double down on my winners. Right? So I think I see that as being like a very plausible software organization strategy. I also think you'll see people who are like, you know, birkures where they they're only looking at like well-known repeatable kind of boring software businesses and they use scale and they use the fact that they're able to control large amounts and volumes of this software in order to accumulate kind of steady gains as they sort of scale up. Uh I think you'll have boutiques that make one piece of software really really well and they're just incredibly good at making this one piece of soft and maybe that's like the oneperson billion dollar company, right?
Like maybe you can actually just be one person maintaining a micro software factory that where your goals are like some some combination of revenue. But in reality today, right, not every company has their Google metric, right? I think Google has this like famous 10 millisecond reduction on page load led to like a hund00 million. I don't know.
I don't know the real numbers, but it's like this famous metric where they had this simple input output engineering metric and money out number, right?
Right? And they were able to take advantage of that for a long time. Very few businesses know their like Google number, right? And if you can start to identify where are the places where our engineering or software development leverage can be like pressed in on, money can go in and outcomes go out the other way. Then you're in a great place for an massively autooptimizing fable 8 like software factory system because you'll be able to say, "Look, here's my goals. here's the button and the lever I want to press. We want to be the infrastructure that helps all of these unique profiles of business assemble their software factory. And we cannot know in advance every piece of what you're going to build, but we can we sort of know the primitives. And I think that then making that really easy for an organization to transform into this shape. That is a really hard problem.
And I think that that's probably where the majority of the value is going to be. letting new companies build with this model will actually be I think fairly easy. Um so that's sort of like where we see this going uh in the long run and and maybe others do too.
>> I'm just going to ask uh maybe perhaps one last question and as the CTO um I think one of your roles is that you do purchasing of products from other companies from other firms.
>> Yeah. And so you probably have the problem of identifying agent ready infra that you need for your products. Um we've had uh cursor start off with cursor origin which is I think a GitHub replacement. Um what are the kind of products that you want to see in the market that you would want to buy that you see agents as going to need in the future?
Yeah, you know, a lot of this I think is quite interesting. The things that we buy today are not necessarily things that I actually want more of, which I would say are systems, closed systems of record.
Anything that has a closed system of record, we do not want to do the necessary work to rebuild the workflow that Salesforce has trained into all of our account executives. And so, we're simply going to buy Salesforce. Slack, great system of record and database for storing all these messages. It simply doesn't make sense. And they've got a network effect as well. So that's that's great for them via Slack connects. So So it's things like that where I'm like, look, I don't know why anyone would try to replace these things. Uh maybe if your Salesforce bill is like massive, right? I think that what I'd love to see more of though are these sorts of like like what 11 Labs produces, right? like deep tech hard problems that connect into agents and software that solve like fundamentally hard research problems that enable a new class of like problem solving, right? And I think that the more that I see tools like that, people who go and specialize into unlocking new modalities or who build out uh like even like hard tech that that is like physical things that enable new ways for humans to interact and imagine interactions with computers. This is the stuff where we would be like willing to to spend huge amounts of money on. Like if somebody came out with some new device that was like obviously a better way to interact with Droid, our agent for computing, we would bet the horse on that on that device, right? Like we would spend huge amounts of money on it.
And and I think that that's the sort of thing that maybe is where the world is going is like there's just way less lowhanging fruit SAS and now we're moving into a world where some sort of more fundamental deep tech or research is required to unlock a really wide variety of value from different players in the AI space.
>> Cool. Uh you know this has been a a great conversation and we will use 11 labs to voice isolate the first section uh where we had a little audio difficulty. So, in the highlights version of this, we'll get the uh the polished sound. Uh but you're I think a very compelling and clear communicator for the vision and we'll have to uh survey some of the other players in the space to see whether they're uh converging on similar ideas or if they're going off in sufficiently different directions. But congratulations on your success so far and we'll hope to see you again here at some point in the future when you've got a big update, something to share. Uh, let us know and it'd be great to hear from you again.
>> Yeah, thanks guys. This is great. Talk to you later.
>> Byebye.
>> Thank you.
>> Awesome. Um, and we have our next guest.
I'm going to bring him up on stage. We have um Andre Breslav who is the a foundational figure in so modern software engineering. He's best known as the creator and lead language designer of Cotlin, the programming language now utilized by over 7 million developers and officially adopted by Google as the primary standard for the Android ecosystem development. After steering the Cotlin project at Jet Brains for over a decade and subsequently founding the successful mental health platform Alter, he's now focused on the next major paradigm shift in computing. He's the founder of Codespeak, a system explicitly designed for the era of AI generated software. As developers increasingly rely on Vibe coding, chatting with models like Claude to rapidly generate pro prototypes, they frequently produce unmaintainable undocumented code bases. Codespeak acts as a new kind of compiler to solve this crisis. It extracts human intent from ephemeral chat sessions and translates it into durable plain English specifications while the LLM generates and maintains the underlying implementation.
Andre, welcome to the show.
>> Hi and thank you very much for this great introduction. Uh I'm very glad to be here.
So, what really caught my eye in studying up a little bit on CodeSpeak, and I I'll give you a chance to kind of introduce the company the way you want to introduce it and and talk about the core ideas because I think this what I'm uh maybe most obsessed with at the moment is kind of a newer um newer product surface for you. But I have this experience all the time where I'm like, "Okay, I'm up to 20 terminal tabs open and I'm, you know, 10 messages deep on average in each and I kind of can't remember what we actually implemented or which ones, you know, I closed my laptop on midway or, you know, any number of other things that sort of happened that I just lost track of my original intent." So this idea that human intent is all that matters and the call to action try intent recovery is um I think a really compelling one even for you know a solo um kind of you know explorer like myself. I'm not even trying to ship enterprise software. Um so tell us you know what you think is kind of most important but then I really am super excited to hear about this uh notion of intent recovery.
>> Uh yeah so just a little bit of context.
So the the sort of foundational idea behind codespeak from the beginning was that uh vibe coding is not the destination it's a step on the journey and there will be future steps and uh we want to build one of those future steps and uh when we started the term agentic engineering was not really popular yet but uh I mean the the the further we go the the more you can see that like coic is at the heart of the idea of agentic engineering Actually when we were starting I wrote down this formula that codes speak equals software engineering minus writing code. So we want to keep all the engineering aspects of it. Uh but you know we we of course see that humans shouldn't be writing code manually anymore. So this idea with intent recovery is pretty fundamental because right now everybody who prompts agents to get working code they're doing work that is being like partly accepted and translated into the code but the rest of it is being discarded and there is this kind of an unfair situation where you know you're talking to your agent in English or like in a natural language anyway uh right and then you get code and you uh check this code into a repo and if you're working in a team other people check your code out but not the human language the code right so you're talking to a machine in a human language but talking to your colleagues on the team in machine language that makes not very much sense so it's it's very obvious that the has to be a next level where we all talk in like reasonably high level language which is close to human language at least. And here like the simple observation behind what we're doing right now at codepeak is that you already wrote these words down. You may have been speaking into a microphone.
Doesn't matter. The words happened and those words were enough to create the code. Like this input determined the code that you got and like it might have been a back and forth and you did some testing and so on forth, but all that input is what determined the code, right? So that input is enough to describe this code and most of the time it's many many times smaller than the out output. So even like re replacing the code with that input would be really nice. Excuse me. Uh but you know the the thing is when you're working with an agent you have uh you know you change your mind basically you you are sort of uh extracting your intent or realizing your intent as you go. So it doesn't really make much sense to just read all your messages from top to bottom. You need to s sort of compress them. You know if you change your mind you need the most up-to-date version. And this is what we do. like we look at at this conversation and um it's a little more complicated than just looking at your messages. But to to simplify, let's say we look at at your messages and we create a specification based on that.
Basically, we extract requirements from what you were communicating. We look at what you requested, what you flagged as errors, which is kind of the flip side of a requirement, and we just put together a list of things you care about that determine the actual output. And then if another person or you later will be looking at this code and we'll have the set of requirements next to it, that gives you a very concise representation of what the code actually does. And you can imagine that this can be happening like with multiple people doing different things in their own branches and then you know if you merge your thing or like submit a pull request or something uh you can look at those requirements instead of code because the code wasn't written by you anyway like what what actual actually comes from a human is the requirements and you know this is how we can kind of elevate what we do to that level and this is what we call intent recovery. So we take this kind of messy session when you change your mind a lot and did a lot of back and forth and we sort of compress it and distill the requirements from it uh and this is your intent like this this is the the essence of what you were doing and basically uh the next step would be if you already have requirements for existing code and you want to change the code the easiest way to do that would be to change the requirements instead of just prompting you know uh prompting all the context around uh what there is already and what you want to change about that is just take what there is as a set of requirements make the changes and say implement and that's that's the overall idea of what we're working on right now.
>> The the observation that you're speaking to your computer in human language it's translating it to code and then your human teammates are primarily just then getting that code without access to the conversation that led to it. I think is a you know obvious in retrospect as many of the best observations are but it's a it's an arresting one because it is a very strange place that we've kind of vied ourselves into in that regard. Um can you I mean you were here for the last uh section of our conversation with Eno previously and I kind of wonder what your take is on some of the same uh questions that we asked him around convergence divergence um I didn't catch any fundraising news from code so kind of wondering how you are thinking about your place in the ecosystem you know is this something that you kind of imagine maturing into another do-it-all platform. um you know that kind of ultimately like automates a lot of the work and and puts people in kind of this um sort of managerial role as he described or do you have a different vision for the endstate or maybe a you know a more kind of uh long-term durable niche that you see yourselves occupying uh while everybody else competes to kind of be the you know the full endto-end uh software factory.
Yeah, I guess there there are different bets people make in the space, right?
So, so some people make a very safe bet uh that works for very few people like you know if you can make very good LLMs go and make very good LLMs they will demand uh but very few people can right so some some people are trying to do this like end to end thing which I think has merit I also think there there is like a lot of uncertainty to that because we don't really know what will work and like we don't have many constraints for that and we know that right now we are very far from full autonomy in agents like they can't really complete serious things on their own no matter what people claim like you know I've talked to some people who make public claims in private and it's all there there there's a lot of nuance to those claims and this is not to say that agents are not useful they're useful I'm not writing code by hand uh but doesn't mean they can't like do everything autonomously so it's a big challenge And what I sort of am wary of there is that there is not enough constraining reality to uh figure out what the end game will be. Uh in what we do I see a lot more constraining reality because what I'm trying to make code speak into is a tool that helps humans in the world of uh coding machines. And there you know my my vision is actually pretty straightforward. So I I don't know what kind of models we get in five years.
Nobody knows. They may be like considerably smarter. They they can be very smart. They can be about as smart as they're today. I don't know. One thing I know is what kind of humans we get in five years. It will be the same kind of humans. We'll be as smart or as dumb as we are today. So I think the the bet to be helping humans is a much safer one.
said what what I'm trying to do is basically help humans navigate the complexity of the systems we build >> assuming that you know machines don't take over so it's not the machines who make all the decisions humans it's it's human job to make the decisions it's humans own the intent and if you're building a complex system you're not getting away with a short prompt it's going to be a lot of information you need to communicate to describe the system you're building and you will not be a single person there you will be a team and we know from years and years of software engineering that organizing a complex description of a system is a challenging task and this is what software engineering is about. If you think back to you know the days of early days of computing, people were like writing machine codes manually and assembly and whatnot and that was a hard part about building software. But then you know we got C and then later we got Java and we got Cotlin and uh you know the level of abstraction in programming languages was raising quite a lot but some things uh just stuck around for the whole time and those things were dry kiss solid the basics of software engineering were always there because it's not about the language it's about what kind of being you are a human being you need these things to navigate complexity and I believe that we will need the same things to navigate complexity in the future. We will be able to manage a lot more complexity. So overall perhaps it will be even more challenging but the fundamentals will be the same and this is why I believe in like agentic engineering as an engineering discipline and this is why we're building something that first of all elevates the level of abstraction like yeah we're replacing programming languages with human language hopefully but then the next step there will be to introduce all the same tools we had in software engineering or in any other kind of engineering we've ever had uh like you know having modules having abstractions, being able to sort of separate concerns in your system, being able to come up with new vocabulary that makes things understandable to humans and so on so forth. Machines will not do it for you >> because it's not their like it's not their role. Uh creating this language is equally important for humans uh to actually using them. So yeah, this is this is inherently a human job, I think.
And uh what we're paid for as engineers is organizing complexity. And I hope this is the job we keep uh as opposed to you know typing in machine commands.
>> So um concretely as I as I understand it codes has a intent extraction pipeline.
You basically start off with the human command and then you extract intent from there and you structure it into specifications which then feed in. Um so and you also very carefully like prune specifications which don't trace back to the human intent as I understand it.
>> So how is it sometimes the human themselves misspecify? I mean this happens all the time with me. I I misspecify all the time and then I have to take a look at the prototype before I'm like oh no that's that's not what I wanted. Right? How do you separate this kind of misspecification on the part of the human from misspecification at the um at the LLM level?
>> I guess there there are two kinds of misspecification. So one is uh sort of self-contradictory or when you request things that cannot be done like you went do something uh you know with uh WS EC2 that WS EC2 does not support uh or like literally you're contradicting yourself uh suggesting requirements that are not consistent and that's also tricky uh but I think what you're asking about is more like when you make an honest mistake and like you don't really know what you want to get and you get something that you didn't want. Uh in my experience, the next step after that is that you look at the prototype, you say this is not what I wanted. I want it changed this way.
And this is where our sort of historic approach to requirements helps because we treat every message as a delta in your requirements. So when we look at the next message we always take it in the context of all the previous conversation and we are trying to extract like what requirements are kept around what requirements are changed what requirements dropped and you know if you say oh this is not what I wanted let's change to something else this is where we just create a diff in your requirements and say okay from this point on these are the requirements that you actually care about >> I can I can just see that helping because uh on every vibe coded as you try to push it into production, you start seeing all of these like issues that you come across and you're like, "Hey, I didn't I didn't mean that.
How h how come, you know, and and I told you kind of like, you know, 10 commits ago, >> right? So, and this is another aspect in this uh it's a very well-known problem that agents often will break working code while building something new, right? because they've just forgot or it's in a different session. They this this session never knew that there was this feature or this requirement and you know I can tell it to like carefully make everything work on on my website on mobile and then I turn to do another feature and that feature breaks all the mobile assumptions because I didn't say mobile in the initial prompt. So uh we can prevent that. Basically we we can just keep all the actual requirements around and we basically we we tie requirements to git history. So if you are standing on top of the main branch then all the requirements that went into the main branch will be actual at that point and they will be in context for whatever new session or existing session runs on top of that branch. I find this very helpful and it's there's another thing we're working on right now that uh kind of surfaces the deltas in requirements every time you submit a new prompt. Uh this is in the works. We we are actually reworking the internals of code speak quite a lot. So the version that accessible right now on the website is kind of previous generation. We are like doing a very interesting rework specifically about git history and matching things and making things a lot faster. So, so yeah and uh I find it very helpful and uh it also helps with other things like you know onboarding a new person like when when I take over someone's uh existing code that they vip coded I can't really browse the code anymore because the code is not like brows I can't talk to claude of course but it's much better to know exactly what my colleague meant when they built that >> and looking at requirements is incredibly that'll be helpful.
>> So, how does this actually get integrated into workflows? Like, are you guys using clawed code and then this is sort of a hook that gets tied in there or uh I'm sure you could do it multiple ways, but what how are you doing it such that I might uh learn from your example?
>> So, the the previous version is sort of a standalone tool. It just generates an MD file with all the requirements for you and you can use that like just put it next to your cloud MD or something.
And uh the next version we will uh make it like more seamless so that it hooks into cloud code and a it adds requirements on the fly and b it updates its understanding of requirements a lot more seamlessly like without you really intervening.
And uh so so that's that's the basically the difference because it's like you can reconstruct uh retrospectively like look at the git history and the session history very carefully and match everything together. It like 99% matches but it's much easier to just be in constant dialogue with you and keep track of all requirements and it's also more helpful because the requirements surface that every prompt and claude is aware of it.
Do you think this has application beyond software? I I'm struck by like legislation. Sounds like the sort of thing that might really benefit from >> a good intent history log.
>> Yeah. So, it's my pet peeve that we should have been doing law in code uh for a pretty long time by now. And I think we'll get to a point where we can turn legal documents to something very close to code. And the important thing is that any like contract, any agreement should be executable. You should be able to run a test like in these conditions like in these circumstances, what does this contract entail? And uh it's tricky to make this work. uh but you know if codesp speakak is very successful I think we can extend what we're doing to that domain as well >> uh yeah and in terms of uh you know generally form factors like what I what I said about uh integrating into existing agents cloud codecs or whatever it's one possible form factor I think there is interesting potential in integrating into IDs because uh a good thing about requirements is that they're sort always around and they're structured. You can surface them in different ways and it's like when you have a an agent session, it's sort of linear context, historic context, but when you are in the context of code, you're in a structured context. You have different components and different level of traction. And it's uh really potentially very helpful to surface requirements uh in relation to the code you're looking at. Like you may not even care about what the code says. It just gives you the the context and the level of traction you want to surface requirements for. So, you know, for for a sizable project, it can be like thousands and thousands of requirements.
You don't want to read them all. And of course, Claude doesn't want to read them all either, but it we know how to figure that out. But for a human, it's really helpful to surface requirements in relation to the given code context or you know we have uh in the works a a plug-in for uh a web framework that lets you just click around in your web app and surface requirements relating to that specific component on the screen.
But it's like it's very limited to graphic user interfaces. But you know so whenever you can sort of scope your requirements through some natural context a person is already in it's very helpful because you don't have to surface like thousands and thousands of requirements. So ids uh potentially like uh developer tools for web frameworks uh you know chrome plug-in or whatever and uh all other natural context a developer will be in.
What have you noticed about the difference in model character? You mentioned, you know, Claude doesn't want to read that. And I, this kind of calls to mind for me various compare and contrast exercises where people sometimes say Claude's a little more humanlike in some ways. Maybe gets, you know, the human condition a little bit better. Uh, Codeex is more maybe diligent, you know, maybe a little little tighter on the instruction following. uh but maybe doesn't have as much instinct to recognize when what you said isn't maybe necessarily exactly what you meant. Uh so what would you say in in your particular uh you know angle on on trying to get value from models?
What have you noticed that might be you know kind of unique takeaways on models and and are you using them um in complimentary ways or are you currently finding like one is kind of just the best for what you need? I'm I'm trying to evaluate more different models but it's extra work so I'm kind of a little behind on that personally like other people in the team are looking into different agents and different models.
Uh one helpful pattern is to uh have one agent as an MCP tool for another like you know if you're your primary agent is Claude you can use codec as an MCP tool that can review cla's work. It's it's not like a hard proven fact that it works better than just using another instance of claude for the same thing, but you know, arguably they have like pretty distinct training sets. So they they can have different perspectives. I personally don't care very much about the style of responses. I always try to get responses like the as robotic as possible, like as concise as possible and so on so forth. So um not super sensitive to that uh like at least to the default styles. I have some like style guidelines uh that I just paste into every agent I work with. Uh yeah there there are also some some very important characteristics uh in terms of latency for example like for some things and and actually like in our case latency is a very important problem because we want to make a lot of small requests and like if you go to opus with every one of them it's going to take ages and this is why the the old version of code speak is kind of slow and uh we we're actually looking into combining different models you know sometimes You can go to Gemini that's Gemini Flash. It's considerably faster, but it's like not as u versatile in some respects or you can try to use a smaller cloud model or whatnot. So, so we're we're looking into these things a little bit. We should do more of that, but I think that would be our biggest concern right now. And uh that uh you know, right now we are sort of in a research phase, so we don't care about cost all that much. But as soon as we get more and more users, we'll be concerned with the cost as well.
>> So do you think you will end up using some of the Chinese open source models?
Is that is that on the horizon?
>> Uh it it can be a thing. Yeah. So Chinese or non-Chinese open source models. It's the interesting thing about the modern software engineering is that more and more of it is not software engineering anymore. more and more of is of it as machine learning and in the machine learning space data is more important than code.
>> Uh very often data is more important than algorithms. So you know if you have a very good data set that kind of captures what you want to do then you can take an open source model and fine-tune it. Uh the big question is where you get the good data set and and that's really tricky and and we we've been doing some work there and we'll keep doing a lot more work there. But yeah, I think uh at some point possibly pretty soon we'll be looking more and more into open source models and fine-tuning them and also like fine-tuning a very broad sense. It doesn't have to be fine tuning specifically. It can be RL or something like that. uh post-training them in some way and that can be a big costsaver for our users. It it can be also a boost in uh latency. So you know there there is potential there and a lot of modern open source models are pretty good. They're still too big to run locally but we can host them ourselves.
>> Cool. Fantastic. I would say we are uh we can put you safely in the category of diverging from the uh end to end uh software factory paradigm companies including factory itself. So that's uh that's good >> and I look forward to the day.
>> First of all, I'll we'll definitely be um trying this out in my stack and see what kind of intent recovery we can uh glean off the you know the cutting room floor of our own work. uh but also really looking forward to the day when you achieve the the big vision of extending this to other areas beyond code and we can live in a um okay >> a much more uh coherent uh predictable high intent high signal legal environment as well. I think there's um >> that could take a bit of time but I I'm really looking forward to this that that world as well. Yeah. I I just I just want to ask maybe maybe one kind of question as a really kind of you know historical figure in programming.
How do you feel about you know the way that software engineering is changing now with these models? I mean looking back I I feel like a lot of software developers have this nostalgia almost for an art that is being lost. So h how does that you know emotionally like how do you feel about this this entire process of handing over the code to the to the model so to speak?
>> I think I'm very skeptical about this actually happening in the in the sense that uh a lot of people are lamenting.
Maybe it's my internal bias which is you know helps me keep optimistic or or something like that. I also remember the UML days. um old enough to do that and I I even caught the the the tail end of working on UML related stuff and that was a complete not like it didn't really work. This one works like much better and it actually brings value. But I also know how much people can sort of project and fantasize about a bright future that like blows things out of proportion. And what I see in reality right now is that software engineers struggle to even improve their own productivity, let alone like delegate everything. And it will improve. People will figure out how to use these tools.
The tools will get better. We will definitely delegate a lot of low-level engineering tasks to those tools. But so far I keep an optimistic outlook on you know humans remaining engineers and as an engineer I never cared about like writing assembly by hand you know some people enjoy that and they remain the experts and they have like good pay well-paying jobs but there are few of those and because there there are few of such people well not because but incidentally there there are a few such people and I'm not one of them so I'm personally I I don't care about doing low-level work. I want to do high level work and I think these things will enable us in doing high level engineering. It's very hard. It's always been very hard and I'm looking forward to to the world where I can like really focus on the hard stuff.
We're definitely looking forward to that world too.
Andre, thank you so much for your time.
We look forward to uh using Codespeak and uh trying it out.
>> Yeah, thanks a lot for having me. It was great to chat.
>> We'll report back. Appreciate it.
>> Thank you.
>> Bye for now.
Um all right, I've got a wrap. I mentioned uh Dean Ball would be mentioned again and it has broken uh news on the uh on the timeline that he is joining OpenAI. So there's a big development uh for him and there's going to be a whole new team that he'll be leading that will be um you know advising open AAI leadership on how to hopefully steer AI policy in the right direction and uh I got to go talk to him now. We're going to hopefully have a episode of the cognitive revolution coming with his kind of full uh backstory reasoning and everything that um has gone into this decision. So, got to be quick on the exit today, but look for that coming soon. And this has been another really good week. We're off tomorrow, >> but we'll be back next week, so folks can also watch out for a weekend uh highlights episode coming soon.
>> All right, bye-bye.
>> Thanks, Posh. Bye for now.
All right.
Calling him back actually right now.
Yeah. Crazy things move fast.
>> Yeah, no doubt.
All right, I'm going to I'm going to let you go.
>> Okay, sounds good. I'm try I'll um >> we'll be around.
>> All right, cheers. Byebye. Twitter.
Related Videos
AI Agent Mastery Certification Course: Lab 4 – Tools & MCP
arizeai
350 views•2026-06-16
Real-time Voice cloning, Kimi K2.7 CODE, GLM 5.2 and 3D reconstruction | AI News
kaiexplainsYT
111 views•2026-06-16
He Believes AI Could Replace Humanity Faster Than Anyone Expects
LondonRealTV
815 views•2026-06-15
General Session by Rami Rahim-The next generation of networking: From vision to self-driving reality
HPE
108 views•2026-06-17
[PLDI 2026] Flatirons 3 - LCTES (Jun 16th)
acmsigplan
191 views•2026-06-16
Google DeepMind’s AI Halves UK Housing Planning Time
60secondsignals
467 views•2026-06-17
The Creators of Claude Code and OpenClaw don't Prompt Their Agents Anymore?!
ColeMedin
569 views•2026-06-18
Why prompt injection is AI's biggest fail
usemultiplier
1K views•2026-06-17
