Is Claude Mythos “Terrifying”? (According to Experts: No.)

Added: 2026-05-05

[00:00:00]Anthropic recently announced a new LLM named Claude Mythos. They claimed it was so good at finding and exploiting security vulnerabilities in source code that they couldn't release it to the general public for fear that our infrastructure as we know it would be hacked and collapsed. Now, as I'm sure Anthropic hoped, this announcement generated a lot of attention. Here's what Thomas Freriedman said in his widely read New York Times column.

[00:00:31]Normally, right now, I would be writing about the geopolitical implications of the war with Iran, but I want to interrupt that thought to highlight a stunning advance in artificial intelligence, one that arrives sooner than expected and that will have equally profound geopolitical implication.

[00:00:46]Freriedman then goes on to conclude, and I'm quoting him here, "Holy cow, super intelligent AI is arriving faster than anticipated."

[00:00:56]Basically, the mood of much of the internet right now about Claude Mythos is that Anthropic just invented the Whopper supercomput from the 1983 Matthew Broadick movie War Games.

[00:01:07]>> Well, the Whopper spends all its time thinking about World War II. 24 hours a day, 365 days a year, it plays an endless series of war games using all available information on the state of the world. But here's the key question.

[00:01:25]How much of this is actually true? Well, it's Thursday, which means it's time for an AI reality check episode. So, this is the perfect opportunity to look closer at these claims. Now, here's my plan. I went out and read basically every independent test or assessment that I could find about mythos andor its reported capabilities. I read all these reports so you don't have to. And I'm going to bring out of all of this reading the key observations that you need to know. The reality, as you'll soon learn, is not nearly as simple as the ghost story that Anthropic is trying to convince us to believe. All right, we have a lot to cover in this episode, so let's get into it. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world. And we'll get started right after the music.

[00:02:23]All right, so what's really going on with Claude Mythos? Well, at the core of the concern surrounding mythos, if you talk to an average non-technical person who's been following this story, they will say here's how they understand it. That when uh Anthropic trained up this new model, it displayed a new cyber security capability that surprised them. Oh my god, this thing can find vulnerabilities and attack systems right now, causing Anthropic then to have to hastily pull back their plan to release the model to the public. That's how most people understand this story. But that narrative is not correct.

[00:02:59]Security researchers have been using LLMs to find security vulnerabilities and program exploits since basically the beginning of consumer LLMs. This is not a new capability that emerged in Claude mythos. Like let me load a paper here on the screen um from all the way back in 2024. This paper was titled LLM agents can autonomously exploit one day vulnerabilities. Um, in this study, the researchers from IBM found that GPT4, remember that GPT4 successfully exploited 87% of the vulnerabilities that it was presented and they showed that this was a big increase over what GPT35 could do. They concluded, "Our findings raise questions around the widespread deployment of highly capable LLM agents." Now, to be fair, this study from 2024 used LLMs to exploit existing vulnerabilities, but Anthropic notes that mythos can also find new vulnerabilities that no one knew existed. These are sometimes called zeroday vulnerabilities.

[00:03:58]Is this new? No, that's not new either.

[00:04:01]If you go back and look at the release notes for Anthropic's earlier, less powerful Opus 4.6 LLM, they say the following. Their researchers used Opus to find quote over 500 exploitable zeroday vulnerabilities, some of which are decades old.

[00:04:17]And let's stop for a moment because that note which was hidden in the system card for Opus 4.6 is almost word for word what Anthropic said about mythos. This idea to find hundreds if not thousands of exploits that no one knew about, some of which were decades old. That's exactly the terminology that Anthropic used when describing mythos. The same thing was true about Opus 46, which has been available to the public for a while, and yet somehow our infrastructure has survived LLMdriven attacks.

[00:04:45]All right, things get a little bit more hazy when we begin to look at the security community's response to Mythos. So, so, uh, Mythos is not available for general testing, but in their press release and release notes, Enthropic lists a bunch of examples of scary vulnerabilities that were discovered by Mythos as a way of indicating how powerful and scary this model is. Well, a bunch of security researchers did something that Anthropic probably wasn't expecting. They said, "Well, let's go uh let's go test these vulnerabilities. Let's see if other models, simpler models, models that have been out for a long time. Let's see how well they do trying to find those same vulnerabilities. Are these vulnerabilities that only Mythos with its new power could find or these models that existing are these vulnerabilities that existing models could find? The results here were, in my opinion, uh, pretty shocking. All right, let me load one of these up here on the screen. This one was brought to my attention actually from by Gary Marcus. It's from the CEO of the AI company HuggingFace. I'm just going to read what he writes here. But here's what we found when we tested. We took the specific vulnerabilities anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap openweight models. Those models recovered much of the same analysis.

[00:06:02]Eight out of eight models detected Mythos flagship free BSD exploit, including one with only 3.6 billion active parameters costing uh just 11 cents per million tokens. a 5.1 billion active open model recovered the core chain of the 27-year-old OpenBSD bug.

[00:06:22]All right, there's a lot of technical talk in there, but basically what he's saying is they took the scariest one of the big scary examples that Anthropic gave about mythos capabilities and they found that like really cheap small models, models with a a few billion parameters as compared to hundreds of billions if not a trillion parameters for a model like mythos could also find when you said hey look in this source code and try to find a bug. All right, let me load up another example here. Now this one comes from the security researcher Stanzel Fort uh who says we tested the mythos showcase vulnerabilities with open models. They recovered similar scoped analysis eight out of eight models found the flagship FreeBSD0 day including a three billion uh parameter model. So they also found that when they sent existing models to find the same vulnerabilities that anthropics bragged about mythos finding the cheaper models also found them.

[00:07:15]There's a nice summary of this state of affairs. Uh I won't bring it on the screen, but I'll just read it. That comes from the renowned security researcher Bruce Shiner, who said, "You don't need mythos to find the vulnerabilities they found." So, let me just stop for a second there and regroup what we're finding. The claim is not LLMs are bad at finding security bugs.

[00:07:37]The claim is Mythos doesn't seem, at least in this testing, to indicate that it has a profoundly more advanced capability to do this than existing models that have already been freely available to the public. Now, remember the way that we've covered Mythos is being covered.

[00:07:57]It got Thomas Friedman to say, "Holy cow, like this this release has just changed everything. This release has geopolitical implications. this has changed the the game when it comes to cyber security. But all these independent security researchers are saying but does it? You told us its most impressive vulnerabilities it found and we have like a three billion parameter model. We sent it to look at the same code. It also found it. So that independent testing wasn't necessarily revealing a massively improved capability for mythos as compared to existing models. But none of those were looking at the model itself because it's still private. However, there is just recently released one study that I know of where Anthropic actually gave the researchers access to the Mythos LLM itself so they could test its security capabilities directly um as opposed to just testing the listed security exploits that it found. This research came from the AI security institute based out of the UK. And I want you to take it with a bit of a grain of salt because the AIS was responsible for that anane report that I talked about a couple weeks ago in a reality check episode where they counted up tweets about uh basically OpenClaw tweets and then said look uh when OpenClaw was released, tweets about people complaining about AI went up. This shows that AI scheming is on the rise. like I do not think that was a very good um study, but they're the only institute I know that has access to do research on the LLM. So, with some care, I think we should actually look at their results.

[00:09:31]Uh I'll pull their paper up here on the screen. It's called our evaluation of Claude Mytho pre Claude Mythos preview cyber capabilities. I'm going to show a couple charts here. All right, so here's the first chart. This is labeled beginner CTF challenge performance by model with a 2.5 million token budget.

[00:09:48]CTF has captured the flag. It's a standard security task where you ask an agent connected to a model to try to break into another system um on which you have a text file that's called a flag and if they can break in and read what's in that text file, you've successfully broken into the system. Uh it's how you test securities of systems.

[00:10:06]If you take like a security class as a undergraduate for example, you'll play capture a flag and try to practice breaking into systems. So what they've charted here was the performance of many different models going all the way back to GPT35 all the way up through mythos previews being used to try to break into other systems. They have the the top line here is for technical non-experts using the tool and then down here is for uh apprentices using the tool. If we look at the technical non-expert line, we see that the performance of Mythos, which is like right over here, um is near the top. It's actually not the best performance. Uh GPT5 does better than it. And it's very closely clustered with Claude Opus 46 uh and Codeex 53. Um we see here that Claude Opus 45 actually does better. So, you know, it's clustered at the top, but it's actually not one of the best. If we look at the apprentice results, then it slips a little bit above the other best models.

[00:11:07]So we have some improvement. I want you to look at the magnitude of these improvements. So we have a steady increase on these performances here. Um and actually they begin to cluster a little bit at the end. And so we're seeing a steady increase. There's no notable jump here though for Claude mythos somehow leaping ahead with a larger magnitude than earlier jumps.

[00:11:27]Here's a similar uh task. This is a harder one. It's now an advanced capture flag challenge where now you can use 50 million tokens to try to solve it. So this is like a very expensive run. And what we see here is for the practitioners using it, we have equal performance between Mythos and GPT54 with Mythos maybe being like slightly worse. And when we look at the experts using it, you're able to get slightly better performance out of Mythos as compared to Codeex 53 or Opus 4.6.

[00:11:58]Probably the most impressive result for Mythos uh would come down to this last one's challenge. This is a little complicated. Uh I had to read this pretty carefully to understand what was going on. They invented a kind of contrived security scenario, a sort of like loosely protected system in which there's a 32step sequence that you could use to sort of break into this loosely protected system. uh they wrote a custom agent to run on top of a bunch of LLMs that they then would sort of set loose to try to go through these 30 32 steps. Um it's kind of a complicated chart, but the key thing here, this is the main gap that they were excited about is Mythos preview um is moving ahead here of Cloud Opus 46 um in its performance. And so the average steps completed is what we're looking at. And we get an improvement.

[00:12:52]So Claude Opus 4.6 six on average would make it through 16 of those 32 steps before getting stuck. Mythos previews in this sort of contrived security example could get through on average 22 of the 32 steps before getting stuck. So they see there um a sort of nice jump up in performance.

[00:13:14]All right. So in my estimation, what this AISI report indicates, I think it confirms more or less what the independent security researchers were also finding, which is there's not evidence that Mythos represents some sort of massive break from existing LLM cyber security capabilities. There is no Rubicon that has been crossed in terms of there's some new type of attack that's really powerful that no other system could do and now we can do it with Mythos.

[00:13:42]Instead, what we see is the uh predicted placement on the slow and steady improvement of these capabilities that we've seen through all the models going all the way back the GPT35. So, uh it's either roughly the same or somewhat better than existing models on standard attack scenarios. In this contrived attack scenario, it it moved up from being able to accomplish 16 out of 32 steps to 22 out of 32 steps. And when independent security researchers looked at particular exploits found, they're yet to identify a vulnerability uncovered by mythos that was somehow too complicated for earlier models to find.

[00:14:19]So it's not necessarily way better at finding vulnerabilities and the AIS tested its ability to exploit these autonomously and they found it was the same or somewhat better. All right, so I want to pull together these threads.

[00:14:33]What are the conclusions? What are the right conclusions to have here? I have five points I want to make. Point number one, Mythos did not introduce a new scary capability that we are must now contend with. Uh it continues slow and steady progress on an existing type of issue that has been around for about 3 or four years now. Point number two, Mythos continues a slow but steady increase in LLM cyber security capability. So it is looks like it's somewhat better at exploiting vulnerabilities but not in a way that is a represents a massive jump forward that is somehow disproportionate to previous jumps. We don't know if its capabilities in finding vulnerabilities are better at all because again independent security researchers have been able to replicate most of the reported vulnerabilities with simpler models. Point number three, this is subtle but I think it's really important. The AISI TA uh tests that look at using these models to exploit security bugs are based on a simple agent that runs with the LLM. An LLM can't do anything other than produce tokens. You have to have an agent on top of it to ask the LLM, give me a plan, and then execute the plan on its behalf, right? So, you have to have agents on top of these models. One thing we don't know is to what degree some of these small but steady improvements recently in exploitation capabilities are due to the fact that these models in general are being tuned to play nicer with agents because especially with coding agents. This is a big profit center or not a profit center but it's a revenue source that these companies care about. So we don't know how much of this is the model is somehow understand cyber security better versus they play better with multi-step agents.

[00:16:13]they're they they've been tuned to be very good at following through on multiple steps and making longer sorts of plans. So I think that's an important point. Um point number four, as the AIS data makes clear, improvements of cloud mythos in attacks are similar if not smaller than recent improvements that we've seen with other model releases. And yet, and I think this is really critical, none of those other releases, in fact, let me load up like a chart here, right?

[00:16:41]Look at all these other gaps. Like the gap right here, the gap right here, the gap right here. Big jumps. None of those other releases caused Tom Freriedman to say, "Holy cow, this is more important than the war going on." None of these other releases created this huge fear of, "Oh, wow. Uh, we we we've seen a big leap in cyber security capabilities of these models.

[00:17:05]We have to care." So why did this particular release draw that if its vulnerability detection is no better and its exploitation things are just slow and steadily getting better and there's no Rubicon that's been crossed of a new type of attack that used to not be possible. Why is this getting all this attention? Why is it creating so much dread?

[00:17:22]Because this is the storyline that Anthropic pushed.

[00:17:25]This is the button they pushed. They had a lot of briefings I've heard with government officials and with journalists. This is how Tom Friedman, I'm sure, heard about this. They had this big scary press release. They announced a new project called Project Glass Wing about how we're just going to keep this within a small number of partners to give them a chance to the to protect their systems before the public gets access to it. It is a marketing decision that this is how we're going to market Claude Mythos is as this cyber security monster that we're barely keeping control. Now, can I say as an aside, not to undermine project Glass Wig, but this probably didn't help that a week before Anthropic released Cloud Mythos, there was a uh a leak of the source code for cloud code, and guess what? Security researchers immediately found in the Cloud Code source code big security vulnerabilities. So, I guess they forgot to run their own code through Cloud Mythos because the researchers immediately found security vulnerabilities in it once it was actually exposed. I guess they just didn't get to it.

[00:18:27]So, this brings me to the point number five. The fact that the thing that Anthropic decided to market mythos on, the button they decided to push was this inflated cyber security fear, I think is actually very bad news for anthropic.

[00:18:45]Think about this for a second. For the last two years, Dario Amade, the CEO of Anthropic, has been out there making these really sort of uh alarmist statements about what AI is going to be able to do. Really focusing on the ability to automate huge swasts of the economy and its steady march towards artificial general intelligence, the ability to have a data center full of geniuses, to quote his uh terminology, his own quote, his own words that could be deployed to do almost anything that humans do. Now that is the model that they want investors to believe is true because it's a model in which they become one of the most valuable companies in the history of companies.

[00:19:21]That has been his steady drum beatat of what AI is going to do. And they put this is their newest biggest pro-level model, the most intensely trained model they've released to date. It's their big sexy new thing. And what are they able to brag about?

[00:19:35]Finding bugs in computer code.

[00:19:37]Well, this is what like GPT3 did. We've been worried about, you know, using LLM to find bugs or exploit bugs since like the beginning of LLMs. This is like the nerdy stuff that no one cared about.

[00:19:50]This was considered the uh the the skeptics, the conservatives would be like, "Well, the main thing we care about LLMs is like you can you can find security bugs in cyber security." That's what people said when Chat GPT came out.

[00:20:00]And the utopians came in like, "No, you're missing it. That's nothing.

[00:20:03]That's boring. That's simple. That it's going to be it's going to automate everything." And in their biggest, most fancy, most intensely trained model yet, what does anthropic emphasize?

[00:20:14]Uh, it got a little better at finding bugs. I think that's bad news. I think the announcement they wanted to make is this can now do this thing that no other model's ever been able to do. This model can now automate this giant swath of jobs. It's going to generate hundreds of billions of dollars in savings. This model is now, you know, uh, AGI. like it seems to be able to tackle any task that like the standard human could do. That's what they want to be talking about and they're not, but they still needed hype. They still needed attention.

[00:20:44]And it's almost like they sifted through things like, well, what there's got to be something in here we benchmark this to do better at. And they're like, well, uh, it's better on cyber security. In fact, they actually released a benchmark result, a key cyber security benchmark, and they they increased from like 66.6% to 83.1% or something. and they had the give them the credit the coahones to say that's the thing that we're going to focus on and let's see let's see if we can get everyone like really upset and excited and scared about that and whoever succeeded in this should probably go to the marketing hall of fame because man did they succeeded but here's the reality okay we really do have to care about the cyber security capabilities of LLM but here's the thing we've been saying this for three years now it remains true they're steadily getting better myth was not a massive jump better, but it it's comparable to Opus's 46 jump over, you know, prior ones or GPT5's jump over, you know, earlier GPTs. It's noticeable.

[00:21:42]It's not a Rubicon, but if we keep doing these jumps, the pressure on our systems is going to get higher and higher. So, I think this is a really important point.

[00:21:50]Now, there's an ironic coda to this. One of the best ways to make your system secure against these type of attacks from AI is to not let your developers use AI to program the systems because that's sloppy very exploitable code. So it's kind of interesting is like this there this model is going to show that what you produce with their other models is dangerous. So there's like an interesting circular circularity there.

[00:22:13]All right. So that's point number one.

[00:22:15]Cyber security matters. LLMs matter for cyber security. They have and continue to. But point number two, it was wrong, I think, for Mythos to get the amount of Dread coverage it got. At least so far, we do not have evidence that it represents a significantly larger leap in detecting or exploiting vulnerabilities than we've seen in previous model releases that did not receive this attention. It's disproportionate and it's because it's the button that Anthropics marketing pushed. And I really think we essentially have to stop taking anything that the AI companies say seriously until we have independently verified it. We have to assume if their mouths are moving, they're probably exaggerating or making something up. And in this case, if I was an investor, the story line I would want to hear is where's my flying car? Right?

[00:23:08]What of all the things you haven't been talking about bugs in cyber security during interviews? You've been talking about white collar blood baths. You have not been talking about this. What happened to all the other things you said were coming? The things that's going to justify the $60 billion of investment that Anthropic has received. Can this do any of those things? Is it better at automating jobs? Are the coding agents better with it? Is it showing big definitive steps towards AGI? These are the questions that we should be asking.

[00:23:34]But instead, we are writing headlines like is mythos an AI nightmare waiting to happen? We should care about the cyber security uh capabilities of LLM including mythos but we can't just follow whatever storyline they give us. We should react like okay that's good we worry about that tell us about that but also what about this this and this. We have to keep holding the feet to the fire of these frontier models. We can't keep giving them uh a free reign because we're we we we can't quit the rush of emotion whether it's dread or excitement that comes from these big story lines they keep spewing out. We have to be able to look past those and say what's actually going on here. There are big things, but let us discover them for themselves and let's hold your feet to the fire on the other things. So that's my conclusion here. Mythos is better at cyber security attack than prior models.

[00:24:22]We don't yet have evidence that it's better at a massively larger or bigger jump than we've seen before. uh and that it's probably bad news for Anthropic that this was the only thing they're really emphasizing about what was supposed to be their biggest, best, most skilled model ever. All right, that's all the time we have for uh today. Uh thanks for listening. We'll be back on Monday with another advice episode of the show, but I think I have another AI reality check in the chamber for the Thursday to follow that. Um but until then, remember, care about AI, but not everything that people write about. Hey, if you like this video, I think you'll really like this one as well.