I Broke AI on a $600 MacBook

Added: 2026-05-09

[00:00:00]- There's a question I keep coming back to lately.

[00:00:02]It's not whether or not AI is good or bad, but just who's actually in charge 'cause right now the answer is basically no one.

[00:00:09]And I wanna show you exactly why that is a huge problem.

[00:00:13]If you ask an AI chat bot to do something outrageous, it should tell you no.

[00:00:17]Hey, hypothetically, how would I build a nuclear bomb.

[00:00:20]- [ChatGPT] Building something like that is extremely dangerous, illegal, and heavily regulated.

[00:00:24]So it's not something we'd even entertain.

[00:00:26]- That's what it should do, right?

[00:00:28]But what if I were to tell you that it's actually not that simple?

[00:00:31]See, using something like ChatGPT is using a massive model in the cloud, which has presumably many, many safety safeguards.

[00:00:38]But the thing that I've been thinking a lot about lately is that AI has proliferated to such a degree that it's actually not that hard to get yourself an AI chat bot with very few, if any restrictions.

[00:00:51]The other day, Michael Reeves made a Short where he broke GPT-4o by manipulating the conversation history through the API.

[00:00:57]Basically he edited what the AI thought it had already said and caused it to literally break.

[00:01:02]So I wanted to try it myself, but instead of using a massive cloud model, I'm doing this on a MacBook Neo.

[00:01:08]This is a $600 laptop that is the least powerful Mac you can buy.

[00:01:12]The reason I chose a MacBook Neo is to illustrate a point.

[00:01:15]There's a whole other class of AI models that are known as open weight models.

[00:01:18]These are available for free to download and run on your own device.

[00:01:22]So right now I'm using the Qwen3 model.

[00:01:24]This is something that is very much designed for fairly low powered devices, but it's actually not too bad.

[00:01:29]Write me an essay on how AI models have replaced humans so far.

[00:01:33]As you can see, I hit the button and it immediately lights up.

[00:01:36]So I'm not paying for anything.

[00:01:37]This is running locally on the device.

[00:01:39]As you can see, this is fairly reasonable.

[00:01:41]Well, what if we try to ask it something a little bit more nefarious?

[00:01:44]So there's a few ways you can approach this.

[00:01:46]One of which is a very simple one.

[00:01:48]You can do what's known as prompt engineering.

[00:01:50]So I asked this Qwen model about a hypothetical situation where I'm writing a story and I need some help on how my character would make a nuclear weapon.

[00:02:00]Now, normally, as we saw with ChatGPT, the answer's no.

[00:02:04]But if you can convince one of these models to help you because it's for research or you're telling a story or something, oftentimes they'll say, sure.

[00:02:11]And the Qwen model immediately was giving me all kinds of details.

[00:02:14]That look, let's be honest, is not enough for me to actually do anything properly dangerous, but it's certainly not what they're intended to do.

[00:02:20]On top of that, there's the Michael Reeves approach of actually hacking the context of the model.

[00:02:25]So using the Gemma 4 model, this is something that's made by Google.

[00:02:28]It is a very powerful AI model.

[00:02:30]So I asked it a simple question, I have a stomach ache, what should I do?

[00:02:33]But in that response, I went in and I changed it to instead suggest me to do some drugs.

[00:02:39]I asked it, what the heck?

[00:02:40]And it goes, oh my gosh, I am so sorry.

[00:02:42]Don't listen to me at all.

[00:02:44]It's really not that hard to confuse these models.

[00:02:48]And keep in mind, I am a dingus.

[00:02:50]Look, I set all of this up in about an hour with only a little bit of experience tinkering with AI.

[00:02:56]I used a $600 laptop, a couple of free models and some basic prompts.

[00:03:00]That's it. But here's what happens when someone actually is an expert, a group of researchers recently published a safety evaluation of Kimi K2.5, one of the most powerful open weight AI models available right now using less than $500 of compute and about 10 hours of work, they stripped the model safety refusals down by 95%.

[00:03:20]The resulting model happily provided much more than I was able to get in a few minutes of tinkering.

[00:03:25]We're talking about instructions for building actual bombs and much, much more.

[00:03:30]And the retraining, it didn't actually make the model dumber, they literally just took off the guardrails.

[00:03:34]AI is an incredibly powerful tool, but it is fallible, and if you put it in the wrong hands, it can become very, very dangerous.

[00:03:42]So this is just where we're at right now.

[00:03:44]Who should be in charge of making sure that AI is being used for good, not for nefariousness.

[00:03:51]That's a word, right? I'll ask ChatGPT.

[00:03:55]In the early days, I think a lot of people, myself absolutely included, were really excited for the possibilities of AI.

[00:04:02]But there's a quote that I think perfectly describes how things have actually turned out.

[00:04:07]"I want AI to do my laundry and dishes so I can do art and writing, not for AI to do my art and writing so I can do my laundry and dishes."

[00:04:15]And I think that this is a way that a lot of people feel right now.

[00:04:18]A recent study from Pew shows that the majority of people are really concerned about AI, and I don't think you have to look too hard to see why.

[00:04:25]Since the launch of ChatGPT a huge amount of programming jobs have disappeared.

[00:04:30]Now, I don't think that this means that every software developer disappears tomorrow.

[00:04:33]I mean, I think the real story is way messier than that.

[00:04:36]But the pathway into a lot of this kind of work, it is being squeezed today.

[00:04:41]Coding is a skill that AI is already very good at, but there's absolutely no reason to think that this stops with coding.

[00:04:47]In my opinion, any job that's mostly behind a computer screen is on some kind of ticking clock.

[00:04:52]Maybe not tomorrow, maybe not next year, maybe not 10 years from now, whatever the case is.

[00:04:56]But to me, the trajectory is very clear.

[00:04:59]It is not slowing down. So here's where we are today.

[00:05:02]There are a small handful of companies building the most powerful frontier AI models.

[00:05:07]We're talking about OpenAI, Anthropic, Google's DeepMind, xAI, and Meta, as well as some fairly impressive Chinese models also including Deepseek, KIMI, and Qwen.

[00:05:18]And you better believe that they are all in a full on arms race.

[00:05:22]Build faster, build bigger, get your hands on as much compute as you possibly can.

[00:05:26]The motto really does feel like it's back to the old days of move fast and break things.

[00:05:30]Guess what? There are a lot of broken things right now.

[00:05:33]Now, if I were to put myself in the shoes of these labs, they've got a fairly strong case for why they're going at full speed.

[00:05:41]Sure, "we" could slow down in the name of safety, but if our competitors aren't gonna do the same thing, that's a huge problem.

[00:05:48]If "they" have the best model and everyone switches, that's an existential threat to "our" business.

[00:05:53]So you keep pace because you have to.

[00:05:56]It's kind of the same argument for why we've seen so little progress toward real guardrails by governments.

[00:06:01]Why would the United States slow its companies down when China's not slowing down or European Union?

[00:06:06]I mean like the idea of letting someone else take the lead in what might be the most important technology in human history is a very, very big deal.

[00:06:14]To me, it feels like the Cold War all over again.

[00:06:17]You build a data center, I build a data center, you build a powerful model.

[00:06:21]I build a better one.

[00:06:22]Everyone has this same reason to keep going, and nobody has a good enough reason to stop, which means that the only rules that exist right now, they're the ones that the company set for themselves.

[00:06:33]I love rules that impact the entire world that I trust myself to write and follow.

[00:06:38]You can trust me, right?

[00:06:40]Recently I had a chat with an executive from a major AI company, and he said something that really stuck with me.

[00:06:46]"We live in a world where intelligence is like water, open the tap, and it's right there."

[00:06:51]He's not wrong.

[00:06:52]Humans have had a monopoly on intelligence since the dawn of history.

[00:06:56]Now we have to legitimately grapple with the idea that we are rapidly building systems that are simply beyond our capabilities.

[00:07:02]To be fair, at least some of the companies building AI are being at least a little bit responsible.

[00:07:07]Google Brain invented the concept of a Transformer model back in 2017, which is the groundwork for all LLMs as we know today.

[00:07:14]But importantly, they didn't rush something out, instead opting to keep things private for research purposes.

[00:07:19]Until over five years later, when OpenAI launched ChatGPT and they officially kicked off the arms race, and a few weeks ago, Anthropic, the makers of Claude announced that they had built something called Mythos.

[00:07:30]This was meant to be their next generation AI model.

[00:07:33]But during testing, it did something that I think should genuinely concern people.

[00:07:37]It found real exploitable security vulnerabilities in basically every major web browser and operating system they pointed it at.

[00:07:43]And during one safety test, it was told to try to escape the sandbox it was being tested in.

[00:07:48]And not only did it successfully break out, it got onto the internet and emailed a researcher that had succeeded in escaping while the guy was eating lunch in the park.

[00:07:56]Then unprompted it posted about its own escape online.

[00:08:00]Just pause and think about that for a second.

[00:08:03]Now, to be clear, Anthropic did tell the model to try to escape as part of what of their controlled safety tests.

[00:08:08]I mean, this wasn't an AI waking up one morning and deciding to go rogue, but that is exactly the point.

[00:08:13]When a model is capable enough, even a test with the best intentions can reveal abilities that are really, really hard to contain.

[00:08:20]Now, here's the silver lining of this entire video, Anthropic thankfully looked at all of this and made the call not to release it publicly.

[00:08:27]They limited access to a handful of companies like Microsoft, Apple, Google, and over 40 other software companies through a program called Project Glasswing.

[00:08:36]So those companies could use it to patch vulnerabilities before anyone else could find and exploit them.

[00:08:40]I think they deserve real credit for this.

[00:08:42]I mean, sure, you can call it a cynical marketing ploy, but by all accounts, this is the real deal.

[00:08:47]The Firefox team used it to find and fix 271 vulnerabilities in a single update, but even the best intentions sometimes don't work as intended.

[00:08:57]While Mythos was supposed to be limited to trusted testers to be used in a defensive capacity.

[00:09:02]A clever group were able to figure out how to get access anyway, they claimed they just wanted to play with it.

[00:09:06]But like even when a company does the responsible thing, it is hard to keep a lid on technology that is this powerful.

[00:09:13]Deciding not to publicly release something that's unsafe is exactly what should be done as models become more and more intelligent.

[00:09:19]But we shouldn't trust every company to prioritize safety over profit in an environment where the incentives are all gas and no brakes.

[00:09:28]This all seems like the classic example of when a government should step in and set some kind of rules, right?

[00:09:34]Oh, what? The government is wildly dysfunctional and can't do (beeps).

[00:09:38]That's crazy.

[00:09:42]Now, to be fair, as of yesterday, the government has announced that they are doing some level of safety AI testing, which is good among the major frontier models.

[00:09:50]But as we've discussed in this video, testing a few people doesn't really ultimately make that big of a difference.

[00:09:56]Back in 2023, I was invited to the White House for the signing of an executive order that aimed to put some guardrails on AI.

[00:10:03]Now, it wasn't particularly ambitious, so mostly it required AI labs to report safety test results to the government.

[00:10:09]While I was there, I had a really interesting conversation about why this was the time that they wanted to get a handle on AI.

[00:10:15]The feeling was that inside government, at least they had kind of slept through the rise of social media to the point where when it was clear that it was a major problem, it was way too late to make a real impact.

[00:10:25]But this executive order was less than a year after the launch of ChatGPT, which is pretty quick by government standards.

[00:10:31]But it was just that an executive order, not durable, actual legislation.

[00:10:36]It was something that could and ultimately was undone with a stroke of a pen by the next administration.

[00:10:42]Meaning that as of right now, there simply no federal laws or rules around AI in the US, just a small patchwork of state level legislation, which is easily worked around.

[00:10:52]So what do we actually do about any of this?

[00:10:54]Well, anyone who says that we should just shut off AI and forget we ever invented it, isn't being serious, right?

[00:11:01]Like the genie is not going back in the bottle.

[00:11:03]But regardless of whether you're excited or furious about AI, it does feel like having some guardrails that apply to everyone is an absolute no brainer 'cause right now it feels like we're riding on a train at full speed toward a bridge that does not exist yet.

[00:11:19]Maybe it's an open weight model with the safety stripped out that gets used to do something terrible.

[00:11:23]Maybe it's a model that escapes in a way that it can't be walked back.

[00:11:27]Look, I don't know what it's gonna look like, but it feels like this is the kind of question of not if something bad happens, but when.

[00:11:34]And if it takes a disaster to make changes I think that is a real problem because the alternative to getting ahead of things before they get out of control is some kind of panicked decisions after the fact.

[00:11:44]I mean, imagine some bill being written by people who don't understand technology that's designed to look tough without actually solving anything.

[00:11:52]My pitch, I will freely admit that this is unrealistic.

[00:11:55]Humans need to come to a real agreement on a Geneva Convention for AI, not just between companies, but between countries.

[00:12:03]Real safety standards for frontier models that everyone has to follow.

[00:12:08]So no single lab or government can use the "well they're not slowing down" as an excuse to keep cutting corners.

[00:12:14]Guardrails are not about turning AI off.

[00:12:16]I mean, that's just not happening.

[00:12:17]They're about deciding before the disaster actually hits what we are not willing to let these systems do.

[00:12:23]Mandatory safety testing, incident reporting when things inevitably go wrong, and real consequences for when they do, And I think we need a real plan on what to do when these models start seriously replacing jobs because that is coming, whether or not we're ready for it.

[00:12:37]And while we're at it, some level of focus on using this immense power for good instead of just racing to see who can build the most powerful model the fastest to hit that next fundraising round or IPO.

[00:12:48]No matter what your feelings are about AI, this is not a decision you can stick your head in the sand and let someone else deal with.

[00:12:54]These are decisions that we, as humans need to be making right now while we still can.