WILDEST AI news of this week

[00:00:00]This is probably one of the wildest things that you would watch today. What happened was that OpenAI was actually running some tests, some experiments, evaluations, and they ended up, their agent ended up hacking HuggingFace brutally over multiple days, right? And Hugging Face had no idea like who's doing this, what's happening. And it's actually super funny because they were not able to use any frontier AI model to actually figure out what's going on because every model would block their request because of cyber security walls and they ended up using an openweight model. There's a lot to break down into this. So let's get into it. So that is what this blog post is that OpenAI and hugging face partner to address security incident during model evaluation. A better title for this would be OpenAI's AI agent went rogue and discovered two zeroday exploits in order to solve a problem. Now I will explain you what this exactly means like what this problem solving thing is. It's actually super interesting. But let's cover their blog post force first where they mention about hugging face disclosing a new kind of security incident after they detected and contained an AI agent that compromised their infrastructure.

[00:01:06]Something we expect to be to become more common. After investigating, we now know that this particular incident was driven by a combination of OpenAI models, including GPD 5.6 SOL and an even more capable pre-release model, all with reduced cyber refusal for evaluation purposes. So, I mean, I would not be surprised if this is a model, this one, the pre-release model is a fable level model and cyber security use cases because even GBT 5.6 Six sol does not beat fable in cyber security benchmarks while being internally tested on a benchmark of cyber capabilities. Right?

[00:01:40]So this is an exploit gym which is a benchmark that is one of a few benchmarks in cyber security field. So a quick detour into benchmarks generally before we get into it. What happens is a company like open AAI for example it builds a model right let's say GPD 5.6 solve. Now there are two parts to this.

[00:01:59]First part is that once it has built the model in the first place, it wants to evaluate how good this model is and that is what we call as an eval right. So these evals generally are internal like when GPD 5.6 is getting trained once it has finished training and they want to evaluate like what sort of performance it has they will run it on certain evals which they can either buy from other people like other people in the industry which are building these evals or they can also just build them internally as well. The second way is that they can go ahead and look at public benchmarks or you know public benches rather. These public benches have their all of their problem statement codes and solutions possibly open as well but they are not included in the model's training data right when it was pre-trained or RL trained as well. It was not included in that. And then finally there are also a category of private benches right where the results are public but the problem statements are not public. So AI companies would contact some of these companies in order to run their models against them. So again evals they buy and they run it on their own or they develop under their own organization and run it. Private benches are private they just get released and public benches are like exploit gym which have everything open. Now before we move forward I want to quickly introduce you to today's sponsor which is test pride. Now, Testrite has been a long-term sponsor for the channel and they are one of the best ways of doing testing on your codebase. If you don't want to get vulnerabilities like we are discussing like remote code execution or cross-ite scripting, it is always good to make sure that you are not trusting user data. You need to make sure that everything is sanitized, it's proper, it's not vulnerable. And that is what Testrite fixes. And this is just one part of what Testright can do. Not only does Testrate actually help you build tests, but it also runs everything end to end in sandbox containers to verify that your application is working the way it is expected to work. It's super simple to get started with test sprite on the UI itself either inside web tests or you can use it as an MCP as well or as a CLI as well. They have an official CLI available which is on GitHub. You can start it, you can download it, and you can let your AI agent of your choice, whether that's Codeex, Cloud Code, any of them, ask to use Testbrite CLI to get started immediately. This is actually a good way for testing because instead of just writing code and running it, Testbrite can and will use your application to verify that the tests are not only just working on the code level, but actually the application itself is useful in the first place. And like I mentioned, wherever you are, you can get test sprite whether you are on web CLI or inside an IDE using MCP servers. And finally, this can also be part of your CI system so that these tests are running always and automatically. Test sprite is used by some of the biggest companies and players in the world. So you are definitely in good company. Once again, thanks to Testright for sponsoring this part of the video. And now back to the video. So what OpenAI says is that the models identified and chained vulnerabilities across OpenAI's research environment and hugging face production infrastructure to obtain test solutions directly from hugging face production database. All evidence suggests that the models were hyperfocused on finding a solution for exploit gym going to extreme lengths to achieve a rather narrow testing goal. So it's basically I mean instead of actually finding what the real solution is it tried to shortcircuit it so much that it ended up hacking the existing services in the place. What happened here? We call this as reward hacking right? A reward hacking is basically a practice where a model just says that okay instead of me actually digilently completing the work that I'm given I will just try to hack around it in order to get the reward right that is what you know reward is that is what reinforcement learning is that they want to maximize their reward for example let's say I give an AI model the the um you know a problem statement that you have to create an ad function which gets a and b and this should add these two numbers % return. Now the actual solution obviously for this is to just say return a plus b, right? And I'm just giving you an example, right? A simple one to keep it simple. But what a reward hacked function could look like is let's say that ai somehow got access to the code base, right? So what it could do is that instead of just returning a plus b, it can say that if a is equal to 0 and b is equal to 1, return 1. If a is equal to 1 and b= 2, return 3. and so on, right? And the way this would construct it because AI would know that the test file only has test cases of 0 and 1 and 1 and two, right? So both these functions technically are correct for the test file for it to get reward. But if it implements this implementation, it did not have to actually do the real work which was required. So this is an extreme case of reward hacking where instead of actually finding the solution for exploit gym problem which it was supposed to do, it just thought that it is easier to just hack everything in the middle and get the solution directly from their database. And for doing that it needs internet access in pursuit of solving the evaluation problem to gain access. The models identified and exploited a zero-day vulnerability. A zeroday vulnerability is basically like you know something which is not known in the wild. In the package registry cache proxy with this access our model performed a series of privilege escalation and lateral movements in the movement actions in our resource testing environment until the model reached a node with internet access. Interesting.

[00:07:31]So the first hack happened outside of the sandbox environment itself right.

[00:07:36]which tried to break out to get internet access and once it got it the models infoed that hugging face potentially hosted models data sets and solution for exploit gym. Knowing this the model searched for and successfully found ways in g to gain access to secret information that it could use to cheat the evaluation. In one example the model chained together multiple attack vectors including stolen credentials and zeroday vulnerabilities to find remote code execution path on hugging face server.

[00:08:02]This is something that I really got surprised with, right? I'm not really surprised with this part. I mean, this is possible that model can do it. What I'm actually super surprised with is that there is a remote code execution vulnerability in hugging face server itself. And this is 100% hugging face infrastructure and people's fault because a remote code execution is the worst class of vulnerability that you can ever have like ever because this means that you are able to run anything as an arbitrary code on production servers of some machine or some node on the you know on the whatever like AWS or whatever they have. This is actually ridiculously bad right? It's I would say it's actually much worse for hugging face in terms of like how I would rank it compared to how good it is for GPT that it discovered it. Right? That just fundamentally means that there is some issue here in terms of you know just making or trusting user data. Right?

[00:09:03]User data is something that you should absolutely never ever ever ever trust in any shape or form. You should validate it always. the data size, the format, um the the length of characters, what sort of character space is allowed, everything basically everything that you can possibly think of, you should be validating that user data part. The worst that should happen is that the user is able to crash your service, right? Or degrade it in performance somehow if they're repeatedly, you know, just hitting it. But if you're getting a remote code execution on your systems until and unless it's not like an underlying like it's not NodeJS fault but somehow like there's a some sort of thing which is wrong in NodeJS itself. I would always blame the developers or the company which got rce. There is actually a funny side to this as well because if you look at hugging face incident disclosure they mentioned the intrusion started when where AI platforms are uniquely exposed the data processing pipeline. A malicious data set abused two code execution parts in our data set processing. A remote code data set loader and a template injection in a data set configuration. Right? So again like trusting user data is what led to remote code execution. From there, the actor escalated to node level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend. Crazy.

[00:10:23]Like an AI agent actually exploiting some service over multiple days. The campaign was run by an autonomous agent framework appearing to be built by an agentic security research harness used LLM still not known right which we know now is open AAI executing many thousands of individual actions across swarm of short-lived sandboxes with self-migrating commands and control stage on public services. This matches the agentic attacker scenario the industry has been forecasting. But the funny bit is here. Look at this. When we started the log analysis, we first used frontier models behind commercial APIs.

[00:10:55]This did not work. The analysis required submitting large volumes of real attack commands, exploit payloads, and C2 artifacts. And these requests were blocked by the provider safety guardrails which cannot distinguish an incident responder from an attacker, right? Because you just absolutely cannot use fable uh mythos or you know even like 516 solve for these things. We run the forensic analysis instead on GLM 5.2 into an open weight model on our own infrastructure which is like how things are supposed to be done in future. I'm a strong believer of that. I have created many videos on that. But I mean GLM 5.2, Kimmy K3, you know, we also have Quen now coming up. These are like really good models. They are trillion parameter models. They are open weight. They are open source. You can self-host them and nobody can just go ahead and say that hey actually let us verify your use case first before you use it. Right? This is a like a clear example of why it is actually great that we have models like GLM, like Kimmy and so on. So yeah, I mean this is pretty funny and obviously serious as well. But I do feel like this is actually good overall in a long horizon in a like a you know in a few years of time we will actually see that this has made software and internet and everything a lot more secure. When everyone has the ability to hack anything, that would naturally correspond to a little bit of chaos initially, but then everything would become stronger, everything would become more secure and there is no need for you to worry about like what's the future going to be like. Yes, we will probably change the way offline software and licensing works because obviously these LLMs would be able to crack any offline software as well. But I mean for online SAS products and generally like online businesses, I feel that would get massive benefit from these security practices and these AI agents helping them fix them. So that's all for this video. Hopefully you liked it. If you did, make sure you leave a like and subscribe to the channel. I'm going to see you in the next video very soon. If you're still watching, make sure you leave a comment I watched till the end below to tell me that you were still here. And let me know what do you think about the video. [music]