Saving Bug Bounty Programs + AMPScript, tessl & GPT-5.5 (Ep. 174)

Added: 2026-05-15

[00:00:00]And it's so readable with camel case.

[00:00:02]>> No, it's way more readable with underscores. You're so used to reading with spaces.

[00:00:06]>> But but it's I don't >> I'm a Python bro. And we got to name all our functions with underscores.

[00:00:11]>> Don't freaking tell me you're a Python bro. I am a Python bro.

[00:00:19]>> Best part of hacking when you can just critical think.

[00:00:29]Yeah, dude.

[00:00:38]>> All right, guys. You know the drill.

[00:00:39]This is the segment of the show where we typically do an ad, but today we got something different for you, and it's an exciting announcement that we are launching the critical thinking pentest group. Okay. Um, we have an amazingly talented group of hackers that live in the CTVBB Discord and in the community um that we're so grateful to be able to work with. And we have a spec particular niche of that called the full-time hunters guild. Um, these are people that are validated to have performed very high level in Pug Bounty. Um, and we are going to be performing pen tests uh for you the listener. if you guys have a scope that you want to be tested. Um, ourselves, meaning me, Reszo, Gretme, me, other members of the CTBB team, and pulling highly qualified members that are aligned with your scope specifically from the full-time hunters guild to also do uh penetration testing as required.

[00:01:30]Um, so we're going to make sure with all the hackers that we have access to that the hackers that are working on your scope are extremely qualified in exactly what you would like to have tested, right? Because we're pulling from a big group of people. Um, so if you're interested in that and you think your organization could benefit from that, check out pentest.ctb.show.

[00:01:48]It's going to be a lot of fun. We're going to tear stuff up. It's going to be amazing. Um, so yeah, drop us a line if you're interested. Um, and then the second piece here is, of course, I want to remind you we do have the full-time hunters guild available. So if you are a hunter that is out there and is killing it and wants to be among other high performing hunters, this is the place to do it. Okay? ctvb.showftthg FTHG for full-time hunters guild. Go ahead and apply if you meet the criteria. Um, and you'll get access to not only a highc caliber community, but also opportunities like the pentest uh, opportunities that we have coming through critical thinking. So, lots of exciting stuff going on. Um, excited to work with you guys if you think you could benefit from some pentest or if you want to join the full-time Hunters Guild community. All right, I'll see you there. Peace.

[00:02:35]Sup, hackers. got the this week in bug bounty segment for you before we hop into the episode. Um the first item on the list today is a article by Yes We Hack entitled cost AI frontier models and more a measure take on the future of security testing. And as we've been talking a ton on the pod recently about you know what is the future of bug bounty? What is the future of uh continuous security testing as we know it? There are a lot of takes and there are a lot of different opinions floating around. So I think it's really important to get a bunch of different perspectives. So, uh, I'd recommend you go ahead and read this article, um, and get a little bit more data, uh, in. And we've also got another article from, uh, Integrity, uh, that we'll link in the description as well called Common AI misconceptions debugged. Okay. Um, and this is also sort of talking about the way that AI is affecting the industry.

[00:03:24]So, two really good articles, one from Yes We Hack, one from Integrity on it.

[00:03:28]Um a quick takeaway that I had from the integrity one was this right here which is um uh validity ratios remain constant even though the volume has been increased dramatically uh across bug bounty at this point right from 2022 to 2025 submissions grew 328%.

[00:03:48]Um and we're going to see that I'm sure even more this year um with with uh AI becoming more prolific.

[00:03:55]Um, but that does mean a lot of more valid reports, too. So, we're just getting a lot of volume. Um, and we got to figure out how to how to sort it out.

[00:04:04]Um, so those are some articles for you to take a look at. Uh, last but not least, we have a Bounty Sync Plus Social event uh that is being held by Integrity in uh the UK in London. That's Thursday, May 21st. We're going to link that uh down below. But that seems like a good opportunity to connect with people in the London area um in the in the Bug Bounty world. So, if you're in the UK, uh definitely check this out if you're near London and can make it over for this event. Thursday, May 21st um in London. Uh all right, I think that's it for the Twib. Let's go to the show.

[00:04:38]>> Dude, so I step away for one week and you put out a Doomer episode, bro.

[00:04:43]>> Listen, I did a I did an awesome ad read. I made a custom thumbnail. I carried a good episode.

[00:04:52]>> Thank you. Thank you, Jason. I appreciate it.

[00:04:53]>> You're right, though. It was a little bit of a doomer episode.

[00:04:55]>> It was a little bit of a doomer episode, man. Um, and I feel you. I feel you. It is concerning. I'm not going to lie. The the landscape is concerning right now.

[00:05:03]>> Hey, I know you probably want to jump to other stuff, but it's on the I'm actually going to do it right now. I'm going to share my screen here.

[00:05:09]>> All right. I made a since we're talking about the last episode, I made a image to represent what I was talking about.

[00:05:18]>> Oh my gosh, dude. What is this?

[00:05:20]>> I really think that this >> funnel why Bug Bounty is dying. What the heck, bro? This is a Bug Bounty podcast.

[00:05:26]>> Yeah, I know. But anyways, listen, the point is this is what people need to be concerned about and it's what they need to push against. This is why they need to adapt and evolve and go deeper and incorporate AI into their hacking. But uh yeah, I feel like that I kind of explained it well, but the point I was trying to get across was this basically, right? Is that AI written code's probably going to get better. There's going to be AI code review, internal hackbot testing filters more, and then when you get down here, you got to compete with the sharks with the likes of me, hence the shark.

[00:05:55]>> Yeah, exactly. So, yeah. Anyways, >> the the superpowered shark, the shark on steroids because we're using clot as well, >> right?

[00:06:03]>> Um so, yeah. No, I I do agree with you.

[00:06:06]But on the flip side of that, you know, they have we also have non software engineers pushing code to production because you can just will will an app into existence now. Y >> and Lord knows that you know the DevOps people or you know whatever are going to also have some problems with how these things are pushed to prod. So, um, yeah, I think I think there will be plenty of stuff to go around, at least for the next 5 years, but it's definitely going to be it's definitely moving faster than I anticipated before. So, >> Right. Yeah. And then, like you said, there's so much counter pressure. It's like there's there's like pressure to get out code way faster. There's way more code going out the door. There's nontechnical people writing so much code that there's no way the technical code people can actually review it. So, yeah.

[00:06:50]>> And and I think we just don't understand the code at a depthful level as much anymore, right? Like I I don't even know. I wrote an app yesterday and the day before um and which I'll talk about eventually on the pod, but I wrote it. I have no idea what API functions are in this thing, >> right?

[00:07:04]>> Not not even a single clue. I've never even looked at the code. I literally just talked to Claude >> and it's beautiful and it works exactly how I want it to work and it's nuanced and it's lovely, but >> I have no idea how it works. Yeah.

[00:07:15]>> You know, so >> yeah, that's scary. And that's why you need some bug hunters to test that out for you, Justin.

[00:07:20]>> Yeah. Seriously. Seriously. Um okay. Um, so, okay, before we jump into the actual um, meat for this episode, I got a little confession, man, because I've gone on the pod and I've flexed and I've said, "Your boy is above burnout is essentially what I've said last time on the pod." Like, burnout, that's for those newer full-time hunters, right?

[00:07:41]You know, um, got a little burned out, man. I got a little burned out.

[00:07:46]>> It only took three back-to-back events is all it took.

[00:07:49]>> Three back-to-back events overlapping.

[00:07:52]And I don't know, man. Freaking Matias Carlson, man. He goes hard. That freicker goes hard when he's in a live hack game event, you know. And >> so you felt the need to keep up.

[00:08:02]>> I felt the need to keep up, you know, and and so I started putting in these crazy hours. I got kids. He doesn't have kids. Screw you, Matias, for not having kids. Um, and and so it's like, you know, I just can't can't get the same volume of hours while still being a present parent, you know, that I used to be able to. So I was like sacrificing the sleep and then Korea hits and I get all these crazy bugs and I'm jet-lagged as freick. And then I come back and I'm like, "Okay, you know, such an awesome time." And then why am I not wanting to create content? Why am I not wanting to hack? Why do I just want to lay here?

[00:08:39]You know, >> dude, it's real. I don't know, man.

[00:08:41]>> And and I think it's not just burnout.

[00:08:44]Like, it's also probably the fact that you're you're doing incredible this year on top of that. You don't have some like ridiculously large financial need pushing you to go find those bugs because you're also doing well. So, it's kind of like on both ends.

[00:08:55]>> It is. It is a little bit on both. But I feel like, you know, I feel like I've had so much fun hacking these past like couple weeks and then all of a sudden, boom. Like, why is the passion not there, >> you know? And then and then, you know, just needed a little time. It just need a little time to come back. And it's back. It's back, baby.

[00:09:13]>> Good.

[00:09:14]>> All right. Um, I got a bunch of stuff.

[00:09:16]You got a bunch of stuff. You want to go first or should I?

[00:09:18]>> Uh, I feel like Yeah, I already went first with that image, so you got to go now.

[00:09:21]>> Oh, okay. All right. I'm up next.

[00:09:22]>> You stole this one. You stole this one from me, by the way. It was going to be on my list, and then I get in here and it's already here and I'm like, "Oh, come on."

[00:09:28]>> All right. Well, I'm going to explain some really, u crazy [ __ ] on this. So, uh, you know, if you want to add to it, you can, but I'm I'm going deep in the crypto bin here.

[00:09:38]>> Perfect. I'm gonna talk more about the AMP script. So, >> okay. Well, here why don't you start us off with AMPScript and then I'll go talk about the um the the crypto stuff.

[00:09:46]>> Yeah, sure. So, um basically >> Yeah, sure.

[00:09:51]>> Uh AMPScript is a custom language basically. It's like >> So, so first let's let's intro it. This is Search Light Cyber's write up on uh the ghost of encryption pass. How we read all your emails in Salesforce Marketing Cloud. So, this is a story of how they pawned Salesforce Marketing Cloud through Ampscript, some crypto stuff, all all that all together, right?

[00:10:13]>> Yep.

[00:10:14]>> Okay. All right. Go go go with AMPScript.

[00:10:16]>> Uh I I've not like since this was on yours, I didn't end up like prepping it, but no, no, no, no. I'm still going to talk about this because I think it's really cool. So, one, basically, AMScript is like serverside templating language. That's that's mostly what you need to know. that is specific to SFDC um Salesforce and specifically to their marketing cloud and uh I mentioned a vulnerability oh man like maybe a year ago that me and me Evan Connley and Chubs found which used uh one of the things in this blog post. So if you scroll down to under foot gun number one treat content. So there's this really cool treat as us content which basically means hey evaluate this >> and then you can also do uh http getit.

[00:10:58]So we hosted more ampscript on shubs's server and then and then input for our 50 character payload treat as content http getit his URL. So then it would basically pull his it would it would pull more ampscript from his server and put it inside of that treat as content.

[00:11:15]So if you're ever limited in a AMPcript injection insertion point, you can use that that backtoback.

[00:11:22]>> Nice. Okay. So you kind of um incurred a double evaluation scenario, right? You had a template injection.

[00:11:31]>> Yes.

[00:11:31]>> And then and then you you needed to get you know more character space.

[00:11:36]>> Yes.

[00:11:37]>> And so you hit an HTTP get and then took the output of that dumped it into street as content and invoked another template injection.

[00:11:43]>> That's that's exactly right. That's sick, dude.

[00:11:46]>> Yeah, that's why I tweeted about it because it was so cool. I should have I should have broken it down like in a whole bunch of tweets, but actually, let me see if if I can share it. I can >> you can track it down. Uh, essentially the the the um the full flow here just to give the the listeners, you know, knowledge about the the article is this is how they poned uh Salesforce Marketing Cloud. They found a uh template injection in the subject line of emails uh because of double evaluation and it's using this weird AMPScript method uh of templates which is percent percent equals or the lesserk known version which is um curly bracket curly bracket equals and that triggers the AMPScript. So make sure we're adding these to our template injection payloads, right? Um, and then you know using this template injection they're able to do some crazy crypto stuff which I'll explain later you know and and it's a a tangental part of the scope but it does come back to the template injection. Um, and yeah really get access to a bunch of the data in the system. Um, that's the full flow of this. And what I want to talk to you guys uh about in just a sec is this uh crypto attack that they did later on in the article. Um, >> I'll show this. I'll Yeah, I'll show it super quick and then then I'll switch back to you. So, >> it's funny because like obviously Marketing Cloud is all about marketing, so it's not in the title, but this was in the first name.

[00:13:06]>> And uh and so the funniest thing was basically getting an email with the exfiltration of all of the users in the database as as your name. So, it's like, "Hi, Justin and and Douglas and Joseph and everyone," you know, like it's like it's like the email is like like a hundred pages long in your Gmail and it's just a huge data leak. But anyways, yeah, it was the first name build and then we put the extra AMPScript uh stuff living on shoved the server here and then it worked like this. So anyways, I thought people would think that was cool. Yeah, >> dude.gg is such a good uh domain.

[00:13:38]>> It is. It's cool, >> man. I I have a good one though. I have a good one. I'm not going to say it out loud, but I'm pretty proud of the one that I have. Um actually, I'll tell you, Richard, just bleep it. I've got >> Yeah. Yeah. Yeah.

[00:13:50]>> That's pretty freaking great, right?

[00:13:52]>> That's very cool. Yeah.

[00:13:53]>> Yeah. Um okay. Um so let's get back into this. So uh first you know the the they get a template injection um via double evaluation in the subjects or titles of emails. Um and then they're essentially what they're trying to do is get um they realize that these emails are also you know available at a domain, right? And so they look at, you know, they click on the like view in browser button or whatever and they look at the the um resulting URL that comes from that and it has this, you know, QS parameter that seems to control where, you know, what email is displayed. So it's got some encrypted values, right?

[00:14:37]>> And they enumerated a bunch of these and they found out that there's three primary formats for this. One of them is a JT that looks pretty solid. The next one is is a is a, you know, hex. And then finally, one they find later is just, you know, raw parameters. Um, I didn't find that till later, I guess, though. Um, and my first point that I wanted to mention here was like this is a really, you know, you're sniffing for blood. Your spidey senses should be tingling here when you see this implementation change because probably somebody found something and and they, you know, sort of fixed it, but they didn't backdate it, right? Um, and a lot of times the pipes that glue all of these things together are the same. So if they're leaving the old, you know, format in place, then you might be able to exploit that even with the newer format being present.

[00:15:27]>> So very interesting.

[00:15:28]>> And the way this often looks because I' I've been a developer before, >> I feel like a lot of times whenever you need to support multiple formats, you'll say like if it's encoded, decode it.

[00:15:37]Else treat it as it is. And so honestly I think I thought what you were going to say as you approach that that um that lesson there. I think the lesson is >> often you should try other formats in parameters like just across bug bounty in general across abstack testing. You know it's kind of the same thing behind like oh if a JSON post request works you should try URL form encoded right? It's like the same thing. It's like if you see and and I actually don't do this very often but I'm going to start now.

[00:16:04]If you see a B 64 encoded payload, try decoding it and then still sending it.

[00:16:09]Sometimes it still works and then that might let you do something funky in the app, right?

[00:16:12]>> Yeah. Or, you know, an encrypted version. One of the things I've seen many times is, you know, after a little while the company realizes they screwed up and switches to this encrypted ID format, right? And then but they're old >> backwards compatible.

[00:16:26]>> Yeah. They have to be, you know, backward compatible at least a little bit, even if it's just for that like minute when they're doing the migration.

[00:16:31]and they forget to never take it away and uh and you know you can still just put the unencrypted version of the ID into that field. Um I've seen that many times. Um but anyway, going back to the this whole situation here, they use a little OS int on this parameter. They find out what the hex, you know, encrypted hex values, what it decodes to. Okay. Uh they find that from a Stack Exchange reference article.

[00:16:57]>> That's awesome.

[00:16:58]>> Which is just lovely. And the days of that unfortunately are are you know few because uh uh like they've got um you know stack stack exchange the number of questions and answers has just like tanked y >> um because of AI >> but um anyway this is the you know the the format that they have and knowing that they start sort of doing some bit flipping on the QS parameter which is what I've said on the pod many times is like don't shy away from these crypto bugs. People implement stuff poorly all the time in an unauthenticated CBC algorithm, guys. You do bit flipping and as long as it doesn't cause like just flip a couple things here and there and sometimes you'll see it actually passes through and then the result of the decrypted, you know, piece which is in the response will just look corrupted like this right here. Um, for those of you on YouTube, right? It'll just look like a, you know, question mark or, you know, a different character or something like that, right? And that means that they're using an unauthenticated CBC, you know, algorithm. Okay? And so anyway, takeaway there, do bit flipping when you see encrypted stuff. Okay?

[00:18:10]Because you found I found a lot of padding oracles and I found a lot of bugs just from doing that. Okay. Um, the really crazy thing about this exploit that I didn't I did not know, and this could just be me not knowing, you know, about uh this exploit, or it could be that they came up with this, um, is that they figured out a way to get the IV for this CBC by doing this really cool technique. Okay. And I I'm not going to lie, I typically nowadays I just read something, you know, in the articles and I have no problem coming on the pod and talking about it. Yeah, >> I did rehearse the segment >> because it is it is hard to talk about like crypto stuff in the audio medium.

[00:18:50]Um, and without a visual, you know, representation, right? I'm going to do my best. I'm not sure how it's going to go.

[00:18:56]>> I'm going to close I'm going to close my eyes and test you. I'm not going to read the article or like I'm just going to close my eyes and listen to just your audio.

[00:19:02]>> Well, okay, that's great. And and I would also recommend for anybody that wants to understand this better, take this whole article and put it into Gemini and then just ask it questions until you understand. Okay. Uh because that's what I did before to refresh my memory on this and it and it's lovely.

[00:19:17]So what they actually did here, the way CBC works is that there's an IV initialization vector that gets um used with CBC and that is what is exorded with the cipher text of the first block.

[00:19:31]Um and so and that that produces um you know the output. So what what we they did here in order to get that IV so that they could decrypt the rest of it. So I want you guys to think of it like this.

[00:19:44]The the decryption algorithm is you know a function D and inside of that is cipher text block one right and then the output of that function gets exorded with the IV. Okay. So D of the cipher text one exord with IV. Okay. So what they did was they patted out the first part of their cipher text with eight null bytes. Okay? So that C that C1 which would be the first block that gets passed into the the decryption algorithm, it becomes all null bytes.

[00:20:21]Okay. And so the when it gets passed into the decryption algorithm, the output of that is just going to be junk.

[00:20:27]And that gets um that gets >> is it predictable junk?

[00:20:32]No, it's just junk. Okay.

[00:20:33]>> And that gets exorded with the IV. Okay.

[00:20:36]So now your junk that that comes from the null bites tanks the IV for you, right? Um and then the way that the second block is is um you know built out is it takes the the second block which was our original first block and it passes that through the decryption function and it exorses it. It exores it with the um cipher text of the first block which is that eight to null >> by null. Yeah. Which is all null, >> right? Which you control, right? So then the result of that is now you have the raw de decrypted you know of that old first block. Okay. So now you've got the old the raw output of the decode from the first block and you've got the IV encrypted version from the first block which is the the normal you know output.

[00:21:30]>> So you can get the IV out.

[00:21:31]>> So then you you do this with over a bunch of different valid IDs and you know a little bit about the plain text from the Stack Overflow article, right?

[00:21:41]and you brute force each bit of the IV until it matches the the pattern the output matches the pattern that you know that the actual plain text has.

[00:21:51]>> Yeah.

[00:21:51]>> Okay. And then over the course of a bunch of IDs you are able to narrow down and extract the IV and then using that IV you can decrypt the first block originally and get the raw text out.

[00:22:04]>> Um >> so it is it is awesome what what they did here. I I think that came across decently. I don't know. Maybe you can tell me.

[00:22:11]>> No, no, no. I think it came across fine.

[00:22:13]The question is, were they smart enough to know that or did they just use AI to do that?

[00:22:17]>> I don't know, man. P, you know, they're also using Padre, which is a um a uh, you know, system for doing these padding oracle CBC related attacks. Um, so that's cool. And also, shout out to Padre. I've used it a couple times.

[00:22:32]Really useful. Um, very extensible. Love that about it. Um, and the the other key thing that I want you guys to take away here, one, that eight byte, um, null bite trick is amazing. Um, two is that, um, definitely, you know, make sure you're using Padre for these sort of things. And you, three, you need to make sure you know, um, the format of the do, you know, something about the plain text version of the value you're supposed to be extracting, right? because that allows you to sanity check your IV checks. Um, so anyway, that's my rant.

[00:23:09]It's a lot of fun getting, you know, I'm not an overly mathematical dude. Like, I don't love math or or crypto or anything like that. I do love crypto, but like I don't love the whenever I look at with those functions with those squiggly scripts and like, you know, all of that stuff. I get overwhelmed and I don't like that. But um you know actually working on these specific exploits and finding these bugs is some of the most gratifying experiences I've had as a hacker dude. I freaking love it.

[00:23:33]>> Well and I think now it's more accessible than ever before because like with with LLMs like you can basically reason through the the the part that you can't comprehend. So like you can basically explain you can like reexlain how you understand it up to the point at which you don't understand it. And then the LM can basically be like oh here's the part you're missing.

[00:23:52]>> Right. So yeah.

[00:23:53]>> Yeah, totally. Um, so anyway, that's my >> wrap this up. What do they do with that?

[00:23:58]>> That's my that's my rant. Um, once they, you know, crack that and are able to they also find an encryption oracle. Um, and then they're able to use that to uh craft these arbitrary um, you know, values that are inside of this encrypted text and extract all emails and all email contents from every single person in all of Salesforce Marketing Cloud because uh, they use the same um, key across everything. So, um, and they asked that that key be not, uh, >> in the blog post >> put in the article, and I'm like, "Oh, geez, that means that [ __ ] is still active." Um, so I don't know, man. It's It's tricky. It's tricky.

[00:24:42]>> It is tricky.

[00:24:44]>> Yeah, let's not go down the disclosure pipeline.

[00:24:46]>> Yeah, I've said I've said my piece there. Um, yeah, definitely look for, but I last thing that I wanted to add was definitely look for all sorts of variants of crypto stuff, right? Look for the JDT version. If there's a hex version, if there's an unencrypted version, you want to use way more or whatever to get all of those old versions of the URLs and audit those old implementations as well.

[00:25:11]>> Sweet. I am gonna talk about skills, which is kind of funny uh considering that we did the whole episode on it.

[00:25:17]Which actually, by the way, I do want to take a quick second here. I have gotten ridiculous amounts of positive feedback about that. Like I've talked to so many people who who told me that they just have been like shilling it to everyone they know. Uh, and then I just got off a call with Ethiak because, you know, I'm an adviser with them. And Andre told me for this event, >> um, he wanted to like kind of compare, you know, their hackbot with with like just using like cloud code, but he had never like set up cloud code before and he said he the only resource he used was he listened to our episode and and then that's it. He he just used that to build out skills and to set it up and then he used that and he found 17 bugs and he was like, I know for a fact I wouldn't have found over half of them without cloud code. So, >> pretty sweet, right?

[00:25:59]>> Yeah, man. In the last live hacking event that we were in too, literally all of the showand tells had a shout out to Claude Code in them. And I'm like, "Oh my gosh, this is nuts.

[00:26:08]>> It's changing everything." Yeah. And so, um, >> he said he said specifically that it also found a zero day in a Java image processing library. So, you should reach out to him because it's pretty cool.

[00:26:19]>> Oh my gosh. Holy crap.

[00:26:20]>> Um, so I wanted to talk about and I can share my screen super quickly here. um a skill optimizer. It is released released by Tessle. Tessle T SSL is a pretty neat company. They do like spec as code stuff, but they have an entire um skill marketplace or it's called a registry.

[00:26:40]Um kind of like skills.sh, but um what they do is they automatically evaluate those skills and then rank them the highest. So, it's kind of a cool skill registry. Anyways, I'm not trying to shield that registry. I just love this skill. They sent out an email about this skill optimizer skill.

[00:26:55]>> Um, it's a little bit meta to talk about this, but basically just point your agent at this at this link that we'll put in the show notes. It's called Tessle Skill Optimizer. And what it does is it sets up a whole bunch of evaluations for your coding agent to see how likely it is to be invoked and then and then attempts to improve that through optimizing the description, through optimizing the content, through optimizing the name of it. And bro, blew my mind. In mine and JD's hackbot, like over half of them went from like 10% invocation rate to 85% invocation rate.

[00:27:31]And it's because we want though.

[00:27:33]>> Well, I'm saying like across the evals like so this is like the Tesla skill optimizer attempting to like write prompts that that where it should invoke and then testing it. It's not invoking across 85% of our runs. I'm saying that like the quality of when it should be invoked based on what the user wants went from 10% to 85%.

[00:27:53]>> How do you tell it when it should be invoked?

[00:27:58]>> What do you mean?

[00:27:59]>> Like like so why am I trusting this skill optimizer to know >> because your description already has when it should be invoked, right? Like you already have that in your skill in the description.

[00:28:10]>> Okay. So it's looking at my description.

[00:28:12]It's saying here are a bunch of test cases which align with that description.

[00:28:15]Yes.

[00:28:16]>> And then it's saying, does it match the description that I just generated all these emails from?

[00:28:22]>> No. Then it's firing those. It's firing all of those tests it just wrote based on what when you want to invoke it >> and seeing how many it invokes on >> and seeing how many invokes on, right?

[00:28:31]And so then it came out to like 10 or 15% for a bunch of my skills.

[00:28:35]>> Wow. Okay.

[00:28:35]>> And so then then it attempts to fix it and then it reruns them.

[00:28:39]>> Interesting.

[00:28:40]>> Yeah.

[00:28:40]>> Interesting. Anyways, my whole point of this is the the main point which is which is like uh really the the the lowhanging fruit that I think everyone in the audience should check is their description front matter field. So as skills in markdown there's a thing called front matter at the top right it's like the the title the description like references or author or something right >> the description field at least in cloud code and I think also in codeex is what gets passed to the agent assuming that you're you know your main prompt isn't so long that it gets truncated anyways the description field there is what actually gets um sent to the agent so that's all it has besid has the title and the description to decide when to invoke right that description field in markdown is not uh multi-line. And so if you have description colon and then new line and then you have some descriptions or you have description colon the first line and then you go down to new lines, all of those extra lines are not getting picked up because markdown does not do multi-line by default. You have and so this is what it fixed on a ton of ours.

[00:29:46]So So here's what you want. Here's what everyone wants. You want you want the description colon and then greater than dash.

[00:29:54]After that, it'll it'll pick up every single line. It'll massively impact include it. Let me actually I'm just going to do a new doc and show everyone what I'm talking about. Just super quick for anyone watching on line.

[00:30:03]>> Yeah, dude. That's crazy though because that means like half of your descript, you know, everything but the first line of your description might not be making it into claude, right?

[00:30:11]>> Which means it may not be triggering these whenever you you need them to be.

[00:30:14]Right.

[00:30:14]>> Right. Exactly. So let's say you have a I have a skill called reszo skill, right? And the description here >> is like this. You know, normally I would say like I want you to invoke when you see a 403, right? The only thing that was going to the agent was this.

[00:30:30]>> Oh my gosh.

[00:30:31]>> And I think in a bunch of mine, I had something like this >> and I thought that this and uh for the listeners basically I had something that was like description colon just greater than. And I think this does work in some markdown formats, but this does not work for the way cloud code works. So, you want it to be like this. Description colon greater than dash. Yep. That's exactly what you want. And that will that will fix all It's a really weird format. I've never seen it before. That is what you want. And it will fix all of your skill invocations.

[00:30:56]>> Okay. I I I also thought when you said greater than, you probably meant less than. No, he actually means greater than.

[00:31:01]>> Greater than dash >> description greater than dash. Yep.

[00:31:05]>> Yeah.

[00:31:05]>> Um Okay. So, a couple things here. It also fixed for many of mine camel case into underscore case. I don't know why, but I think that it got higher invocation rates with that. So, if your skill names are camo case, switch them to snake case. And >> I freaking hate that.

[00:31:22]>> Why?

[00:31:24]>> Cuz I You're a camel case, bro.

[00:31:26]>> Yeah, dude. Totally. All day.

[00:31:28]>> Oh, don't me. Snake is so much better.

[00:31:31]>> It It inflates the size of things. And it's so readable with camel case.

[00:31:36]>> No, it's way more readable with underscores. You're so used to reading with spaces. But but it's I >> I'm a Python bro and we got to name all of our functions.

[00:31:44]>> Don't freaking tell me you're a Python bro. I am a Python bro 2.7 back in the day that was the standard and then they changed it.

[00:31:54]>> Okay. Well, listen here. At least tell me you're a twospace Python bro and not a fourspace Python bro.

[00:31:59]>> Get get out. Get out. Get out. You're you are fired.

[00:32:04]>> Do you actually use four spaces or do you use tabs? Please tell me you're not tabs.

[00:32:07]>> I use tabs. I use tabs, bro. I use tabs.

[00:32:10]>> And you're on Windows and you're on >> I don't use anything now because I I >> This is This is Maybe this is one of those situations where, you know, opposites attract, right? Where we're >> Yeah.

[00:32:23]>> Yeah. Okay. All right. Anyway, so you're saying somehow for some reason Claude is wrong and bad and it wants you to use underscores. Is that correct?

[00:32:31]>> Yes. Yeah. So skill name should be underscores or at least in my testing it it uh invokes better when you do that.

[00:32:37]And the very last thing I was going to mention is some of my skills had the word claude in them. Apparently that is like a huge no no. So if you have like I had a skill called DM other clauds and I would allow my cloud locally to message my cloud on my VPS.

[00:32:51]>> I changed it to DM other agents and all of a sudden it like massively improved.

[00:32:55]I think that the word claude is like a reserved word and so it either gets like parsed or or doesn't view it as well or something. Um, yeah. I didn't look into the nitty-gritty details, but yeah.

[00:33:05]>> Interesting. That is That is pretty cool.

[00:33:07]>> Yeah.

[00:33:07]>> Okay.

[00:33:09]I I'm just going to take a breather for a second, you know, from that that massive disappointment that just uh, you know, hit me. Um, and I guess we'll talk about It's all right. You know, Watchtowers got me covered. They're going to they're going to, you know, get me focused on this next write up. Okay.

[00:33:25]>> So, um, this next write up is the internet is falling down.

[00:33:30]>> Falling down. falling down. Uh, and it is none other than the C panel off bypass, which is breaking everything.

[00:33:37]So, by the time this episode airs, you guys probably have already like digested all of this, but I did want to walk through a couple of the um, beautiful pieces about this exploit. Um, one, it's Pearl. So, um, you know, when you're looking at the, um, the, you know, actual code here, it really helps a lot, right? And, you know, sometimes we run into these situations where we're trying to reverse engineer something and it's like a binary and it's just such a pain. Um, I will say like I've I've hacked on a couple things that are Pearl or um, you know, Python and it's just so much more fun. Uh that being said, I know for a fact a lot of good hackers have looked at this and not found this bug. Um so really cool that um you know this vulnerability was discovered and it this is you know watchtower reverse engineering it um reverse engineering the patch. Um so here is the situation.

[00:34:36]um they look at the patch, they figure out that they find this beautiful comment that says filter against slashrn uh from values before writing kills the crlf injection primitive against the on disk key value record format and that is the most in-depth message I have ever seen in a batch in my entire life. Um >> like it almost gives it away. Yeah, it totally like really I mean not not totally gives it away but I mean it gives you puts right to the meat you know like the primitive that you need.

[00:35:09]>> So the situation here was that um part of the off material was being parsed and then written in the um file system to a file. And guys I just freaking love this. Let me see if I can find the file uh right here. Right. um var cPanel sessions raw and then the session ID and I just love this [ __ ] guys this is exactly what I'm talking about like you we've got to thoroughly assess these systems and and understand how the full offflow works right is it actually writing files to disk you know with with your authentication material and in this scenario it really is and that should just being bing um you know in your head when you see something like that uh and the way that they had to make this exploit work was uh They actually used two different authentication methods.

[00:35:59]One from a cookie and one from an O header. So that's another really beautiful thing is like look at different routes for authentication.

[00:36:07]Long story short, by combining the cookie and the authentication header, they're able to smuggle a SLR/n in and get that written to disk uh in this session descriptor file which is SLR/n delimited. Okay, so now they're injecting arbitrary attributes into a session. Um, and they were able to establish a pre-author session by uh submitting an invalid um, you know, username and password, right? Um, and so that was really cool. Now they're injecting attributes into there, but they keep hitting these roadblocks and I just I love the way Watchtower writes, dude. They're like, you know, they're like, and that's that's it. Is this the big red button time? You know, and then they're they submit it and they're like, [ __ ] you know, like what's going on? Uh, why isn't this working? Why are we always treated so badly is what they say?

[00:36:56]Um, and then they hit another roadblock with their 403. Um, and there's like a cached version of this, you know, r/n delimited file that is loaded into JSON.

[00:37:06]And the endpoints prefer the cached version. So now they've got to find, you know, a primitive in the system that uh forces a reload from from the the raw file rather than the cache. And then they finally find that and and they make it work. And then they keep going and then they they run it again and they hit a 403 again, right? And um so they just keep going and keep going and keep going and do we deserve this.

[00:37:31]>> Yeah, you'd think we were done at this point, you know. Um give us strength. I I love these articles, man. Um but uh anyway, as they continued down with the the exploit, they figured it out. They got the cash invalidated. they inject in injected another, you know, value into the um file which bypassed a a um defense in depth measure where they did the password check again and finally they got the full off bypass. Um and I just think that's such an example of like force of will in these exploits, right? Obviously, if they're patching it, there's something that's been done here, but they are persevering, running into blockade after blockade after blockade and getting the full O bypass, which is freaking >> I think there are lots of moments when companies trying to reverse patches >> uh end up finding tangental bugs because like because like that first primitive was there and they got that easily, but then they got blocked and then they had to find another primitive that isn't exactly clear in the patch. And then when they find one, it's like, oh, was that actually a different route to the vulnerability?

[00:38:39]>> Yeah, I bet, you know, who knows what the actual thread actor was using or the original finder, you know. Um, >> so very cool. And actually sort of mentioning that, um, Search Light Cyber also did a write up on this O bypass and they released a C panel um, a high integrity, you know, high fidelity check, which is what they always do, which I freaking love them for, right?

[00:39:00]is they release these like really quick and easy little scripts. You can point at a a host and say, "Hey, >> is the is this vulnerable?" Right.

[00:39:09]>> Yeah.

[00:39:09]>> Awesome. They're so cool for that. Um and they added two things to this one.

[00:39:14]Um they added uh the way a third way of accessing this. You know, there were two ways mentioned in the main article, but there's actually a third way through um a another port that's open on cPanel where you can hit this super weird reverse proxy rule and it and it um hits one of the management ports um in the back end. So there's actually you can't just block off the management ports in the WHM service uh to to patch this bug.

[00:39:42]You have to also patch uh you know block off um a third one. And I'm trying to figure out which one it is. I don't have it right in front of me right now, but there's another service that you need to block off as well. Um, and then finally, they also mentioned that, hey, if you're checking for that invalid password to get the the pre-auth session that I mentioned, if you're doing that on route every time, it's going to lock everybody out. Uh, so here's >> it. Is it just crazy to me that they keep this stored on disk in anyways?

[00:40:13]Like I just would assume this would always be in a database in like every app.

[00:40:17]>> Yeah, dude. I don't know. I mean, that does give me legacy vibes. I know PHP also does some stuff like this, so maybe it's just a, you know, Pearl PHP legacy thing. Um, but yeah, I think it is a little whack. And um and yeah, just wanted to shout out the um the SL Cyber team there for releasing the scanner and also finding some really good um other mechanisms that don't cause lockout uh to confirm that the vulnerability exists on your target. So we'll we'll uh we'll link that in the description if you guys want to check that out as well.

[00:40:48]>> Perfect. So I um wanted to give just a little bit of feedback from the last episode. Mhm.

[00:40:56]>> Obviously, I know we talked about a little bit at the beginning. Um, I've been doing like u hacker advisory board for hacker one and then also bug crowd just here recently because I think they bring me in because of the AI knowledge and stuff and then they're just all struggling, right? I mean, all the all the programs and platforms are struggling with this volume.

[00:41:16]>> And so, I kind of thought it would be neat if for program managers who are listening and also platforms. I just wanted to bring one like kind of key piece of advice that I thought of in the most recent HAB meeting. Um, it's nothing confidential or anything, but I just think that for companies that are struggling and for the fact that when platforms are struggling, if they basically gave this option to all of their programs, it would reduce the total traffic and total volume by like, you know, 50% or something crazy and allow everyone to catch up and then allow us to get back to have faster triage and faster bounty times, right? Cuz it's hard on everybody. they feel behind on their SLAs's. We feel frustrated because they're taking forever to get stuff triaged and to get b and payouts out.

[00:41:57]And then when you're waiting a long time in triage, sometimes the bug gets fixed and then that's really frustrating, hard to handle >> happening.

[00:42:03]>> So, so I think that my um I just have this like one little concise pitch that I think would be really beneficial. If you're a program manager or you're a CSM for a platform and you've got companies that are struggling, just offer them like this like uh plate, this like platter of options. It's like here's the things you can do and you should do some of them. You could take your program private or you know I don't love that idea but this is just one option. You could require you could require videos for for for and we talked about this even like you know >> you could require videos for your um reports. You could require higher signal so you could bump up your signal requirements. I've never seen a company like bump it to like five or six on hacker one but that'd be kind of interesting to see what happens. You could allow trusted or verified people only. So you kind of like, you know, disallow people who are not like verified in the system.

[00:42:50]>> Um, and actually I just saw James Kettle is talking about our episode. He said he really liked it, the most recent episode um, in Discord. And he said that what they've done for the Porsche program is they've left highs and criticals at the same payouts, but they've reduced the bounties on their lows and mediums.

[00:43:04]>> And we're seeing that a lot actually.

[00:43:06]>> I think that's kind of reasonable. Um, just because like those are going to be harder to get to these days. Um, some of the out output that I've learned from the Hacker Advisor Board meetings is that there's kind of like an exponential curve of like unfixed bugs. It's not fully exponential yet, but it looks like an exponential curve with the number of like outstanding reports across all across all um programs.

[00:43:28]>> And so what that tells me is they're having a hard time fixing it as fast as the bugs are coming in, right? And I think we knew that would be the case this year as everyone's scaling up their hacks. like as all of this low and medium hanging fruit gets found, the developers of these companies can't keep up. And so I think that one way you can, you know, kind of cut down on the lows and mediums and also save on your security budget because you might be like be literally running out and like struggling to get back more um finances to cover those is to either, you know, stop paying lows and mediums, which I don't love, or reduce the the payouts on the lows and mediums. But I think Justin, you would definitely say try to protect your highs and crits because one, that's where the real impact is and two that like if you if you reduce those, one, you might not actually get hackers looking at your program anymore because right now what I'm deciding to point my hackbot at is really at like what is the highest payout I can get?

[00:44:16]Like what is the critical payouts? So you're not going to get, you know, as protected, but also it's just like what matters to us. So >> yeah. Yeah. I certainly think that we are in a different environment than we were in when I said, you know, praising the low and mediums, right? I, you know, you're worried, I'm worried for the whole ecosystem in general.

[00:44:38]>> Yeah.

[00:44:39]>> Um, and I think that if they have to, you know, make some changes for lows and mediums, I understand that.

[00:44:47]>> I'm finding a lot more highs and crits than I used to because of the hack bots, right?

[00:44:51]>> Um, >> so I'll I'll say that. Would I like to see it? Absolutely not. I would love to not see it, you know. Uh but because the Abbots are also freaking good at finding mediums, you know, lows and mediums.

[00:45:03]>> Um I like your signal idea. I'm looking at the leaderboard right now. There are a lot of people in the top 100 that have, you know, below five signal.

[00:45:13]>> Really?

[00:45:14]>> I was kind of thinking five would be the cut off. The top 30 pretty much all have above um six signal. I think ironically today is new I think is at like 5.393 or something like that. Um, but you know, you could make your cut off at like five, >> right?

[00:45:28]>> You know, or something like that and that might help. That will cut out some of the top hackers, which is crazy. Um, but that's a part of the game, I guess.

[00:45:37]>> Yeah.

[00:45:37]>> Um, the one that I like the most though is uh requiring video PSC >> and uh or paying for submission.

[00:45:48]>> Yeah. Yeah. Yeah.

[00:45:49]>> Those are the ones that I like. I think that those are great ideas and that would really boost the ecosystem a lot.

[00:45:55]>> They implemented that on hack and proof.

[00:45:56]I think it's just unlikely with the speed and pace of development at Hacker One and Bug Crowd that that would actually come out. Um I think it would be more likely they could convert like your signal or rep into like points you can spend and then if you run out of those >> um you get it. That's a good idea. The the other idea I had, Justin, I think you might really like this one because it doesn't penalize people like you and I, >> is that um you could do a a a system where you get a bounty reducer percentage based on the amount of slop you have or or or maybe you even get an increase if you if you never push any slop. So like if you start getting NAS, you get like a 10% uh bounty reduction, then a 20%, then a 30%, right? And so it doesn't ever leave us in this situation where a person finds a bug and they can't submit it because I think that that often leads to like bad public disclosures because they get mad and they're posting on Twitter and it's like because their signal's too low, right?

[00:46:47]It's like what am I supposed to do? How am I supposed to report it? And like you end up in these situations where it's that. But I think that if if it's like no, you can still submit it, but you're going to get a 50% bounty because you've been like wasting so much of our time.

[00:46:57]It kind of makes the balance between time and effort for them to triage more reasonable.

[00:47:02]>> That doesn't really decrease the triage burden though. Like I mean it will over the long term but it doesn't stop the bleeding right now. I feel like we're in a tourniquet situation here really where we need to like stop the bleeding right now like you said right and that's why I think hacken proofs dollar to submit a bug no one gives a [ __ ] about a dollar right you know if you're submitting a bug right if you are submitting a real bug that has the potential to be paid thousands of dollars you don't give a [ __ ] about a dollar >> well they did notice that $2 one and $2 uh penalties don't reduce slop but when you get to five it does reduce it. Yeah.

[00:47:32]So there was there was cut off points.

[00:47:34]They did at $2 and it it didn't stop slop at all. At at five it stopped like 80% and by like $8 or $10 it stopped like 100%.

[00:47:42]>> So they should just do $5 is my point.

[00:47:44]Or >> refunded, right?

[00:47:45]>> Yeah. And it's refunded if it's a valid bug. Yeah. But then also you could even scale it based on region of the world.

[00:47:50]And then of course there's like VPN issues and all that. But I do think that like you know $5 for a new hacker they might struggle or not really want to or something. But yeah, it's still five bucks. Yeah, I think that I think that we would maybe see grants or something like that in that situation where um you know if you get a vouch or something like that from somebody in the system.

[00:48:11]>> Yeah.

[00:48:11]>> That you are going to not submit [ __ ] >> then you know you get a voucher for like 50 bucks in free submissions or something like that. I think that'll be pretty interesting. I like it.

[00:48:20]>> I like it. It doesn't it doesn't affect it doesn't affect the real players. It will affect new new players.

[00:48:28]But good luck new players moving into the system. I mean, there really is a lot of benefits to you right now with Claude and and you can, you know, get a lot of things explained to you. Um, but there's also a lot of competition. So, it's it's an interesting interesting time.

[00:48:42]>> Um, all right. Am I up next?

[00:48:44]>> Well, actually, just one more thing on that. I think it's actually a really uh key insight. I I work with a third world hacker as like somebody that I sometimes collaborate with >> and um he will we we've made like you know tens of thousands of dollars together over over just this just this year >> and and um the way that he brings a lot of value to me and that we end up collaborating so um in such a beneficial way is because he'll be like, "Hey, I'm in this program. I'm looking at this thing. It's kind of interesting. will you put your cloud on it? And I it's basically no time to me. I'm like, "Yeah, sure. Here, cloud, take a look at this." And then it finds something and then we go back and we find stuff and and he has cloud code, too. But I think that just like the my system or my setup for some reason will find stuff that his doesn't. And so if you are if you are a new person, I think reaching out to like some top hackers and being like, "Hey, you know, I've been looking at this thing." Because even top hackers with a lot of hack bots that are scaling, we still can't look at everything all the time. So, if you find something interesting, a way that you can make sure you get a good uh or like you you make sure you're not leaving stuff hanging is by reaching out to some top hunters that you know and being like, "Hey, I found this lead. Do you mind to like take a look?"

[00:49:53]>> Yeah, that's that makes sense. Um I don't know that I can deal with any more influx of that. Uh maybe send it send it to Joseph.

[00:50:01]>> I don't know. I'm pretty busy, too. But yeah, there's there's plenty of people in the in the critical thinkers chat that are like very highly talented. Just look for a collaborator. Yeah, >> absolutely. Um, all right. Speaking of, uh, you know, collaborators of yours, um, we have, uh, an article by XSS doctor, um, with XSS doc and Monke's new startup, Star Strike. Um, they are, they released an article called achieving deterministic prompt injection through client side feedback loops. And I wanted to run you guys through this really quick because I think this shows the doc's mastery of client side concepts.

[00:50:37]um and really beautiful weaponization of them. Okay. Yeah. So the TLDDR of this article is that there is a um Q parameter on a chatbot which allows you to do prompt injection >> 20 30 50% of the time that will result in XSS. That is not great for triage because what's going to happen is they're going to click it and it's not going to work and then they're going to NMI you and it's going to waste your life.

[00:51:02]>> Yep.

[00:51:02]>> And it's not a a great exploit, right?

[00:51:04]you know, it it sometimes uh people will decrease the bounty for nondeterministic AI exploits.

[00:51:10]>> So, what they cover in this um in this article is how to make it more deterministic.

[00:51:17]Um there's also a post message based race condition in here which utilizes the technique that we discussed on uh the um hackalong that we did on Adobe not too long ago in the critical thinking discord where you pop up a small window and keep that in the front of the victim window uh to make sure that you're that both of the the victim window and the attacker window are in focus or in view which limits the uh rate limiting on post messages. is being sent across origin. So that's a a good technique uh that I wanted to pass off to you guys. Uh but the whole setup here is um Doc pops open a new window. This is the attacker's window. And in the background, he multiple times sends the prompt injection. Sends it once, waits 10 seconds. If he does not get a call back from the XSS firing, he will reinject the prompt again and then wait another 10 seconds. Doesn't get a call back, send it again. Right? And so 30 seconds of sitting on the page, you have, you know, a massive chance of getting this exploit to actually work.

[00:52:21]Okay.

[00:52:21]>> So cool.

[00:52:22]>> Um, >> very cool client side feedback loops.

[00:52:25]Really good idea. Great way to increase the um, integrity of your exploits, whether it be uh, prompt injection related stuff or race conditions or both like this situation is. Um, I I did want to mention one little bit here uh, a modification that I had to this article.

[00:52:41]Um he solves a problem here and maybe he has a good reason for doing this. I don't know. But he solves a problem here by uh of the victim iframe that is receiving the race condition. Uh he solves that not having a reference to the attacker's window by making the parent the opener of the attacker's window as well. So it's a mutual opener situation. So the attacker's window is an opener of the victim window and the victim window is an opener of the attacker's window. Um this is possible.

[00:53:11]You can read it how to how to do it in the article. Um he says the reason for that is that the victim had no uh window reference to the attacker page. Um however the attacker had a reference to the victim page. Um and my solution to this would have been a little bit different. My solution would have been to just use the post message event source. So whenever you send a a a post message to an iframe um there will be an event object that gets passed into the post message handler and that event object has a source uh attribute. That source is a window reference to the frame that created the post message that is being sent.

[00:53:51]>> So I would have >> So that statement is actually wrong.

[00:53:53]It's not that there's no reference. It's just that well at this current time it doesn't have a reference but as soon as you send a post message in in trigger the XSS if you send another post message then uh you can register an event handler to snag that post message and then uh log the event source and then shoot back out to the attackers page. Um that's how I would have solved the problem. Uh just wanted to show that out there. But I think this trick of having mutual um opener >> opener >> uh is also a really cool trick. uh for you guys to know as well and it's really beautifully displayed here on their blog with um you know a graph of this whole attack. So great work by um XSS doc and Mon at Starlight or Starstrike.ai.

[00:54:37]>> Yeah. Yeah. Good work.

[00:54:39]>> All right, we are low getting lower on time but we're still doing fine. I just wanted to mention to everyone u GPT5 or sorry 55 GPT5. So, I've got two uh links I'm going to show. Um >> well, actually, one, can we just >> tweet about SLGO, bro? Is that as OP as it sounds?

[00:55:00]>> It's so OP. It's the exact solution that you wanted to run overnight.

[00:55:05]>> You just give it a goal of like find five crits and then even if it finds two crits, it'll just keep going until like it literally will just work until it reaches its success condition. And it's so good. Like to me, it almost I hate this.

[00:55:17]>> How does this not exist in cloud code, dude?

[00:55:19]>> Great question. But it it uh to me it's like the final cheat code that makes this the barrier to entry to this basically zero. So let me tell you why I was finally convinced to buy a codeex uh sub and let me tell you how it went. So I'm going to share my screen on this LinkedIn post.

[00:55:38]>> There's one of these that I agree with.

[00:55:40]>> DMs first.

[00:55:41]>> Yeah.

[00:55:42]>> Uh so actually that's true. I always do that on X. I always show my DMs and they have to scrub it. So, this is a very confusing chart and I also think it may have been cherrypicked, but it still was enough to convince me. So, this says GPT55 delivers the best performance we've seen to date. For listeners, I'm sorry, this is confusing, but there's basically two lines. There's a white line, which is white box, and it compares GP5, Gemini 3, Opus 45, 52, and Opus 46. I don't know why they didn't keep the white line going for the latest models, but anyways, because like we have no we have no idea on this white line how Opus 47 or GPT55 do. But the point is that when it comes to white box testing, the cloud models are better.

[00:56:26]You can just tell by Opus 46 having a huge increase here. Now, I will say the Y-ax on this is really dumb. It's vulnerabilities found before first miss.

[00:56:33]What do they mean by miss? What do they mean by miss? Does that mean like before it didn't find a bug or or that means it submitted a false positive? So they picked a really obnoxious y-axis. But uh anyways, this blackbox line is what convinced me. So they have a dotted blackbox line. So this is like, you know, again, it's just like the tracking of the vulnerabilities found before miss across Opus 45 all the way through Opus 47 and then GPT5. And when you look here at Opus uh 46, it's like a And also, why is this not on the like how can you have how can you have less than one vulnerability? Is this like a 4.9?

[00:57:09]>> No, no, that's five, bro. That's five.

[00:57:11]>> Yeah, it Well, they missed the mark.

[00:57:13]Anyways, my gosh, my whole point is Opus46 is much lower than GPT55. And so, and then they they posted this other thing which I'm not uh >> Well, okay. Okay, let me let me let me comment on that really quick. Can you pull it back up for a sec? So essentially what we're seeing here is Opus 4.6 is is finding um like four vulnerabilities before first miss on average and with GPT 5.5 this is in a blackbox context it jumps up to like eight or nine.

[00:57:38]>> Yeah.

[00:57:39]>> Uh so it almost doubles right which is insane.

[00:57:43]>> Yeah. And and they obviously you know are people who I relatively well trust a little bit on like how to like like Expo should be able to evaluate these models, right? They've been doing this for like three years at this point. they should be able to evaluate these models and they and they did more than just this.

[00:57:57]So I'm going to read this other post which I again and this one I'm even more skeptical of but um we'll read it anyways. Albert Ziggler, the head of AI at Expo says GPT55 without access to source code is a better hacker than most previous models with source code. Okay, he says most here. I thought he said than open 6, which is what I just straight up didn't believe, but I think this is definitely true. I I think that statement is definitely true. If you assume he's talking about like GPT54 or GPT53 or or other companies, I don't think it is better than 46 or 47 with code like with white source white box code, >> but in general, so anyways, this this convinced me to try it, Justin. Within the first 30 minutes, it found three P1s of me running GPT. No, I'm not kidding at all. Now, I will say it it did have a little bit of advantage. It was a fresh invite program on Buck. like it was like a new program that just dropped. So, you know, take take that with a grain of salt, but still that was faster and more efficient than what I had seen for Opus.

[00:58:55]Now, for anyone wanting to try it, here's all I did. I sim linked my cloud MD to agent MD from Claude to do.codex folder. Then, I just had Claude Code sim link all of the folders in the in the skills folder to the codec skills folder and then just ran it. I got and I and I am cyber approved or whatever, but I got no rejections. I I used SlashGold to run this overnight. It found like lots of other bugs. My my intuition is that it's like 10 or 20% better than 47 and 46 um at blackbox testing. And Justin, I like didn't run out of tokens. I like I had one/goal that ran for 14 hours. And then when I checked, I'd only used like like 15% of my weekly usage stats. Um, meaning that I could obvious that that was like 130th of the monthly stats for the $200 a month. Anyways, it's wildly effective. I think people should basically be building their systems to be usable by both cloud code and codeex.

[00:59:49]And I think that, you know, in the past I was like, "No, cloud code, cloud code, cloud code." Now, I think it's totally reasonable to use either system and you'll be highly successful.

[00:59:57]>> Dang, dude. That's crazy. That's crazy.

[01:00:00]I I I will say also, you know, with just 4.6, I also had a target on Bug Crowd this past week that uh is is a fresh fresh program. So, I wonder if I wonder if we're going to dupe. Uh hopefully not.

[01:00:12]>> No, it's not. Okay.

[01:00:13]>> Okay.

[01:00:14]>> Yeah, you can bleep that, Richard. Yes, please.

[01:00:16]>> Um Yeah. Okay. So, we're good. But, uh I literally I was sitting here with my friends, right? And we're like, let's just kick this off. Let's see how it goes. I kid you not, within 15 minutes, it had an a like a JWT takeover. Yeah.

[01:00:30]JWT forging bypass. Bro, >> you uh you know, I need to actually I need to audit your um JWT skill if you don't mind.

[01:00:36]>> Yeah. Yeah. I got you. Um, so >> I was I was saying that as a joke cuz I want to steal it.

[01:00:41]>> Oh, well, dude, you shared me so much. I I DM you all the time. I'm like, tell me exactly how you do this.

[01:00:46]>> Oh, you did do that yesterday. Actually, when you did that yesterday, I was like, he's really sussing me out here.

[01:00:50]>> I'm sussing you out. Well, I'm also taking a different approach, though, so we can compare and contrast. So, um, the question I asked you yesterday, I did it different than you and JD. So, um, we'll we'll we'll compare notes on that a different time. Um, all right. I've got one last, uh, write up. This one is super sick. Uh, save the best for last in my opinion here. Um, freaking love, dude. He's such a good hacker. Uh, and anytime I read anything by him, uh, I am floored. So this is uh >> actually wait just for our listeners really quickly if you don't know Rio talk obviously we've mentioned him a ton in the podcast but this guy was like basically fresh in the bug bounty scene >> and he came to Defcon and did like you know H1 405 wait H147 >> H172 >> 702 H172 and just won it got MVH >> at like 18 >> yeah as like an 18-year-old. So anyways, statistical skill level here.

[01:01:46]>> Anyway, absolute beast. Um, I've mentioned on the pod, I think he is the closest thing that we have to AI, >> you know, like like just just seeing him consume minified JavaScript code like a book.

[01:02:03]>> Like just I I've never seen anything like it at any before to be perfectly honest. Um anyway, so without hyping him up too much, flat uh you know, really flat security in Japan really snagged a good one when they got here. And this write up is on their blog. Um and it is remote code execution on Google Cloud with a single directory deletion. Um and this is on Google Cloud's Looker. Um so Looker is a business intelligence platform. Um I've actually seen other bugs in this, so probably a good place to hunt. um he is able to get a self-hosted version so he can reverse engineer it and he's looking at how all of this uh stuff works here in Ruby. Um, and the the TLDDR of the situation is he finds a way to um delete an arbitrary directory from within his own repository uh via a um you know confusion in the validate path name function uh for deleting the the directory.

[01:03:10]>> Yeah.

[01:03:10]>> And he's able to use that to delete the.getit git repo for uh one of his projects that he's uploaded.

[01:03:19]>> Yeah.

[01:03:20]>> And then he says something that just freaking blew my mind and that everybody here should be paying very very very close attention to. Okay. Since it is possible to trick Git into using forged git configurations if the doggget directory is corrupt or deleted, the validate named dur method checks the directory to be deleted, includes.git, and raises an error if it does. Okay, so you're not supposed to be able to delete the.git directory. Interesting.

[01:03:46]>> And then he just he just kind of slips it in there since it is possible to trick Git into forging as everybody knows it. It's possible to trick Git into using forge git configurations if the.git directory is corrupted or deleted blahy blah blah blah. I went back and I was like, wait, what? So if I can delete the.git repo from my my or the.git folder from my repository folder.

[01:04:09]>> Yeah. and you run command inside that folder, >> you can get RC. This is essentially what this reduces to.

[01:04:17]>> Wow. Do you know do you know how many >> coding sand like AI coding sandbox features we've tested that that might be relevant to?

[01:04:24]>> Yeah, dude. A ton. And and so essentially he explains the reason for this. Um, if the git directory is deleted, the next git command executed against this repository will fail to find the git directory and it will look for git configurations in the work tree directory instead.

[01:04:40]>> Wow.

[01:04:40]>> Therefore, if the work tree contains the files that resemble the contents of a doggget, git assumes that you are in let me where is >> are you kidding?

[01:04:49]>> Yeah, git assumes that you are in thegit directory and we'll just load these files straight from the the root of the work tree. So, let me just explain how this work.

[01:04:59]>> So, what's the what's the folder you want to put it in? Tell me. I need I need the juice right now, Justin. So, >> yeah, dude. I mean, look at this graph that's on the screen.

[01:05:05]>> Yeah.

[01:05:05]>> Look at this on the screen.

[01:05:06]>> You can put it in any of those. Oh, you just put it You just put it straight in it. It thinks it's already inside.Get, >> right? So, literally, it thinks it's already inside. So, they're running a git command from the the root directory of the repository. Typically, it says, "Okay, let me look at the.git, you know, folder. You know, if it's not there, oh [ __ ] I guess we're inside the repo already." So it just read tries to read config right and then treats it as.git/config. So this is the whole situation right you guys can't we can't upload you know malicious.git folders to git right that's the you know that's where a lot of security in all of these systems relies on thegit repo is sacred right and if I can check that to make sure there aren't hooks or whatever you know we're all good right but then if you can delete that.getit git repo.

[01:05:53]It'll just treat the config file that's in the root of your repo as the git config and you can register hooks and then you're set.

[01:05:59]>> Dude, we've got a week. Can we go look for this in and >> uh as soon as this calls out.

[01:06:07]>> We're late, bro. We're late. I think this this thing shipped in March.

[01:06:12]>> Yeah, March.

[01:06:12]>> Yeah, but I mean the number of people that saw this and then drew the same conclusion to us is probably pretty low.

[01:06:16]I mean, people just read their eyes glaze over. This is good stuff. Anyway, >> honestly, is this vulnerability in git?

[01:06:23]>> No, this is just >> Are you sure?

[01:06:25]>> No. No. There there's some code branch inside git, inside git code that is saying if.git is not there, look in the current directory. And I think that that is probably a vulnerability. I think it should always make sure it's in agit.

[01:06:37]>> I don't know, man. Maybe I'm misunderstanding it, but uh that seems like the way that git is just going to work. I can definitely see the GitHub team being like, "Nah, bro." Like, this is how it works, you know? So anyway, then it's trivial. Okay. No, I'm sorry.

[01:06:52]It's not trivial from there. Let me let me let me also give another shout out to Genius. So now he's got the config file in there, right? Um and and so let me explain.

[01:07:05]He the way he got the the directory to delete was actually not deleting the git directory at all. He deleted the entire repository directory, right? The entire one, including his own config files, right? But the way that the delete recursive works in Ruby, you know, it will recursively go in and delete all the files.

[01:07:24]>> Yeah.

[01:07:25]>> So he he created a folder that's massive that has a ton of files in it, right?

[01:07:29]That's going to take the delete after the.getit and before his config, right?

[01:07:36]So, so what'll happen is this. He'll say, "Delete the whole repository folder." And he'll say, "Okay, great.

[01:07:43]Going to recursively delete." It goes in there, deletes the dotgit, right? So now we're in a vulnerable state.

[01:07:48]>> Then it finds massive folder, you know, and then 1500 million, you know, files, uh, you know, nested within that. It goes deep. It goes deep. It goes deep.

[01:07:58]It goes deep. It's deleting files. It's deleting files. It's taking forever. In the meantime, Riota is hitting another endpoint that runs git status on that same folder in the middle of deletion.

[01:08:08]>> Oh gosh. triggers the config for um you know the >> he's literally an AI Justin >> dude triggers the FS monitor hook and then runs arbitrary commands on the the Google prod >> he's a robot >> server >> freaking beautiful man I love it and he explains the nuance of like the XF4 file system and how um the Ruby file utils rm-rf uh you know recursively deletes stuff it it was beautifully done but the concept is that and it was just chef's kiss, dude.

[01:08:43]>> Yeah, that's insane.

[01:08:44]>> Yeah. And then he explains even the full privilege escalation um inside of Google Kubernetes cluster as well um because he's able to check the service account in /wiron/secs kubernetes ioservice account and found some excessive permissions that they're able to update secrets for all the other Kubernetes clusters as well which results in full privilege escalation. So >> wow.

[01:09:06]>> Yeah, >> that's you're right. You did save the best for last.

[01:09:09]>> That was a freaking beautiful bug, right? The race condition, the get like confusion.

[01:09:16]>> Yeah, dude. It's just beautiful.

[01:09:18]>> Yeah.

[01:09:19]>> Oh, man.

[01:09:20]>> Yeah. Shout out to Tuck and flat. Um, all right. You got anything else or is that a wrap? Right on time.

[01:09:25]>> Let me check the notes. No, it looks good. I'm done. Yep.

[01:09:28]>> All right, dude. That's the pod. Peace.

[01:09:30]>> Thanks.

[01:09:32]>> And that's a wrap on this episode of Critical Thinking. Thanks so much for watching to the end, y'all. If you want more critical thinking content, uh, or if you want to support the show, head over to ctbb.show/isord.

[01:09:42]You can hop in the community. There's lots of great highlevel hacking discussion happening there on top of master classes, hackalongs, exclusive content, and a full-time hunters guild.

[01:09:51]If you're a full-time hunter, it's a great time. Trust me. All right, I'll see you there.