The GitHub situation just got worse...

Added: 2026-05-12

[00:00:00]If you live in the world, you've probably heard about GitHub. And you've probably heard that GitHub is having a really rough month. Starting with pull requests that when pulled in would undo previous pull requests, causing entire merges to get reverted and engineers to get yelled at, as well as major projects by large developers like Mitchell Hashimoto's Ghosty, removing them from GitHub, saying that GitHub is no longer a serious place for serious work. Now GitHub has responded trying to provide a solution to the problem saying sorry and quoting a graph where green line go up.

[00:00:32]Now I do want to highlight the problem that GitHub has to solve is actually very difficult and while this is fairly comical this is a problem that I personally could not solve. So props to the GitHub team for trying to solve this. This is mostly in justest. But amidst all the chaos, I I saw all these articles about the GitHub saying sorry, the pull request migrations. And then in the same Twitter thread, I saw, oh, by the way, Whiz Research hacked github.com. And it only took a single push, a single push to GitHub, compromised GitHub. Let's talk about how they did it. It's actually it's beautiful. It's it's it's fantisimo.

[00:01:06]Now, I am going to riff on an article from whiz.io. So, go check out their article for more details. They did a great job here. And also they have collaborated with GitHub so this bug is fixed but it's really neat what they found out to understand how this works we have to go into kind of the architecture of GitHub right what happens under the hood when you make a git push so on the left you have your user pushing data to GitHub over SSH very common use and over there and when you land in GitHub you land in this binary called Babbled babbled I think babbled is actually going to be the babbled for routing where you use a distance vector routing protocol to figure out what node to go to I'm assuming kind of like a load balancer I'm not sure if that's what it's used for But at the end of the day, you land in Babbleda. When talking to Babbled, you're able to use push options. Push options are a part of the normal git spec where you can literally just include strings that get sent to the git server. Maybe you have a custom application that catches the commit.

[00:01:54]Maybe you have a pre-commit hook that runs that ultimately depends on some custom data. You can set up your environment to push certain data to the server that you may need to customize.

[00:02:02]Now, under the hood, what Babeld is doing is it's taking all these push options and converting them into a single HTTP header. This header is an Xstat header which is going to contain all the data inside of all of your push options. This is likely because the protocol that carries the push options is some binary protocol over SSH and we have to talk between Babeld and Git RPCD over some other protocol maybe HTTP. So to pass that same data all the push options get put into this Xstat header.

[00:02:29]Not a big deal. Ah but enter a 30-year-old problem. Input sanitization.

[00:02:36]How do we format HTTP headers? We format HTTP headers via the RFC that says you have the header name, a colon, a space, and then the contents of that header ended by a semicolon. But what happens if Babelda doesn't sanitize semicolons coming in from the user on a git push?

[00:02:54]Well, what you can do is insert into the Xstat header a series of push options that contain semicolons. And now contents that's supposed to arrive in the Xstat header, an area that contains untrusted user data, now can appear outside of the Xstat header, allowing it to look like it comes authoritatively from the Babeld server. The example here is normally what you're not allowed to do is commit large blobs. This is rejected and that rejection is enabled as a true boolean. But by doing a push option injection that contains a semicolon, they can include the data large blob rejection enabled equals false. And because it is the last header that is written, it will overwrite the previous entry of the option. By overwriting these headers, they now have arbitrary header injection from a privilege standpoint. Meaning Babe will push the data to the RPC server and Xstat will only contain data that's supposed to be evil, that's untrusted.

[00:03:46]But the data on the outside is also evil, but get RPC doesn't know. And guys, real quick before we keep going, I want to talk about today's video sponsor, Code Rabbit. I know there's a lot of AI hype out there, okay? With AI coming online every day, there are millions and millions of lines of code generated just by AI models. One really powerful thing that I think AI can do for us is automatically review the code that's generated to look for security vulnerabilities. Code Rabbit is an AI code review tool that will look at your poll requests, reason about your code, and then make actionable recommendations about what you can do to fix issues with your code. All I have to do is provide an access token for a GitLab user that has access scoped to the projects I want it to review. And then I can go into Code Rabbit and set up a personality or a profile for how I want it to do these reviews. Here I said I want it to create concise release notes as a bullet list.

[00:04:30]Your main goal is to find security vulnerabilities on the back end of a server. I'm telling the code reviewer specifically what I want it to review.

[00:04:36]All I have to do is actually submit the merge request and then Code Rabbit kicks off actively running through the code, reading all the commits and seeing exactly what changes in the PR need to be reviewed. Code Rabbit identified some serious security vulnerabilities. Looks like my intern added a new endpoint where there are security vulnerabilities. Who wrote this feature?

[00:04:53]Oh, and if you don't want to wait until a poll request to do your code review, Code Rabbit CLI tool has you covered.

[00:04:58]Code rabbit CLI is a code review tool that uses Code Rabbit rabbit, integrates in all of your TuE apps like Claude, Codeex, and Gemini. With Code Rabbit CLI, you can vibe code with confidence before the slop gets out the door. The best way to help the channel out is to go try the sponsor out. Go give Code Rabbit a shot on your repos. You can use Code Rabbit for free at the URL below.

[00:05:14]Code Rabbit, thank you for sponsoring the video. Let's get back to it. Okay, so they enabled large blobs. Who cares?

[00:05:19]Well, there are other kinds of headers that can be enabled that have really weird side effects inside of the Git environment. For example, the weird world of hooks, pre-commit hooks and postcommit hooks. Now, all this testing was occurring on GitHub Enterprise Server, a local version of GitHub that you can run to do testing or to emulate the GitHub environment. This will matter here in a second. GitHub Enterprise Server supports admined custom pre-receive hooks, scripts that run before a push is accepted. Again, normal GitHub behavior on its own if it didn't have compromised headers. By reverse engineering the pre-receive binary, we discovered it has two execution paths.

[00:05:53]One controlled entirely by the Rails environment field from the Xstat header, which again, by the way, they now control both of these by injecting a nonproduction Rails environment value through the Xstat header semicolon injection technique and injecting a custom hooks directory to tell the Rails environment where to look. And also by injecting a crafted hook entry that has a path traversal, they're able to use the concatenation of those two paths inside of the Rails environment to run an arbitrary binary on the system. The Rails environment is a sandbox escape.

[00:06:23]The custom hook directory tells them where to start looking for binaries and the reproce receive hooks is a path traversal that walks to the top of the active server, giving them the ability to run any binary, ultimately allowing them to run ID as git rce through a git push. Beautiful. Glorious. Now again they started this on a local version of GitHub enterprise server. They wanted to hack GitHub. They tried to push the same exploit chain to the server. Nothing worked. After some additional re they had to find a flag that tells the server to run in enterprise mode. Effectively emulating GitHub enterprise server. Now normally there's a flag set inside of the Xstat header that prevents the user from being able to be in enterprise mode. Completely removing the ability to use these custom hooks, right? These hooks that give them the control over the server. But again this is just a flag in the Xstat header. And because of this, they're able to override that flag and give them arbitrary command execution within GitHub. Instead of running the ID program from before, they run host name. And from the output of this, they can see that they're getting command execution in their git push.

[00:07:19]Now, you may be wondering, who cares?

[00:07:21]Who cares if they get execution as git?

[00:07:22]Git's just a no permission user on the server, right? What they actually found is because of the way that git is set up, the user is able to access not just your repos on the server, but also every other repo on that server. So because of that, if you have maybe a private repo that lives in a certain git server, that git user that they landed as was able to read all of the storage nodes on that device, completely bypassing the ability for someone to mark the repo as private and all that nonsense because Babe didn't sanitize out their semicolons, allowing any user to inject a field into the Xstat header, allowing them to overwrite privileged headers. What a world. What a world. Now, props to GitHub. Okay, GitHub did fix this vulnerability, I think, in 2 hours after Whiz reported it. Whiz research reached out on 34 March 4th and the rce was confirmed by them and acknowledged and fixed within the same day which is that's actually really great from a security standpoint. Love that. Now I know what you're thinking. Would Rust have fixed this? Would Rust the world's most secure programming language with no vulnerabilities ever have fixed this bug?

[00:08:20]No, not at all. This is this has nothing to do with memory corruption. This is not a race condition. This has nothing to do with the problems that Rust claims to fix. This is literally just because there is no sanitization in the way that this is parsed. So babe, babbled babbled d pushes headers into another header, but because of the way that they're delimited, you're able to arbitrarily put data into that same location and it will appear as authoritative data to the person who gets it down the line. And then on top of that, we have these privilege fields that can be taken advantage of because the data that's received into them is thought to be authoritative. It's ought to be done by the server, not by the user. classic case of assumptions separately are great on their own, but when combined and exploited by one minor bug, the whole system comes crashing down. But again, fixed within the same day, which is a great a great thing for the world.

[00:09:05]Anyway, guys, that's it for now. Thanks for watching. I appreciate it. If you like this kind of stuff, do me a favor.

[00:09:09]Let me know in the comments below. Hit subscribe and then go check out this other video about a similar bug that I think you will also enjoy. We'll see you over there. Little kiss on the cheek.

[00:09:17]No. Okay. All right. Chew it.