Ghosty Leaves Github

本站添加: 2026-05-11

[00:00:00]Let's talk a little bit about GitHub.

[00:00:01]This is Mario Rodriguez and he is the chief product officer of GitHub. By the way, for those that don't know, there is no CEO of GitHub. As far as I can tell, there are two VPs of core AI in which are a charge of GitHub as CEOs are acting as somewhat of a CEO. So, GitHub does not in fact exist with someone at the helm, but instead somebody who's high up in an adjacent organization saying this is GitHub's goals effectively. Nobody's actually steering the ship for the sake of the ship itself. They're steering the ship for the sake of say core AI for what what AI is going to be doing. Does that make sense? But, so Mario Rodriguez tweets out this thing right here and they they look at this blog which we will look at here in a second. I actually haven't looked at the core reason why the incident the big incident happened.

[00:00:51]We'll go over in a second. Um, but we're going to I have a YouTube video on the basic big incident. Uh, so I say this, how can we trust what you say? You have no Listen, I have no idea as your tenure CPO, right? Uh, cuz who knows, maybe the guy's brand new, whatever it is, but apparently he's been there for a year and a half. But this was his response, right? And I said, uh, basically about your transparency, how is April 23rd, uh, PR history altering issue not represented here, right? Like why is it not on the graph? How can we see nothing on the graph? Yet operations were a 100%. He just responds with the incident involved a bug that affected poll request merge through merge Q. We did status poll request for that incident.

[00:01:25]The regression was introduced. None of it is point none of it actually answered the question like is this guy. So then I helped him. Here's poll requests on the 23rd, right? Like it's crazy, right?

[00:01:35]They did they're not even they're not even marketing on their own status page and then this answer is just like what I would assume an LLM makes. It's the most LLM ass AI response of all time. So there's this. But let's let's keep on going. By the way, they did actually ended up they actually ended up after I pointed it out saying, "Hey, this is ridiculous." They actually ended up doing uh this thing right here. So, they did put it in as at least something, which is good cuz they should have cuz it's very unfair for them not to have something in there. So, good on GitHub after I somewhat ruthlessly pulled it out. Let's actually look at what the um what the stat is. I I want to actually look at the recent incidents. Okay, there we go. So, let's look at this one.

[00:02:10]This one's apparently the postmortem.

[00:02:12]Let's read through the postmortem. Let's see how we feel. On April 23 uh 3rd, pull requests experienced a regression affecting merge Q operations. Pull requests merged through merge Q using the squash merge method produced incorrect merge commits when a merge group contained more than one pull request. In affected cases, changes from previous pull requests and prior commits were inadvertently reverted by subsequent merges. So it does use the term reverted which I believe is incorrect. Did anybody here experience that issue? Cuz there was like 200 and some 200 and some orgs. Yes, at work.

[00:02:44]Okay, that's crazy that there's two people claiming like it was like 2,000 PRs, but it was over 230 organizations.

[00:02:49]So, like the chances are pretty small. I mean, it's shocking that uh what's it called? That people are saying yes.

[00:02:54]Okay, so with you two who are saying yes, you experienced this at work. Was it a revert? Anyone here got the revert?

[00:03:00]Because for those that don't know, this is why I really hate the language that they're using because I I try not to be one of those really annoying people that are like, "Nah, actually you're you're being pedantic and you're and you're miss you're misspelling it. it actually, you know, like I don't want to be that guy. That guy kind of sucks. Uh but there is most certainly a value in this case to be specific because a revert. So let's just pretend you have three commits, right? And you wanted to revert uh one of these commits. Let's just say you wanted to revert the middle commit.

[00:03:27]And we're going to call this one uh A, B, and C. If you wanted to revert B, a get revert is kind of like uh antimatter B. it would actually have a, you know, it would actually have a notb commit that would be added to your history. So you can see that B was added here and not B was added here. Does that make sense? It's a new commit. So when someone says, oh, the commits were reverted to me that says, okay, new commits were added to the tree that reverted previous ones. And if that's the case, then it's actually pretty easy to go, oh, this is exactly what happened. So when they say, hey, it was reverted. To me, when I hear that, it means that there was an additional commit made that reverted whatever set of changes. So, that way, if I needed to inspect history, I should be able to go over history and see which commits were actually reverted and then be able to reapply those commits. Correct? But that's not what happened. So, when we say this, I what I want to be is I want to make sure that everyone understands what was happening with git. Let's pretend we had these three right here, A, B, and C, and you wanted to put on D.

[00:04:26]Wow. Uh, as far as we can tell, we're going to hopefully read this, maybe we'll get more information, but if you were to merge that in, you could potentially have a history that looks like this. A to D after Jesus, uh, and B and C would just simply be gone. They're not reverted. They're just not, right?

[00:04:43]Like, they're just they're just not not present. All right, this makes a big uh complicity. Bad. Yeah. Yeah. How could that happen? Uh, it's pretty easy. So, I actually make a guess in in in a video I released is that somewhere inside the pipe they're using force, which seems crazy to me that they're using force.

[00:04:58]They're not using force with lease.

[00:04:59]They're using force. Something says, at least because how could you take like let's pretend this is the actual current history of Maine, by the way. Look at that. Look at that line I drew. That was a good line. Um, all these are good lines. Look at all of them. They're great. But let's pretend you had this as your history. How could you change your history to this? You could only change it with a force push, right? Is there any other way you can change history that I I'm unsure of how you could change history. No, Squash doesn't change history of the branch it's on.

[00:05:26]Squash like or h how should I say this?

[00:05:29]Squash does you can squash commits, but when you're doing a merge Q, the merge Q actually creates a new branch, right?

[00:05:37]So, this is actually a whole new branch that needs to be merged onto your main line. But you can't merge, you can't just merged onto your main line because B and C are missing. You have diverged from history. So, these were they weren't squashed out either because a squash typically whenever you squash you keep all of your commits here. I can just do like a really quick obvious one, right? Uh let's go like this. Let's go.

[00:05:58]Vim right here. And I'm just going to go like this. Test. Uh hello. Uh go right here and do this. And then go like that.

[00:06:05]Yeah. Yeah. Yeah. Yeah. Yeah. Test. Uh one. There we go. Whoopsies. Yep. That's good enough. Right. Quit. And then go uh down. There we go. Yep. Yep. Yep. Yep.

[00:06:15]Yep. CC two. Awesome. Uh, there we go.

[00:06:19]And three. Right. And so if I were to go in here and I were to do a squash, I'd go get uh get rebase. I uh head ty 2, right? Oh, actually, let's do ty three.

[00:06:30]Uh, three. There we go. So, if I Let's see. Invalid upstream. Oh, is it?

[00:06:35]Okay. I know there's a way to do this here. Uh, it's two. You can't do the last one. So, okay. My bad. My bad. My bad. Here, let's just let's just get one more in here just to make it obvious.

[00:06:46]There we go. Beautiful. Get rebase I uh head till the3. There we go. So now I have these ones. So typically whenever you do a squash, you're doing something like this, right? You're taking those ones and you're squashing these two commits to make it into one singular commit. And so when I go like this, it says, "Hey, here's your commit message."

[00:07:05]I do that. Uh you can uh you can get uh ref log to see how we're now here. We're now we're now into this new one that we've created right here which is at head zero. Uh this is a combination of two two of the uh two of the commits right here. You can see that this was the commit four. This was commit three.

[00:07:21]This was commit two. You can kind of watch it happening right here. So you can just see it happening. So like that commit still exists on your system. It's just that when I go get log, you can see that it's only two. That's what a squash is for those that don't know what a squash is. It simply takes the commits and puts them together. for some people don't know that what what was going on here is that it was actually dropping commits like these things were dropped.

[00:07:42]So that's not a squash. If these two are missing afterwards, my only suspicion is that this thing this branch was force pushed, right? Or you were somehow stupid enough to create a uh to create a feature that actually uses um that actually uses uh what's it called? Drop.

[00:08:02]Right.

[00:08:04]So let's go like this. Actually, hold on. Let's go like that. Hold on. Uh CC, I'm just Oh, you can't do that. Yeah, you can't do it. Empty. Uh five. There we go. Hold on just one second. I can show you this. Uh let's go like this one. This should be dropped. All right, there we go. Hey, don't look at my elf.

[00:08:17]Don't look at my elf. Get out of here.

[00:08:19]Six. All right, stop looking at my elf.

[00:08:21]All right, do that one. CC. Uh seven.

[00:08:23]There we go. Bam.

[00:08:26]All right, get uh get rebase I uh head three. There we go. So now you can see them. So instead of doing that one, you can actually you can actually drop this.

[00:08:35]So if I put a D in instead and right quit, now I have a conflict. Right now you have to like merge all this crap together. Everything gets super super complicated. Like it just dropping is actually really really hard. So I don't think that they just simply dropped commits. I don't I don't think that that's what happened. My my my assumption is that they simply just they did exactly what this one article said, which is they actually took like let's pretend that this change with A and D is right here and you just branched off of A and you created D and then you made a poll request with this. This is what my assumption is that happened is that this is what they used and they literally just forced push that to uh to main to master. You know, I I'm going to I used main for my PC boys and then master for my based boys. Okay, got them. All right, let's see what happened here. All right. So, during the impact window, 230 repositories, 2000 uh 292 poll requests were affected. We initially shared a slightly higher numbers because the first assessment was in intentionally conservative. Well, if you look at the uh what's it called? Yes. Yes. I believe the first the first numbers we saw were 2804 by the COO of GitHub. What's his name? The COO GitHub. I forgot his name.

[00:09:41]There you go. Kyle. Kyle's a really nice guy. I mean, I I'm not I'm not here to to dunk on Kyle. Whatever. I'm not I'm not concerned. Anyway, so yeah, they they're now saying it's actually significantly lower, which is kind of crazy cuz people were claiming I mean there's three people apparently in this chat. Three people in this chat in Twitch chat, there's only 1,600 of you.

[00:09:58]Three people in this chat apparently have had this happen to them. Like, is that true? Are these numbers right? Is it naughty of me to believe that these numbers aren't just me and my alt accounts? Nobody can verify. Yeah, because Acorn, you said you had it.

[00:10:10]can't believe I I just feel like I feel like so one of the problems I see also is that there's a bunch of AI bros out there who will simply blame Claude, right? They're just going to be like, "Oh, Claude messed up again. Just reverted. Claude just messed up my behavior, not having any idea that uh it actually was something else." It's kind of funny. All right, I want to see the actual thing. Really? They didn't even give us a postmortem. Okay, so I'm a little bit I'm a little bit annoyed by that. I was hoping that they released a second article saying stuff happened and they really want you to know that they were actually experiencing a whole bunch of uh new traffic. So therefore uh that's actually the real problem, right?

[00:10:43]Like that's that's honestly how I read this is like, "Hey guys, it's actually really hard. We're experiencing a ton of traffic." Which by the way, I totally get that. But by 2023, by the end of 2023, you should have said, "Hey, something really big is happening." By the end of 2024, you should have been fully prepared for the fact that you're about to increase massively. like none of this. I mean, I feel like as a coach, like if imagine if you had a CEO, maybe you would have seen this coming, right?

[00:11:06]It does feel it feels a little bit nonsensey, right? I'm disappointed that we had a whole another um right that we had a whole another post about everything and it's all about none of it is about what happened. I do feel like they need to if they're going to increase transparency, which they saying right here, we're going to increase it, then I want then I want to see it. Also, the fact that they said they recently updated the GitHub status page to include availability numbers. GitHub status page had availability numbers.

[00:11:29]You guys are the ones that took them off. What am I crazy here? Like, but they add a new ones, dude. And so, I mean, I do feel bad. I like honestly I feel bad for GitHub because they are the they are the first wave. They're the first wave of the agentic revolution, which means that they're getting so much code which does not even deserve to see the light of a CPU, right? Like this code is burning down rainforest and is utterly garbage doodoo code that deserves none of this. So, I do feel bad. I do feel bad that GitHub is GitHub is like the first ones to get the the DOS of the future, right? That is a private slop. It is private slop. I really feel like if they want to be transparent, they I expect within at least the next 2 days if they are fully transparent a well-written postmortem. I don't see it right. This whole thing was about like, hey, being foundation like we published an update on the two incidences. They didn't publish an update, right? This was not an update.

[00:12:20]This was you telling us how hard it is.

[00:12:22]This is not we we already know. We already know all this. You already told us all this. Oh, that's frustrating. I'm actually more frustrated. I thought we were going to get at least a postmortem.

[00:12:30]Now we're not even getting a postmortem.

[00:12:31]I'm actually like angry. I'm actually angry at GitHub. Oh my gosh. The result of this, by the way, for those that don't know, is directly after that. So, we're talking about PRs not even PRing correctly. Within 1 day, this happened.

[00:12:44]Poll requests. You can no longer look at poll requests. It said eight right here.

[00:12:48]None in here. 24 hours afterwards. Press one in the chat if you if you got hit with this one. I'm actually curious now.

[00:12:54]If I remember correctly, if you you I'm on repositories, and if I click repositories, it red the search. So, I'm clicking the page I'm already on. I'm on this page. Isn't that incredible? Just absolutely incredible. Incredible technology we have right there. It's not cached. Beautiful. Beautiful. Beautiful.

[00:13:11]Absolutely. Uh if you if you click it enough times, you you'll get rate limited, too. It's intended cuz F5 is slower. All right. So, yeah, poll requests were broken. And all of this led to this right here. Ghosty is leaving GitHub. I am GitHub user 1,299.

[00:13:27]Joined February 2008. I've visited GitHub almost every single day over the uh over 18 years. It's never been a question for me where I'd put my projects. Always GitHub. I'm super sad to say this, but it's time to go.

[00:13:40]Writing this makes me irrationally sad, but Ghosty will be leaving GitHub. I'm GitHub user 1299, by the way. I love how that sounds. That sounds so robotic and in the future. GitHub user 1299, explain yourself. Um, I joined GitHub I think in 2011, maybe 2012. So, I'm way past in 2008. I was using Tortois SVN. I was using DreamHost to dream host my SVN projects. I didn't even know about Git or GitHub. Since then, I've opened GitHub every single day. We've already read that. GitHub is the place that has made me the most happy. I've always made time for it. When I went through tough breakups, I lost myself in open source on GitHub. during college at 4:00 a.m.

[00:14:17]when everyone else has passed out. Let me get one commit in during my honeymoon while my wife was still asleep. Yeah, GitHub. It's where I've historically been the happiest and I wanted to be. By the way, uh Mitchell is just a huge dork. But a proper dork, by the way, a proper dork. Touch grass, bro. I mean, why would I say that? Why? Why, dude?

[00:14:33]Let a dork be a dork. You know what I mean? He doesn't need to touch grass. He does. If he doesn't want to touch grass, don't he doesn't have to touch grass.

[00:14:39]You know what I mean? I mean, he's the only person that I know that went from starting a very extremely successful, many, many, many million dollar company, going from CEO to CTO to individual contributor. He went backwards. Highest share owner in the company, individual contributor. Okay, the bros based. He just he just wants to he just wants to program. Okay, let the let the man do what he wants to do. Even the annoying stuff. Some people doom scroll social media. I've been doom scrolling GitHub issue since before uh that was a word.

[00:15:06]On vacations, I'd have bookmarks of different projects on GitHub. I wanted to study not just source code but OSS processes uh how other maintainers reacted to difficult situations etc etc. Uh believe it or not I like this. Some might call this sick but this is my hobby and work and passion all align and for most of my life they got uh to also live in one place on the internet GitHub. Did you know I started Vagrant my first successful open source project in a large part because I hoped it would get me a job at GitHub. It's no secret and I've said this repetitively not repetitively repeatedly. And in uh my first public talk about Vagrant when I was a mere 20 years old, I joked, "Maybe GitHub will hire me if it's good."

[00:15:41]GitHub was my dream job. I didn't ever get to work there. Not their fault, but it was the perfect place I wanted to be.

[00:15:47]The engineers were incredible. The product was incredible. And uh it was something I lived and breathed every day. I still do and constantly have for these 18 years enough time for an entire human to become an adult all on GitHub.

[00:15:58]Lately, I've been very publicly critical of GitHub. I've been mean about it. I've been angry about it. I have hurt people's feelings. I've been lashing out because GitHub is failing me every single day. And it is personal. It is irrationally personal. I love GitHub more than a person should love a thing.

[00:16:14]And I'm mad at it. I'm sorry about the hurt feelings to the people working on it. So I I I relate to this in the sense that there is something so inherently deeply disappointing when you when you interact and love something over and over again and then it sucks. Like for me, I'm going to be real here. So I had a very similar experience. I really uh liked Rust for a long time for those that don't know. I was I was uh I was on the Rust wagon for a while, right? And then I realized that the people running Rust, they just hate everybody. You know what I mean? There's this weird thing that happens and I I totally relate to this in the sense that you can develop like an irrational liking of something.

[00:16:50]That's just a thing, right? It it doesn't deserve that kind of adoration, but somehow you give it that type of adoration. And when you start seeing it fail for the dumbest reasons possible, it's just really upsetting. Unlike Mitchell, I would not say I'm sorry for the hurt feelings to the people working on it. For me personally, because I think those people uh generally speaking are horrible people. And so that's just that. Uh so I'm a little bit different.

[00:17:11]And so for Mitchell, it's just GitHub.

[00:17:12]And honestly, pretty based. Uh I felt this way for a long time, but for the past month, I've kept a journal where I put uh an X next to every date where GitHub outage has negatively impacted my ability to work. All right, I'm I'm clicking this link. Uh to the Git uh to the Git is distributed crowd. The issue uh isn't Git, it's the infrastructure we rely on around PRS and actions. Yes.

[00:17:32]Okay, good. Almost every day uh has an X. On that day I'm writing this post, I've been unable to do any PR review for 2 hours because there is a GitHub actions outage. There is no longer a place for serious work. Uh if it just blocks you out for hours per day, every day it's not a fun place for me to be anymore. I want to be there, but it doesn't want me to be there. I want to get to I want to get work done and it doesn't want me to get work done. I want to ship software and it doesn't want me to ship software. Yeah, for those that don't know, like jokingly uh uh the real GitHub status page. So, this is the status page that they show you, right?

[00:18:06]And this is kind of like the jokingly one, the missing status page >> where they have like uptime currently at this number, a much much different number with 96 incidences in the last 90 days. For those that don't know about this, the reason and people are like, well, this is actually really unfair and it's not actually true or accurate. The problem is is that when you have say one outage in pull requests, each one of these outages, even though it's not the whole site, like the site still works, if you're maintaining a project, you can no longer look at pull requests, right?

[00:18:35]Like it does not actually matter that you can you can go and search because you need to go and do a pull request. So when actions are down, which by the way, actions are down all the time, right?

[00:18:46]Actions are down all the time. This one's saying 97 uh% uptime. Obviously, the GitHub status page uh calculates actions being down at a different percentage. I'm not sure what the the record like there's two right here saying, "Hey, there were some with actions which aren't even present right here." So, I actually think that it could potentially be under reporting here or over reporting here. It's actually really hard to tell. But, nonetheless, if actions are down, your poll requests are down. If poll requests are not working, your poll requests are down. if issues are not working, you might not be able to like there's just so many of these things that happen over and over again that people just don't really have a good uh you know a good grip on. And so if you if you go here and this is more of how I use honestly this is more how I use GitHub. I jump on here and I clone things out, right? I have a very light usage of GitHub. I'm not using it like a power user like um what's it called? Mitchell is because I just don't have any big drinks that or big drinks, dang it. I just read this prime drink and dehydrated pee. I don't have any big uh projects that I actually want. I I don't like doing open source.

[00:19:45]There's a lot there's a whole group of people that don't realize how bad it is for someone who's always on GitHub. Like if you're lightly on GitHub, this is probably not a very big problem cuz you just kind of go on if PRs are down, you got a lot of work anyways, you go over to Jira or wherever the hell you do stuff and then you go and you look at some stuff and and you make some more changes and by the time you're done anyways, the thing's back up and you're just like, "Okay, whatever. Hey, go do this." you have to wait, you know, 36 hours for a review anyways. So, it's not like you're really um what's it called?

[00:20:13]It's not like you're really like, "Oh my gosh, if it's not up in the next hour, this isn't going to work." I mean, for us, it was, you know, we had Jira, we had GitHub, all the stuff was tracked on Jira, all the actions were done on GitHub. So, it's like I had this constant switching back and forth. And so, if something was down for a moment, it didn't really bother me because no poll request got through within 48 hours anyways. So, it's like, "Oh, okay, whatever. I don't care. Our team is small. It just works." I want to be better. Let's see. I want it to be better, but I also want to code and I can't code with GitHub anymore. I'm sorry. After 18 years, I got to go. I have loved to come back one day, but this will have to be predicated on real results and improvements, not words and promises. I'll share more details about where Ghosty project will be moving in the coming months. We have a plan, but I'm also very much still in discussions with multiple providers, both commercial and foss. It'll take time for us to remove all the dependencies from, let's say, on GitHub, and we have a plan in place to do it as incrementally as possible. We plan on keeping a readonly mirror available on GitHub at the current URL. My personal projects and other work will remain on GitHub for now. Ghosty is where I, our maintainers and our open source community are most impacted. So that is the focus of this change. We'll see where it goes after that. So Mitchell's the real casualty here. There's going to be a lot of people. Uh Zigg already moved off for this exact reason. For those that I mean well Zigg moved off for two reason. Um unfortunately the uh what's it called?

[00:21:27]Unfortunately the lead Andrew the lead guy, he just cannot stop. Uh he's really really really intense on the activism stuff. So something about Israel or ICE.

[00:21:36]There we go. Ice is IN HERE LIKE GET LIKE RIGHT like he can't help it. But uh it's actions that really made him very very very upset and he's very very very upset at the fact that Zigg constantly was getting just absolutely destroyed by actions and so they moved off of it because of that. So Zig is Zig is already jumped off the train. Zig is already done. Zig is already leaving the world uh of GitHub and they've been gone for a while. Obviously, GitHub uh sponsors are a big deal for them. They have a migration plan. They're going to be going on to I think Codeberg. Was it Codeberg? Codeberg. Yeah, Codeberg. So, they're now on Codeberg. This was this was last November. And that's because of how bad GitHub has been. And that was in November. That was long before uh the general this thing where they're like, "Oh my gosh, so much more stuff has been happening." Well, they only doubled the amount of traffic that they were having for all their basic function from 2023 to 2024. And during all that time, stuff was already going massively downhill.

[00:22:31]From 2024 to 2025, it was also, you know, it was also, yes, growing, but not immensely like it is in the last couple months, but it was still terrible to the point where other companies were leaving. So, it's not too shocking that Ghosty, honestly, not shocking. If you're hosting a big project that requires you to be on GitHub all day, I'm surprised you're remaining on GitHub. I only use it because I use it lightly. I'm not completely stuck on uh what's it called? I'm not I'm not using it nearly the same as someone like Kagosti is. And so I have a very very different one. Uh Zig is the most starred repo in Codeberg. That's pretty cool. Is any of this shocking? I guess no for me. None of it's shocking. None of it's none of it is what all of this should have been easily like seen coming years and years ago, right? Like I I would be I would have seen this coming at least 2 years ago and I'm surprised it's honestly taken this long. Like to me, what's super impressive is how long how long people have stuck with GitHub.

[00:23:25]Like they've been they have such goodwill built up that they've been able to absolutely [ __ ] on the product, not have a CEO, and run it dangerously into the ground for so long. It is absolutely