x264 explained: The video encoder that dominates Internet video | Lex Fridman Podcast

Added: 2026-05-11

[00:00:02]I don't think we quite talked about x264 properly. We talked about David. Can we return backtrack a little bit to x264, this thing that powers basically all of video on the internet. So can you tell me the story of x264 and Kieran, you're actually a contributor to x264.

[00:00:23]>> So x264 is is a video encoder for the H.264 video standard.

[00:00:28]It dominates internet video, but also other areas such as Blu-ray discs. And Blu-ray discs are interesting because the people that make them really want the highest quality.

[00:00:36]And there's some really cool high-end films that have been encoded broadcasting and all sorts of other areas. x264 was a big step change um cuz it kind of happened at the right time as well. A lot of the development took place when HD video was coming out.

[00:00:52]Intel Core 2 and Nehalem CPUs were getting fast. You could do real-time video, but the most the most important thing was um a key sort of focus on visual metrics.

[00:01:04]So industry and academia for for 20 years before was obsessed with um mathematical metrics or what's known as peak signal to noise ratio. So mean squared error, logarithm of mean squared error. And that led to tons of issues because mean squared error um leads to blurring cuz you actually want to you want to minimize you want to add a little bit of error to everything to to reduce the mean squared error as opposed to having a big error. And that led to loads and loads of blurring.

[00:01:28]So but hobbyists bucked that trend. It was for their own personal videos, mostly anime.

[00:01:33]So there were two there were two things they did differently and there's a big iterative feedback loop with the community.

[00:01:39]They did some stuff differently um two two big things, psychovisual rate distortion. So using block energy trying to compensate for human perception when making decisions.

[00:01:50]So the psychovisual distortion, that's the critical thing. That's the thing I mean it's kind of revolutionary like that we can like rethink.

[00:02:01]Don't don't make it like this kind of theoretic thing of compression.

[00:02:06]Make it all about >> being pleasing visually to the eye.

[00:02:09]>> Yeah yeah. So compressing in a way that loses the least amount of information for the stuff that matters for us humans.

[00:02:15]>> Yes, exactly. As opposed to what industry some parts of industry are still obsessed by this which is mathematical numbers that don't look good in reality. And then adaptive quantization was the other big one where it was biasing bits against um complex areas and and redistributing them to less complex areas like grass.

[00:02:34]Grass has some high frequencies, but it's kind of it's less complex overall compared to more complicated things. And this came around by um Park Joy. So Park Joy was really the canonical sample that was It's the running around in the park.

[00:02:49]>> This one. Yeah. So this guy was really the um So this this was created by Swedish television in um the beginning of HD and it was done on film and it was no expense spared in terms of production quality and it was given away for free.

[00:03:04]This was really and this is the sample really that sorts the men from the boys in terms of it has so many challenges with the trees, with the water, with the grass, with the motion, with the I don't think there's there's still been any any public test sequence as good as that these days.

[00:03:22]>> So for people who are just listening, we're looking at a bunch of humans running along a river.

[00:03:29]So you have the reflection, a lot of high information textures everywhere, the leaves and the lighting playing with the leaves and all of this. You could show clearly that encoders with high PSNR will blur everything.

[00:03:42]>> will blur everything and you could see actually I could turn on psychovisual stuff. I could turn on adaptive quantization and it would just look so much better, but your your metrics and these metrics are at the time were at the time were considered so holy. These are the holy metrics that are untouchable. PSNR is the most important thing. Uh can you speak to how do you measure psychovisual stuff? Like how do you turn how pleasing a compression is for a human eye into a number? Is that even possible?

[00:04:11]>> That's what that's what Netflix have been trying to do with VMAF. They said they've used a machine learning model.

[00:04:15]That's a more recent thing, but back in the when x264 was being developed, that's by eye you're basically >> eye. It was it was developers on their laptops. So it's not like even with big companies with professional screens or anything. It's And that was actually one of the goals which was I don't the developers at the time, Lauren Merrick in particular is I don't want to test this on a $30,000 screen. It's I want this to look good on someone's laptop at home. Yeah. Brilliant. Um there is another sample which is a sample that is a Planet Earth killer sample that I absolutely love and you are going to see why right now.

[00:04:52]>> Um it's a ton of birds, right? Flying and the more it goes, the more there are birds and at the end, right? It's almost like you have millions of birds. It's the most complex thing ever to encode, right? And you well you're watching it on YouTube and you see how bad the YouTube encoding is actually, right? Um and this is like phenomenal to to optimize and get um perfect quality in a constant bitrate.

[00:05:22]Um there was a lot of optimization mostly by by Lauren also on anime, right? For a long time anime was very badly encoded because there was a ton of bending, right? And so you see those those issue and there was a ton of things. Um so x264 is like and today it's still the reference to any encoder new encoder AV1, AV2, VVC, HEVC.

[00:05:47]Everyone compares to x264. One of my favorite films um Cinema Paradiso. I know the engineer who created the Blu-ray initially. He showed me the comparisons of x264 versus others and uh that that's completely different and I think a bunch a bunch of guys in the Blu-ray world started using x264. Um I think the big one was Chris Anderson from Warner Brothers. He did the whole Friends box set with that's a quite like a thing a person on the street actually watches and wants to look good and so they kind of took a risk in their jobs doing that cuz they're in a big company. That big company can buy whatever they want and they said no no no, I want to use this free and open source thing so that things look good for my my customers and build the best and to this day I personally still try and avoid watching the most cinematic films on streaming services and buy the physical discs because they look they look good without even having to buy an expensive TV. I think that's the key thing. And x264 is yet another example of open source project. It was started by Laurent Aimar when he was at the Ecole Centrale Paris where VLC was born and then you got generation of people like Lauren, like Jason, like Mans, like so many Andrew Henrik Henrik [clears throat] and this is Anton and this is where the assembly thing that we use now on FFmpeg David and so on was born, right? So x264 is like amazing project with people who were really all over the world and I think most of them never met each other.

[00:07:11]But all of them according to Kieran or large percentage love anime.

[00:07:16]There's several things I've never got into and one of them is anime and I need to >> I watch anime so much. Um especially at that time um like at the time it was like a lot of anime content doesn't exist commercially, right? So we are before Crunchyroll, right? So what happens is usually people who love anime who take some things some DVDs in Japan and rip them because this there is no commercial offering and some of the people who are what we call fansubbers are basically translating themselves to make subtitles, right? And at that time you download completely illegally. It was the only way to do that, right? And so all of that was handcrafted and it fits the open source community, right?

[00:08:01]Because they needed tools to encode to do fansubbing, right? One of the most amazing open source project for subtitles is called Aegisub and it's it's a languages. There are weird textures in anime that I don't think you get in real life content. I think that was a key one which was optimizing these weird textures that you get because anime is not done in a normal fashion. Yeah, the way you produce is not you mostly produce it like on screens, right? Since a bit of time and you have all those gradients, right? In colors because they are very easy to produce digitally, very complex to to produce in real life.

[00:08:43]And and the subtitles also are very complex because you need to have often the Japanese and then you need to have the diacritics, right?

[00:08:52]The the what we call the ruby, right?

[00:08:53]Which is the hiragana and the katakana for the kanji. And then because of course you so that you have the official subtitling, but you also need the English subtitles or the French subtitles because you want to to learn that, right?

[00:09:06]And there is so many things crazy on subtitles and we had like crazy samples on on subtitles that we've seen all around. So this is an important part of the culture, but also because there was no official offering. There was no way of doing that.

[00:09:22]Uh can you speak to the difference in H.264 and AV1 and then x264 and David? This is this big step. Can you help people understand are some of the streaming sites moving more towards that direction of AV1?

[00:09:38]Let's be honest. All of those codecs since MPEG-2 video are the same concepts.

[00:09:47]The same concept about inverse transform, about intra prediction, motion compensation, entropy coding. All of them. However, each generation gives you a bump between 25 and 50% more compression for the same quality.

[00:10:06]Um and so you had the MPEG-2, you had the DivX area, you have H.264 which was like changing, right? H.264 improved so much.

[00:10:16]And then you had more, right? You had HVEC, um you had VP9 at the same time of HVEC. VP9 is a bit similar to HVEC in terms of compression, but it's royalty-free because in multimedia there is ton of patents and the licensing after H.264 became out of hand, right?

[00:10:36]And could cost hundreds of millions of dollars per year. So it made no sense.

[00:10:40]So Google did this VP9 and the Alliance for Open Media did this new codec called AV1.

[00:10:47]Um so you can imagine that AV1 saves between 40 and 60% less bandwidth than H.264 for for the same quality visual quality. At a given bit rate. At a given bit rate, right? So that's really like you increase the quality either you you set the bit rate and you increase the quality or you set the quality and you decrease your bit rate. But because now you move from from SD to HD and HD to 4K and 4K to 4K HDR, like you're increasing the size by like two factor two, three, four, right? So you need to have better compression to keep it in terms of something that is manageable. It's more coding tools, more bigger blocks, lots more sub-partitions in each block. It's just exponentially more complex. It's more complex because the the encoder needs to search more possibilities, right? So you for example, um one of the things that is easy to understand is to predict a block a a color block to another, you have directions, right? You can go left, right, bottom, up and then in terms of uh like the other quadrants, right? Uh what are called north north east, north west and and so on, right? But that's eight directions. Then you can do more direction, you can do 16 or 69 or 128, right? You can and every time your encoder is going to spend more time to see well this blocks is exactly this one um and those type of tools that you can bring and the encoder needs to check which of the tools are going to compress you better. And so I guess that AV1 encoding is two order of magnitudes more than H.264 in terms of CPU cycle, right?

[00:12:30]Order of magnitudes, right? Yeah. And as we discussed CPUs are not getting faster. You're just throwing more cores at the problem. But also it's a fact that you encode once and you have hundreds of millions of users, right? So for example, YouTube, very good example.

[00:12:44]YouTube encodes almost everything in H.264, but the popular video get re-encoded in AV1 because it costs more, of course, to encode, but you encode once and you send that to millions, right? So it's a trade-off between encoding time and complexity and and CPU usage on the server side and on the client side because at the end if you're distributing a video to hundreds of thousands of people and the size is half of the other, then it's better.

[00:13:14]It's better for your battery, it's better for your modem, etc. etc.