Video codecs explained: H.264, AV1, HEVC, VVC | Lex Fridman Podcast

Added: 2026-05-09

[00:00:02]We hinted at awesomeness of codex, the depth and the richness and the complexity of everything involved there.

[00:00:08]What let's try to define what is a video codec? What what's involved there? What what does it mean to compress something?

[00:00:15]You already started to hint at it, but can can we elaborate a little bit more?

[00:00:19]So there's a huge amount of redundancy in any video both spatial and temporal.

[00:00:24]And the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process.

[00:00:31]So more often than not using several orders of magnitude more compute to compress because that's more costly versus both costly both financially and in CPU resources versus the decompression. So it's asymmetric in that respect. Often the case because compression is done once, but there could be lots of viewers of another file.

[00:00:50]So to take that information and compress it by 100x, 200x removing redundant information and using mathematical properties to make that small, but also have properties such as error resilience. So as as JB suggested VLC in the beginning was was used to play UDP network feeds. And UDP network feeds lose packets. And so some of the design goals of a codec is also to be recoverable. You need to actually be able to join the stream. It's not necessarily a file. You need to join get on the decoding process and start decoding. And and to give it more image to to to people who are not familiar, right? Like when you're going to see any type of movie, right? You're going to see the camera is going to pan, right? And and and travel. And you realize that for example, all the background is the same from for like a minute, right? Or 30 seconds, right? So you can reuse the cloud that you see on the background. You can reuse that from a frame to another, right? And so it's gets the more the more memory you have, the more power, the more comparisons you can make, right? And so the more compressed you can be. And most of the modern codecs are basically doing that. So just to make it even more explicit. So what is video? Video is a bunch of pixels of RGB three values and you have a grid of pixels and you have let's say 24 30 or 60 of frames a second and you just have all these pixels repeating and showing different stuff 30 times a second. And so the question, the philosophical the technical question is how can I compress all of that store all of that at 100x?

[00:02:38]>> 1000x, right? 1000x.

[00:02:40]>> The target is 1000x, right?

[00:02:41]>> And the goal is when you say redundancy what is redundant meaning stuff at best that humans wouldn't notice if it was missing. So for example, you have a picture of a cloud, right? And from the next frame they're still going to be the same cloud. So it's redundant. You could just put it once and not do it, right?

[00:03:01]Or you have a a black background behind me. For example, the black is the same on the whole picture, right? So you can say, "Well, you know, in this picture take the pixels that you have on the top left and the one on the top right. I'm not going to give the value. I'm just going to tell you it's the same as the top left." And then you can say for frame one, reuse something from the previous frame or the previous previous frame and so on and so on, right? So you could basically it's unlimited, but then it's limited in terms of memory or in terms of compute power. Because for example, if you need to compare pixels on 200 frames in the past on 4K resolutions, it's a huge amount of compute. And then when you're showing it, you have to do the decompress of all of that. So you is it the codec that has the encoding and the decoding is a is a coupled process that you're developing?

[00:03:54]>> Exactly, right? And those are two different um trade-offs, right? Are you going to compress more, but then it might be more difficult to to to decode?

[00:04:05]Are you going to to make it a codec that is more complex to encode and easier to decode? Are you going to make a codec that is easier to encode because you need to be fast, but then the the client side, the player is going to spend more time? That's why you have so many different type of codecs is that it's not always easy. And to make it even more complex, modern codecs like AV1, AV2 or VVC are actually not codecs.

[00:04:29]They're a collection of tools, right?

[00:04:31]They are multiple tools, multiple codecs in the same codec to depending on the image get the more compression. So just to elaborate codecs like AV1, VVC have a much wide have a wide audience.

[00:04:45]It could be a screen share content. It could be video. It could be animation.

[00:04:50]All of these require different coding tools. So what happens these days is a collection of tools are put in and called AV1, called AV2, called VVC to allow for different use cases. So you may be on Zoom and sharing your PowerPoint and then you need to show the audience a video. That codec needs to start changing its tool set depending on the content to compress in a different way. And like you said, there's a a bunch of incredible engineers behind each part of that, each part of the tools that make up AV1. Sure.