The impossible task of testing FFmpeg code | Lex Fridman Podcast

Added: 2026-05-11

[00:00:02]That reminds me of a thing that you uh mentioned to me about fate testing and like the insanely rigorous process that's used to test everything that's incorporated into um into FFmpeg. Can you take me through the testing process?

[00:00:16]>> Yeah, so FFmpeg has a system called Fate FFmpeg automated testing environment.

[00:00:20]>> Because FFmpeg runs on so many different oss and can be compiled with so many different compilers, there's been a crazy number of configurations. So you can see the absurd combination of um compiler variants, operating system variants, instruction sets. You can see at the top Mac OS has tons of different variants because it has iOS, it has tvOS. I'm looking at a page fate.fmpeg.org or 81 minutes ago, 76 minutes ago looking at the different architectures, the operating system, the different compilers, Apple clang version, uh >> combinations are the combination.

[00:01:01]>> These are all run by volunteers. So these are all volunteer systems. The ones at the top, for example, the max I host in my office for example, uh host all sorts of different stuff. Other people host other things. So it's really there to make sure and because FFmpeg does quite complex C code for example you do have miso compilations. So the compiler will sometimes compile C code incorrectly for example. This happens once in a while. Oh there's like there's a log of all the compilations.

[00:01:29]>> Yeah. Log of all the compilations. All the tests. I think one of the other ones will show all the tests passing.

[00:01:34]>> If you click you can see all the tests.

[00:01:36]Yeah.

[00:01:36]>> Back all test successfully >> in logs test. Yeah. So you see all those tests are passing of all the different codecs, all the different filter transformations, all the [laughter] the level of scale is is quite crazy on all the combinations.

[00:01:52]It's it's not just a matrix at this point. It's like a pivot table of different combinations.

[00:01:57]>> That's nuts.

[00:01:58]>> And it's a key part of what we do because you you may be able to test something locally. you make a change but actually that breaks GCC version 11 on Mac or something like that and you're able to then fix that. We also have miso compilation so the C code sometimes the compiler can have a bug in it where it creates the wrong output and that can have quite a big effect sometimes on a vid on a video because of the way frames have dependencies even a small change in the output can can cascade to actually quite big glitches. You see power PC, you see risk, you see PC, there was risk, there was weird stuff in the past like decisual studio different >> Visual Studio, Intel compiler, Apple client, you name it.

[00:02:42]>> What are some of the pain points like maybe do you have emotional triggers uh maybe nightmares about a particular operating system, a particular container codec combination?

[00:02:54]>> For me, it's really easy because so I have a day job. Um my company builds the company I started builds um equipment for broadcasting sports matches between TV stadiums and studios for example we have to work with 10 bit video and 10 bit video has um a set of challenges that you can't process 10 bit data natively on a CPU so that means you have to stick it in um 16 bits so that means you have six wasted bits so there's different packing formats to actually pack the data more efficiently because when you send that over for a network you lose cuz you need to you need to save that 40%. For example, on PCI Express you may only have bus bandwidth to do that and so I think internally we have about some some are some are industry ones and some are internal to our own hardware that we build. We have a I think a 5x5 or 6x6 matrix of every single format to every single other format conversion. In fact, one one of them I sent you and they're all written in handwritten assembly and they're all written and they all support different CPU generations.

[00:03:58]So, this is really traumatic handling all these different combinations times a million.

[00:04:03]>> By the way, the company you're talking about is Open Broadcast.

[00:04:06]>> No, no relation to the free OBS streaming service.

[00:04:08]>> Yeah. Yeah.