The Truth About Debian's Reproducible Packages | Source Code Ep. 20

Added: 2026-05-26

[00:00:00]Forgive me, I know. The new version of Vivaldi is out with a redesign. Looks interesting. Open Seuss is trying to push systemd even harder and now its installer Agama, which between us, the last time I tried it was awful, supports and favors system boot. Well done. Keep going down this road. KDE 6.7 is in beta and when it comes out, I'll make a dedicated video because this release, oh boy, it seems like a lot. Honestly, I have the feeling KDE is reaching a certain maturity. They haven't redesigned anything for at least a year.

[00:00:31]Well done. A sincere compliment.

[00:00:34]Anyway, the usual blah blah blah of the week's news. But this week, I want to talk to you about a single piece of news. And I'll try to explain to you, at least from a technical standpoint, how things really stand with this whole story of Debian's reproducible packages.

[00:00:52]Oh, wait. I was supposed to tell you about something else, but briefly. When I read this piece of news, my eyes went wide. Rust could eliminate 70% of the Linux kernel CVEEs.

[00:01:03]Sure, just like I could win the New York Marathon, maybe go to the moon and stand on my hands for more than a few seconds.

[00:01:09]And I assure you, my balance is precarious even in a normal posture.

[00:01:13]Anyway, let's try to be serious. Maybe I do talk some but as a famous politician once said, "I may not be great, but I don't see any giants around me." Rewriting something in another language is the most stupid and senseless thing you can do. It's no accident they're called programming languages precisely because they're languages. Picture all of Shakespeare rewritten in German or the other way around Gotha rewritten in English. Yes, translations exist, even excellent ones, but they remain translations. The power, the beauty, and the depth of a work in the language it was written in is unsurpassable.

[00:01:52]And the same goes for programs. And we're not talking about a little hello world window of a few lines of code.

[00:01:58]We're talking about a kernel with millions of lines. Let me say it plainly. Let's not spread bad information. Let's not feed false myths.

[00:02:07]Let's not create urban legends. The introduction of Rust code into Linux is not a subject to tackle with slogans.

[00:02:14]And it strikes me as absurd that even the colonel's own lead developers frame it in these terms. Even if they're speaking at a Rust conference, they should have been more cautious. The kernel is a complex project. It has 40 million lines of code, many of which are drivers. Rewriting or reintroducing the Rust language, and making it coexist with C, in my opinion, will bring more problems than benefits. And the time spent on this whole process will be more than double what it would take to write excellent C code. But this is my opinion and my reasoning isn't ideological against Rust. It's precisely semantic. I would be just as opposed to rewriting Cosmic Desktop in C given that it was designed in Rust from the very start.

[00:02:58]But what's the point? It's just a waste of time, smoke in your eyes, propaganda.

[00:03:03]What I think our target should be is elegant, functional, simple code that solves real problems. Let's try to be more realistic there. This connects right to the main piece of news I want to talk to you about because I love the way the world reacts to words, to labels. We all live immersed in this great circus of the fantastic product, the exceptional performance, the obscure acronyms that by the mere fact of existing confer on an object an aura of intrinsic desiraability.

[00:03:32]Pure aesthetics, semantic decorations, while the state of the facts underneath is an entirely different story. And this happens in our world too. Especially in our world which despite being populated by extremely competent people is by no means immune to this kind of idolatry.

[00:03:49]Quite the opposite. It leans into it.

[00:03:51]Linux is secure. Arch is the best dro.

[00:03:54]Whand is the future. Rust is memory safe. Therefore, it's safe. Slogans that function as amulets. You utter them. And the technical conversation closes.

[00:04:04]Debian's announcement of reproducible packages fits squarely into this narrative. And I already know that a few years from now I'll have someone in the comments writing me, "Yeah, but look, Debian has reproducible packages as if it were a magic formula, a stamp of guarantee. The end of the discussion."

[00:04:21]Really? So, let's do this properly.

[00:04:24]First, we look at what they are, what they're for, and why they're useful.

[00:04:28]Then, we analyze Debian's announcement and what it actually entails. Because between we've introduced this thing and that thing means what you think it means, there's an ocean in between and nobody feels like swimming through it.

[00:04:41]Part one, what reproducible packages actually are. Stripped of all the noise, the concept is trivial. A build is reproducible when starting from the same source code and the same compilation conditions, it produces a binary file that is identical bit forbit, regardless of who runs the compilation when and on which machine. Sounds obvious, it isn't.

[00:05:03]A normal compilation is full of hidden non-determinism. And this is the first point nobody tells you about. The typical sources of divergence are at least eight. First, embedded timestamps.

[00:05:15]There are C language macros that during compilation write into the binary the exact date and time of that moment.

[00:05:22]There's the last modified date of the files inside archives. There are build dates written into binaries for debugging purposes. compile a minute later and the binary is already different. Second, file system ordering.

[00:05:35]When a program asks the system for the list of files in a folder, that list has no guaranteed order. So the linker, the program that assembles the compiled pieces, receives them in different orders depending on the underlying file system. Same symbols, different addresses. Third, build parallelism.

[00:05:54]Compiling using eight parallel processes produces a different assembly order from compiling with 16 and the more aggressive optimizations make things worse because certain compiler decisions change with the number of threads.

[00:06:08]Fourth, absolute paths. Compilers write the path of the source file into the binary. So a builder working in a maintainer's personal folder produces a different binary from one working in a system folder. Fifth, language and time zone. The ordering of strings, the format of dates and automatically generated documentation, the output of certain tools, all of it changes depending on the language and the time zone set on the machine. Sixth, process identifier, machine name, number of CPUs. These are data that end up written into build logs and those logs sometimes get shipped inside documentation packages. Seventh, actual randomness, randomly generated build identifiers, random values used for certain optimizations, stored memory offsets.

[00:06:53]Eighth, the exact versions of build dependencies, a slightly different system header, a compiler with a distribution specific patches, and the binary changes.

[00:07:03]The work of the reproducible builds project, which is a cross-distribution effort, not just Debian, has for 10 years been precisely this, closing these holes one at a time. They standardized an environment variable called source date epoch which the tool chain honors to fix the build date. They fixed tar so it could sort files by name and zero out their modification date. They pushed developers to make the option that removes absolute paths from the binary the default in the rust and go compilers. They taught Debian's packaging tools to normalize file order and so on across thousands of packages.

[00:07:39]This is the point I want to be surgical about because this is where 90% of the misinformation lives. Reproducible packages do not serve to verify that the binary corresponds to the source. This phrasing is wrong and I'll explain why below. Protect you from malicious code in the upstream. Protect you from a compromised compiler. Guarantee that the software does what it claims. They serve exactly and only one purpose. to allow independent rebuilders to take the declared source and the build metadata, the build info file with the exact versions of all dependencies, recompile and compare their own output against the binary distributed by the maintainer. If the bits match, there's one guarantee.

[00:08:23]The binary is mechanically derivable from that source in that environment. No more, no less. What does this give you in practice? A specific defense against one attack vector. Compromise of the official builder. If the attacker penetrates Debian's build server and injects code during the compilation of the openest server package, without reproducibility, nobody notices. The package arrives signed through the official channels. Users install it. End of story. With reproducibility, dozens of independent rebuilders in theory recompile the same source, get different bits from the official package and the attack surfaces. This is important. This is useful. This deserves the work that's gone into it, but it is only this. Three points that are systematically left out of the narrative. First, reproducibility does not defend against trusting trust.

[00:09:15]This is the attack Ken Thompson described in 1984. A compromised compiler produces compromised binaries in a perfectly deterministic way.

[00:09:23]Recompiling the source with that same compiler returns the same malicious bits every time.

[00:09:29]The build is perfectly reproducible. It is also perfectly compromised. The only known defense is David A. Wheeler's diverse double compiling which requires recompiling with independent compilers, something Debian neither does nor mandates.

[00:09:44]Second, reproducibility does not defend against upstream attacks. The Exilma case from March 2024 is the textbook example. Gotan spent two years building trust within the upstream project, then injected a back door into the tarball's build process. That poison tarball would have compiled perfectly reproducibly in Debbian. The signature of the attack wasn't in Debian's infrastructure. It was in the source Debian received.

[00:10:10]Reproducibility, zero protection. Third, Debian style reproducibility is the weak version. There is strong reproducibility. Anyone with any reasonable environment gets the same bits and there is weak reproducibility.

[00:10:24]Anyone with exactly the environment we fixed in the build info file gets the same bits.

[00:10:30]Debian does the second one. This means the build info file has to contain the exact versions of hundreds of dependencies and the rebuilder must be able to retrieve them.

[00:10:40]That's what Debian's historical archive is for. If the attacker controls even just one of those recorded build dependencies, the rebuild confirms the attack. It's security through delegation of trust, not elimination of trust. Put brutally, Debian style reproducibility protects you against an attacker who compromises the single builder and only that against an attacker who compromises upstream the tool chain or a build dependency. Nothing. On the 9th of May 2026, the Debian release team announced that migration from unstable to testing will be blocked for packages failing the reproducibility check in view of Debian 14 Forky expected in 2027.

[00:11:22]On the official channels and in the trade press, it was sold as Debian must ship reproducible packages. A giant leap in commitment, they say.

[00:11:31]Now, let's take the announcement apart calmly. The Debian policy manual was not modified. Bug number 8444431, which has been asking for 9 years to promote reproducibility from should to must in the policy, is still open. What was done is an operational change in Britany, the script that handles testing migration. Britney now refuses packages that don't pass the tests of Debian's verification site. Sounds like a formal distinction, but it isn't. The policy is ratified, written, appealable, has a review process. A Brittany rule is an operational decision by the release team modifiable, susceptible to exception, subject to casebyase interpretation.

[00:12:12]We have de facto enforcement without a djury norm. This is an inversion of Debian's process and it deserves to be noted. On the day of the announcement, Debian's verification site was showing over 98% of packages reproducible for Forky across all main architectures with 23,728 good packages. The number was waved around everywhere. It is also as presented misleading. The packages that are easy to make reproducible became so years ago. Modern tool chains recent build systems mason cargo go actively maintained packages solved. The remaining 2% is not uniformly distributed. It's concentrated where it hurts the most. Ancient auto tools code generators with embedded timestamps.

[00:12:54]Java O'Camel hasll with compiler quirks.

[00:12:57]scientific packages with forran dependencies, firmware with binary blobs, often infrastructural packages, often orphan or near orphan packages maintained by one person in their spare time. The verification sites test strongly normalizes the environment paths, machine name, time zone, language, number of CPUs, kernel. It's the weak reproducibility I mentioned above. The 98% refers to that specific test, not to anyone recompiling gets the same bits. what it actually entails translated into operational consequences. For maintainers of the residual packages, the problem is theirs even when the cause is upstream. The package fails the check, doesn't migrate, and the maintainer has to choose between patching downstream, permanent technical debt, pushing upstream, and waiting weeks or months, or watching the package get removed from testing.

[00:13:50]For orphan package maintainers, this very often means the third option. for minority architectures. Riskv64, PPC 64-EL, S390X, MIPS, and now Loom 64 promoted in December 2025 have fewer builders, less test hardware, and fewer people capable of debugging architecture specific reproducibility issues. Codegen Indian, struck packing, dynamic linker symbol ordering. A package can be reproducible on AMD 64 and fail on S390X for reasons that require an expert in that architecture.

[00:14:22]The policy creates asymmetric pressure on the weak architectures and this cascades into which architectures remain release blockers for public messaging.

[00:14:31]And this is where I come back to my intro. The announcement was received with the usual liturgy, celebratory articles, claims that Debian is now more secure, hacker news threads praising the rigor, but the marginal security added is very specific and very limited protection against the compromise of the single official bill. Damon, nothing about trusting trust, nothing about upstream attacks like XZ, nothing about back doors in the tool chain. And this matters to us as informed users because a few years from now, someone will write me in the comments, "Look, Davian has reproducible packages as though that were an argument." It isn't. It's a precise technical property with a narrow perimeter of guarantee and implemented with a mechanism, a Britney gate without policy ratification that ought to make Debian itself uncomfortable. What Debian could have done done better so as not to be the one who only complains. Ratify the policy before enabling enforcement, not 6 months after. Publish an explicit threat model. what we protect against, what we don't, leaving users the chance to understand what they've actually bought. Fund a central team to attack the non-reproducible residue rather than offloading the cost onto individual maintainers.

[00:15:49]Introduce the gate gradually, a warning and unstable, a soft block in testing with a public traceable override, a hard block only for security critical packages, tool chain, kernel, init crypto. push in parallel for actual independent rebuilders because without third parties who actually recompile and verify, reproducibility is only a potential property. Reproducible packages are a good thing. The work done by the reproducible builds project over 10 years is excellent, technically serious, and deserves respect. Nothing I've said above contradicts that. What I take issue with is the transformation of a precise technical property into a label of glory. A stamp you brandish in conversations as proof of something it doesn't prove. DBN has reproducible packages is not the answer to is DBN secure. It's the answer to a much narrower question. If an attacker penetrates DBN's build server and tries to inject code into a single package, can we catch it by comparing against an independent rebuild? Yes, now we can.

[00:16:51]Good. Let's move on. But labels on their own secure nothing. And we who ought to know this better than anyone else keep buying them just like everybody else.

[00:17:01]See you in the next episode. And as always, may Linux be with you.