Rust in Ubuntu

[00:00:16]Hey everyone, welcome to another Rust Rover live stream. I am Orhun. I'm a Rust developer advocate at Jet Brains.

[00:00:25]I'm also an open-source maintainer and lead maintainer of Ratatouille.

[00:00:31]Today we are talking about a topic that brings a lot of things that we care about together. Rust, Linux, open source system software and bunch of other things. So Ubuntu is a mostly w most widely like deployed operating system links distribution. It has 15 million deployments all around the world. It has been used like I think 33% but don't quote me on that. I got this info from an LLM and it's a general like it's been like deployed on general purpose edge computing server servers and a lot of things and with the 25.3 release rust started taking over Ubuntu and today we have the person behind this rust adoption uh he is responsible for many rustification of Ubuntu tools and Ubuntu um tooling. Um John John Seager, welcome to the live stream. How are you doing?

[00:01:43]>> Thank you very much. It's good to be here. I'm uh I'm doing well. It's a little warm in the UK at the moment, but otherwise doing well. We're not used to it over here, so you know how it is.

[00:01:52]>> Yeah. So, you are the VP of engineering at Canonicle. And um I want to just like first like talk about uh your history with with uh Rust and Ubuntu in general.

[00:02:04]So how did you uh get to this position?

[00:02:08]Can you just talk about that briefly?

[00:02:10]>> Yeah, so I've been at Canonicle for just over five years now. For the first four years of that, I was focused on our cloud automation portfolio. Essentially, we have an open-source control plane called Guju and we build operators. And then two years ago, I was asked to take on leadership of Abuntu. And I I wouldn't necessarily say that at the forefront of my mind at that point, Rust was part of the story. But what was part of my story or I felt very strongly we had to do was take a moment to reflect on the fact that Abuntu had been around for 20 years at that point. Uh and you know firstly what a kind of wild ride that had been and in many senses against the odds it had been quite successful.

[00:02:53]But my view, the more I thought about it, was that the set of things that enabled it to be successful for those first 20 years probably were not the same set of things that were going to enable it to stay relevant and be successful for the following 20 years.

[00:03:06]So I did a bit of soulsearching I guess bit of thinking about what I want how I wanted to shape the platform and a big part of the story for all of our users whether it be uh users in education students whether it's startups or even fortune 500 companies or you know journalists whatever it might be is security and resilience and the in my opinion at this point rust provides the most compelling set of tools of the systems, programming languages that we have available to us for progressing that agenda essentially as well as often bringing the benefits of increased performance and I think critically a crowd of people who are interested in thinking about this category of problems. Uh I said this at Rust Nation back in February, one of the most interesting things about the Rust programming language is the people who are drawn to it. And so I wanted also to provide a path or an obvious path if you will for people who like solving these kinds of problems to come and contribute to them too.

[00:04:08]>> So when you talk about Rust I I watched your talk at Rust Nation and uh I read some of your blog posts. You talk about resiliency. Can you just explain that to us what Rust why Rust is resilient and why did you see that as as a resil resilient programming language at the time because there might be some other options but you specifically went with Rust and I want to just understand this part a little bit.

[00:04:36]>> Yeah. So the the obvious thing here is the memory safety aspect of it. uh and that is I still think one of the key drivers. My view is that there are other memory safe languages out there but I think what Rust has really pioneered is the approach. I think the borrow checker is a very interesting way of approaching the problem and I think it forces developers to think more carefully about the safety of their applications of their code. Uh [clears throat] we at Canonicle most of our revenue is made from patching security vulnerabilities and support. So the bluntly the fewer of those we have the better. Uh I actually didn't really see this until some months after I first announced the thing.

[00:05:20]Initially it was all about how can we basically set a course for adopting more resilient software. And I see resilience at the kind of intersection of security and reliability.

[00:05:33]So with increased memory safety often you do get better security but you also get better reliability. Right? An application that panics could well be more susceptible to an attack, but it might also just be susceptible to falling over and failing, right? That that is uh that's not necessarily always a security issue, but it is a reliability issue. Uh and in certain industries, um particularly heavily regulated industries, things like automotive, industrial, those things are very important. So there is a there is a sort of safety and security angle if you would if you think about it in those terms. So I think the the safety aspect is also very interesting. Most people will traditionally associate ibuntu with servers and desktops but of course it also runs in cars. It'll run in various safety critical systems all throughout the world and fewer panics you know more resilience broadly is is a nice property.

[00:06:28]>> Yeah absolutely. And when we have a Linux distribution such as Ubuntu running on these low-level systems and when you need some kind of safety on these tools and utilities that is shipped with Ubuntu um some of them comes to mind like the these Linux specific ones such as the sudo or like the core utils and those were actually being now replaced with the rust um rust versions like the the rust equivalents.

[00:06:59]So um maybe you can give us some insight on what which tools and like which parts of Ubuntu is now using Rust and um how those were going.

[00:07:11]>> Yeah. So the ones that we've shipped kind of generally available if you will in the latest LTS are indeed the core utils and sudo RS. Those projects are quite different in how they approach their work in that the core utils project the the u utils core utils project aims to be sort of 100% bug forbug like for like compatible with the GNU alternatives which kind of makes it an easy choice in a sense right you don't break any users uh except of course we found there were differences often in kind of esoteric uses uses of those things so we actually first did this in 254 for some of the utilities and and we kind of had a couple of cycles to polish ship because we knew there would be issues and we still occasionally run into them. There was one quite recently that we ran into that stopped us from building daily ISO images for a couple of days while we worked the problem. The pseudo RS tool is quite different that does not aim to be 100% compatible with the existing utility. That is basically saying if we were to reimagine what sudo would look like in 2026 having learned 30 years of security lessons, what would it look like? Like what would be the reasonable set of features for it to have? And so we have actually broken some workflows. Um some people are angry about that but we did it at the inter you know at the boundary of an LTS version which is a major operating system version and one way of mitigating that is um a lot of the work that we did while we're not responsible for authoring these rewrites ourselves. We have funded those projects to do work for us to get them ready but we're not the essentially the owners of those projects. The work that we have done is essentially the underlying engineering to allow people to revert back to the existing utilities if they need to. So if someone desperately wants to go to 2604 but knows that there is something they use in the OG sudo that doesn't work in the new sudu they can they can swap it out. [snorts] Uh so that's what we've done. In the upcoming release the first bits of NTPDRS will land which will replace Crony. uh that won't be a complete replacement in this cycle, but by 2804 all of the time syncing will be done by NTPDRS and the work that we're funding the Trifecta Tech Foundation for is basically to get that ready to have a single a single application that can do NTP NTS and PTP client and server and be fully integrated with the distribution with App Armor and all of the various other concerns. And then there is a Greenfield project that we're running in collaboration with the Russell's project called UPKI which is all about bringing browser grade cryptography to low-level system utilities particularly around TLS. So things like certificate revocation intermediate pre-loading.

[00:09:39]There is currently no answer for that in things like curl or wget or open SSL believe it or not. Um the other major operating systems like Mac OS and Windows have had this sort of figured out for some time. Linux doesn't have an answer. So this is an answer and my hope is that that goes much beyond Ubuntu uh and lands in as many places as it possibly can.

[00:10:02]>> So as a longtime user of Arch Linux and a recent user of Ubuntu on my work laptop, I always took Crony as granted.

[00:10:13]So I am not really aware of the crit criticality of uh of crony and can you maybe explain like why npdrs is such an important like uh rust replacement for for crony and what are you like planning to do? How is this going to affect as as a regular user or like as someone using at a larger scale? Uh how is this important? So the impact in my view is at its most critical in real time safety [clears throat and snorts] critical systems. So if you think rolling you know running some microcontroller on a plane in a car in a robot that has scary looking tools on the end of its arm accurate timekeeping can sometimes be safety critical. It also has security connotations particularly in cryptography. if you're dealing with keys that rotate often, things like that. Uh the time syncing can be very important. It can be very important for working with positional systems. None of that is to say that Crony doesn't do a good job of syncing your time accurately. Uh where it gets complicated is in uh the implementation of PTP, which is the precision time protocol, which most people don't really have much use for, but that's where it gets extra interesting, right? So that's when you get to the very very high precision time syncing and setting that up on Linux has always been quite unfun right you have always had to use crony and Linux PTP and potentially some external time source perhaps satellite time source perhaps some dedicated you know bespoke time source and configuring all of those to talk to each other can be unfun and so uh what we are trying to do is basically condense all of the functions that you might need for any application of time syncing into one utility with one configuration file that can kind of do all of it and with the added benefit that it's very efficient and it's memory safe. Um what that will look like is I hope that we will be able to create a tool that helps users migrate configurations. Uh those tools are always complicated and they always miss edge cases. But again this will this will happen at the boundary of an LCS release, right? And so the beautiful thing about that is if you have a really complicated Crony config running on 2604, we will keep maintaining that for 15 years. You don't have to change, right? Crony will still remain in the archive. You'll still be able to get it, but we would like to switch people by default to NTPDRS.

[00:12:38]>> Yeah. So another project that you mentioned uh UPKI. Um so this is also like U stands for universal, right?

[00:12:47]First of all, >> yeah. So um it's a as far as I understood it's a universal um demon for certificate handling intermediate certificate preloading and stuff like this right can you maybe explain this a little bit because that's also very very interesting not for just Ubuntu but I think for uh other distributions as well >> other distributions and even other operating systems like it should work fine on BSD and Mac and Windows 2 so it's not a demon interesting ly.

[00:13:18][clears throat] So what this was actually in response to an LWN.NET article essentially illustrating how this piece has been missing. So a critical part of the web PKI infrastructure is the ability to issue certificates but also to revoke them.

[00:13:34]That was traditionally done with certificate revocation lists where essentially the certificate authorities that issue certificates also publish a list of certificates that are revoked.

[00:13:42]And the idea is that if your browser encounters or if your tool encounters a certificate that is revoked, it should consider it untrusted.

[00:13:49]In the interim, there is a there is a protocol called the OCSP, the online certificate.

[00:13:57]I forget exactly what it stands for, OCSP, but OCSP is going away. OCSP had that's embarrassing. OCSP had a couple of difficulties with respect to privacy in the sense that every time you requested a certificate, you would also have to hit an OCSP endpoint. uh to basically say I would like to understand the revocation status of this certificate which had the side effect of leaking all of the certificates that you were requesting right um there were some revisions to the protocol that fixed that or or rather mitigated it but ultimately OCSP was basically sunsetted and in the interim this article came out to say Linux doesn't do this the browsers do this in isolation so Firefox has an implementation chromium has an implementation but curl liss cell would get all the stuff that you use at the low level just doesn't pay any attention to CRLs at all.

[00:14:48]So along the journey to Firefox doing this themselves they Mozilla developed quite an interesting storage mechanism called CR Lite which is for storing revocation data in a binary format that is very very efficient. So essentially they publish a list a sort of you download a chunk of CL light data and then each day you can kind of fetch just a delta which is very very bandwidth efficient. And so what UPKI does is it runs depending on how the distribution packages it. It might run once a day, twice a day, once a week and it essentially just goes and fetches the latest CL light data from a mirroring service that we run and caches it on the file system. Then the additional work we have to do is essentially upstream in projects like curl and libsl and gutls to hook into those libraries and projects at the point where they are checking certificates to also check whether the certificate is revoked. And thankfully lib SSL has this kind of callbacks mechanism. And so part of the UPKI project is also to land the upstream contribution to all of these SSL libraries and important projects in the WebPKI ecosystem so that they can use UPKI if it exists. Down the line we may also introduce support for intermediate preloading. So that is where if you talk to a website and you request it certificate rather than getting the full chain you just get the last certificate uh and the route maybe uh a lot of a surprising number of websites do this and it's technically incorrect but browsers work around this with a thing called intermediate pre-loading where they can say okay well I know the leaf certificate I know the route and so I can fill in the gaps that is a capability we can put into UPKI and it may also serve as a nice proving ground or launching pad essentially for things like Merkel tree certificates which would be a next generation cryptography uh implementation essentially for the keys themselves and there are a couple other things certificate transparency enforcement for example is another thing it could do but right now we're focused on launching essentially 1.0 for certificate revocation and that should be in Auntu 2610 which comes out in October.

[00:16:55]>> That's really exciting. It's great to hear that like these things that you are working on and like these rust tools are now also like they're available for other distributions and operating systems as well. I am uh wondering like is is everything uh like the pseudo rs gaugori tools npdrs uh those tools are now like available for like other distros as well right these are not like Ubuntu specific because I remember >> they never they never were Ubuntu specific so I want to be really clear like we did not do these rewrites the uutils core util stuff existed before us sudo rs existed before well rather existed before we took interest so um they are I think available For some other dros they're packaged variously like Nixos allows you to use the core utils and pseudo RS stuff. I think it's packaged in fedora some of it. Uh what >> I actually package it for arch. Yeah it's package. Yes.

[00:17:51]>> Nice. So the the big thing that we did I think is we were the first major distribution to just say we're going to make it the default and we're going to fund the work needed to make that happen. Uh so hopefully that has made it easier for other distributions to make the same step.

[00:18:04]>> Yeah. The reason why I asked that because I once saw your blog post about a tool called Oxidizer I think.

[00:18:11]>> Yeah.

[00:18:11]>> Right. And this this was basically just for making the adoption easier. Right.

[00:18:16]So um if you know so so Ubuntu basically accepted those tools and started adopting them. They were already existed but now Ubuntu is just shipping them as part of the default uh version like this. True.

[00:18:32]>> Yeah.

[00:18:33]>> Yeah. Um, and Oxidizer was a bit of a it was just a bit of fun in the sense that when I said that we were going to do this, Oxidizer provided a nice little tool someone could run and get kind of a preview. The packages already existed, they were a bit broken. Some things didn't work, but it was kind of a way of saying to people like, "Look, if you have a like a place where you experiment, go try it out and tell us if something breaks, right? Like tell us if we're going to break you and then we can factor that in as we as we roll the default change out."

[00:18:57]>> Yeah.

[00:18:58]So I want to like ask why change these decade decade old Linux tools and you know not continue maintain these implementations but there's actually someone in the chat that asked uh a spicier question. I want to read that.

[00:19:16]How do you reply to those who say Rust replacements of core functionality are not ready and buggy? And I think this is kind of uh related to what I wanted to ask because there is some legacy tools they written in C they have been you know part of these distros for a while and there's now the rust version for those tools. So um there are like two paths here. Either you pick these you know legacy tools maintaining keep maintaining them or like you you pick the like you you rewrite them in rust and each one has its own like advantages and disadvantages. I think how do you see this distinction and how do you reply to uh this question?

[00:19:59]>> Yeah I get asked this quite a lot. Um in [clears throat] reality GNU core utils has been around for 30 years. It's great software like we all rely on it. I have nothing against it. I'm very grateful for the people that maintained it, but it still has bugs. Do you know what I mean? So the argument that the Rust one has bugs is not really a valid argument.

[00:20:16]It also still sometimes gets memory safety bugs filed against it. Very rarely, right? Like this is this software has had lots of eyes on it. And my point is that I expect Abuntu to be successful for the next 20 years. And I would like to take as much of the burden the you know if we can remove the lowhanging fruit to make the maintenance easier and the platform more secure and more resilient by default then we should do that and in reality these projects never get to be polished if no one uses them. So the GNU core utils project sorry the u utils core utils project the rust rewrite was already at some very high 80s percentage of tests passing against the upstream GNU test suite.

[00:20:59]It's now into the 90s and it's into the 90s because many thousands more people have used it. We found the bugs and we funded funded the developer to fix them.

[00:21:06]And so [sighs and gasps] yes could be rough for some people but essentially it was a calculated risk. We knew that there would be some edge cases that would break, but we were kind of willing to progress that agenda because in the long run, what it will mean is more of the base Linux system is sort of preh hardened against an entire category of attack. And there are actually some places where we can achieve better performance because of some of the primitives that Rust offers. So there are some parts of the GNU core utils which are super duper optimized and there's bits of assembly in there and like it's all very clever but there are and and so there are some of the GNU things that are slightly faster than the utils ones but actually the response I've seen in the Uutills community is to go chase that is to go and make the there's one interesting example where I ran a bunch of benchmarks synthetic benchmarks which proved that the word count utility for whatever reason was 11% slower than the GNU one. It's now 9% faster in most use cases because somebody saw that, thought it was interesting, and went and fixed it. And I think that optimization would obviously have been possible in GNU core utils, but I think Rust makes that kind of optimization more accessible to more developers. And there are more developers, particularly junior developers, who are interested in writing and contributing to Rust projects. And I want to attract that crowd. I can't just rely on all of the people that have been doing it for the last 30 years because what if they want to retire or what if they just aren't interested anymore? Like we have to appeal to the set of people who are interested in doing open source work and if all of if more of them are available in Rust, I'd like to tap into that.

[00:22:43]>> Yeah, absolutely. So last year I did something very similar to Rust in Ubuntu. So I'm part of the Arch Linux distribution and we had this really old like package management system and I was a part of project called ALPM Arch Linux package manager and we basically re rewrote bunch of parts of the Pac-Man package manager in Rust. small utilities, parsers, um some other stuff such as the package build like uh parser like data database handling, file handling and so on. And what we realized was more that we kind of peak into these like old formats, these like old legacy files, we f we found bugs and we found like things that really didn't make sense. So in your case, do you have some example of this with let's say corotils or or pseudo where there's some behavior that that was there for a long time and people just took it for granted because it was there but you look at it from a new perspective and you realize well this doesn't make sense. Let's just change this.

[00:23:58]>> So I do have examples but it's it again it comes back to how the two projects handle it. So what I will say is one of the things I really love about the UYU tools project and the pseudo RS project is that both of them are in regular contact and contribute patches to the original projects. So when we find things uh when [clears throat] Sylvest finds things in the core utils that are broken in upstream core utils, he'll go fix it there as well. Uh when the pseudo RS folks find a security issue in pseudo RS that also affects the original sudo, they collaborate and make sure they get fixed at the same time. Um, interestingly, I think Sylvest was in something like the top 10 contributors of the latest GNU core utils release, even though he's the project lead of UTS. And I love that like this I hope hopefully that illustrates to anybody who's skeptical that this is being done in good faith, right? It's not about winning and losing. It's just some people working on some software. But the two projects handle it differently. GNU utils and u Ututils are supposed to be 100% compatible. We had something recently where I with the help of a friend we we were troubleshooting a broken snap. It was for Obsidian and Obsidian packaged this old script from this like weird dependency buried somewhere down the tree. And it turns out that the head command when you read when you tell head to read a number of bytes, uh if you tried to read say five bytes or n bytes in a file that had no bytes, the GNU call utils return zero, which is weird because in reality it should return an error. Couldn't return any bytes. And that's exactly what the UU tools project did and that's an incompatibility. So the fixes of course we then change the maintainer then changed the new implementation to make it behave the same as the old one even though it was kind of counterintuitive.

[00:25:38]Whereas in pseudo RS sometimes they do make breaking changes and say this is a this is a departure from sudo and this is the reason why like we believe that this is a you know better for security or or whatever it might be. An example of that which lots of people lost their mind over was that when you type your pseudo password in the terminal, you now get the little asteris marks by default.

[00:25:56]You get some feedback in the terminal.

[00:25:58]Lots of people, far more people than I realized have opinions about that. Uh [clears throat] [sighs] >> that's great actually. I I would love to have that. Um it was always a bit confusing to not see what I'm typing.

[00:26:11]So, uh, when it comes to the the reaction to to what you're doing, was there like any resistance internally or externally? And how did you really react to that?

[00:26:26]>> Yeah, there was uh not so much internally. I think, you know, people came to me with questions and concerns and I think I comforted them somewhat by saying, "Look, I really want to try this and I really believe it's the right thing, but I'm also not going to destroy Abuntu's reputation just to be right."

[00:26:41]So, we had lots of points along the development cycle of the 2604 LTS where we kind of had to make a decision and go, "Oh, we've broken a bunch of stuff.

[00:26:49]Do we still want to go for it?" Uh, and and every time those are just you have to make an informed decision and it is taking a gamble, of course, it is an educated one, but we are taking a gamble. The biggest concern I think outside was about licensing uh >> and I was a little confused by that uh or I guess not confused been around the open source community long enough to know that that's important and the big change is that GNU core utils is of course GPL whereas the Uls project is MIT licensed and lots of people found that to be unacceptable and my response is honestly uh I think the GPL is incredibly important important and I think it has hugely contributed to the success and proliferation of Linux and open source and in fact most of Canonical's projects use derivations of the GPL license themselves but in this case this is not code that we are writing we are being a distribution we are picking someone else's project who you know whose values we choose or we believe align with ours and we are shipping it if the values of that project cease to be aligned with ours we will ship something else So in some senses and there was a there was this kind of undertone of well Canonicle is doing this so it can make a pile of money out of people to which my response has always been if you can think of a way for Canonicle to make money out of the CP command I'd love to hear it. Do you know what I mean? Like it it's just not on the agenda at the end of the day. It's one of those changes where I just felt like the license of the project that we were distributing to our users in this case was irrelevant. [clears throat] uh we have already proven I think that we are invested in the long term in contributing our changes upstream and making them available. Not only did we fund a whole bunch of the development work in the project but we also have our own engineers contributing patches in the open and would encourage others to do separately. So I in a sense I think it is good that people always bring the license question up but I think there is nuance in any decision like that and in this case I just I don't see a way that this could backfire for the community ultimately. Uh >> I was in similar situations myself. Um Rust ecosystem usually prefers MIT plus Apache 2 and I would say I don't want to generalize it but usually the Linux ecosystem is GPL 2 GPL 3 sometimes. So uh what are the licenses that you're using right now in these projects? I know that these are like separate repos managed by like separate people but uh like generally do you have let's say GPL code at all or do you all go for MIT any like >> internally internally most of Canonical's projects are GPL or AGPL with some exceptions so we use AGPL in kind of networkbased services like Lex uh guju and such uh [clears throat] the only places we really use the Apache license for example is in ecosystems where we expect so so for example we in my first four years at conical I spent it building operators cloudnative operators and that community almost exclusively uses Apache licensing and so our operators are nearly all Apache licensed but nearly nearly all of the software that canonical creates is some derivation of the GPL and I I don't expect that to change like this it is not a policy shift it is not a bid to get out of Linux it is not a bid to get away from GPL. It is just not a very interesting fact in this part of the in this particular decision in my view. I I I think it would be it would signal or rather I could see why people would think it would signal something bad if Cononicle was suddenly changing all of its licenses and abandoning the GPL.

[00:30:35]It's absolutely not what we're doing. We are just swapping a utility. And in reality, there's loads of software in the archive that isn't GPL license, right? There's BSD licensed stuff and MIT licensed stuff. There's proprietary stuff in Multiverse. It all comes down to which projects we think are going to serve our users best ultimately.

[00:30:52]>> Mhm. Yeah, that's usually the the best way to go with it. I think the the license dispute is is like all all the time it's there. Uh especially when something is open source, but um yeah, ultimately we should really just consider what we're shipping. So, we have one question from the chat. Uh Giorgio asks, "How do you think Canonicle's direction will influence other distribution actions? Do you think it will induce more Rust adoption elsewhere?"

[00:31:23]>> I hope so. I think it already has. Uh I saw a Fedora change proposal to adopt Pseudo RS by default a few months after we did. Um Nexos is traditionally more forwardleaning here. Um and I think they certainly will. My hope is that we are able to use our quite privileged position that we have earned over the years to make this easy for other distributions. Like this is another sort of sorry to come back to the licensing.

[00:31:48]It's another thing where I don't see this as some huge competitive advantage that we have a rust CP command. Do you know what I mean? Like it it just isn't a defining feature of the platform. And in reality, the whole Linux ecosystem will benefit if more people adopt this, if more distributions adopt this. I fully expect them to honestly because I don't see the reason not to. Um I don't judge that those that won't, but yes, I I expect it will trickle out in other distributions as well.

[00:32:16]>> I'm wondering if there's any lessons that you want to share in terms of the difficulties that you experienced like adopting Rust at at this scale. So um what's what was the like the hardest thing that you you encountered while adopting Rust?

[00:32:35]And in our case, I think some of the things were not because Rust. Some of the things were just because scale.

[00:32:41]Replacing the 60 or 70 most typed commands in Linux with an alternate implementation turned up some bugs.

[00:32:48]Matias Endler did some great analysis on this. He wrote a blog about one of the particular bugs that blocked us for a while about how, you know, while you do get memory safety from Rust, that doesn't stop you from having bugs in the code. And he did a really nice kind of opinion piece on it. We packaging is hard particularly in so modern languages that use package managers that speak to crates.io. We actually take a slightly different approach to how that's done.

[00:33:11]The Debian Debian tends towards creating a deb for every crate and landing that in the Debian archive. We don't do that.

[00:33:18]We do a vendoring thing a bit like Nyx where we essentially take all of the crates, dump them in a tarable and ship them as part of the the finished artifact. So that's been complicated.

[00:33:28]Again, back to licensing. Part of that is like you have to have a copyright file for every sit like Debian policy requires that you have a copyright file for every open source project that you you ship. And so we had to build some tooling around that. Um but honestly it hasn't you know aside from working through the backlog of bugs and features and things like that it hasn't been very painful. I was a little bit worried that we had put too much upon one of the maintainers at one point you know with uh while we committed funding and the funding that we committed was based on what the maintainers asked for. So before we started any of this or did anything public, you know, I went and spoke to the maintainers of the project and said, "Hey, I'm thinking about shipping your thing by default. What do you think? Is it ready? What would it take for us to support you?" And broadly what they asked for, we gave them in terms of funding and and support. But as we neared 2604, as you might imagine, lots of bugs, like we have lots of users, and so there was a point where I thought we had overcooked it slightly. uh and I would like to maybe think that through a bit better in future so we don't you know end up burying projects in bugs uh I things are stable and we work through it but ultimately I think there was a stressful few weeks for a couple of folks on those projects just as the adoption kind of increased and the we one of the things we did was funded two independent security audits on the core utils projects and we didn't for sudu because they'd already had a couple of formal security audits and we were content our internal security team looked at it and agreed that it was good for core utils they hadn't been a full security audit. So, we funded one round, fixed a bunch of issues, then funded another round and fixed a bunch of issues. And again, that just created lots of issues, right? Like lots of work to be done. And sometimes funding isn't all of it if you don't have the people to distribute the funding to to share the work, right?

[00:35:11]So, >> yeah, I definitely see that like funding open source while while doing this is is really valuable, but open source is open source, you know. So you just you just uh sometimes get buried in in bugs but it is what it is.

[00:35:30]>> Mhm.

[00:35:30]>> So uh you mentioned crates.io and packaging. How do you like handle packaging in in Ubuntu like and how do you see the crates IO ecosystem? What do you think about it?

[00:35:44]>> I think it's challenging. I think it has the same challenges realistically as npm and as pipi. like they're all they all suffer from a similar problem in that you are essentially allowing people to upload untrusted code and so part of the reason we announced some months ago that we joined the Russ Foundation and part of that was to be a part of the conversation on create.io security, supply chain security, what can be done at the infrastructure level to kind of shore that up and make sure that it we don't end up the butt of you know the left pad meme which I'm sure you remember from some years ago in the JavaScript ecosystem in reality while it's all very funny to laugh um that could easily have been Rust the the two things the two systems are just not that different we vendor like I said before we vendor things in devs so we sort of have like parallel universes in a sense and we have what's in the archive where we package for each Abuntu release a given version of the Rust compiler uh and all those pieces and the Rust crates that make up the packages in Auntu. But then we have the snap ecosystem and the the container image ecosystem where we can ship different versions of things. Uh and [sighs] so [gasps] while snaps people are varying opinions about snaps they're built the build system is generalized. So it's a what we call we have a team called the Starcraft team which isn't anywhere near as cool as it sounds. It's a it's a packaging tools team. But the idea behind that is we have a single grammar that represents how to build software and ship it in snaps, in container images, in VM images, in whatever format you need. And so a lot of our work on how to package something, if you think about for example the package build format for Arch Linux in your world or uh there's a team that handles this in Nixos, like what are the set of helpers that allow you to specify a Nix derivation with the right cargo hash and things like that.

[00:37:29]We have a similar set of things going on in our build system sort of tooling team. Uh and their view they they are looking at how can we build a common grammar for building software irrespective of what the output type is.

[00:37:42]Noting that most of the time you are still cloning some code building the code getting a binary and then shipping it in some container format whether that's a tarball or a deb or a OCI image or whatever.

[00:37:57]Do you like do anything extra to like give some insurance for the Rust packages like because [clears throat] you mentioned crates IO and it's it's very similar to the to the other like packaging ecosystems. It's vulnerable in the same ways. So do you like do some like extra things?

[00:38:21]>> Um I don't know sbomb generation.

[00:38:23]>> Yeah. some uh checking hashes or reproducible builds maybe um can you maybe talk about that a little?

[00:38:31]>> We don't have a really strong reproducible builds story for the archive in the traditional sense. Uh packages are all built in you know offline ephemeral instances that can only you that can only pull from known sources.

[00:38:45]So they are uh not reproducible in the kind of like reproducible build brand sense but uh they are deterministic for sure. And from an sbond perspective we have just in this cycle implemented cargo auditable for all of the deb all of the software packaged as devs in the archive which means you essentially get like a little embedded sbomb of all the crates and versions that we used to build that software. So that change happened a couple of months ago in the archive which means every time a package gets rebuilt now it will get or rest package gets rebuilt cargo will be part of that >> that's pretty good like um I was not aware of that we should probably do the same for our Linux as well cargo autoable is just really really nice tool nice so u there's one question uh do you see rust replacing libraries called by and called by C binaries.

[00:39:43]Do you mean like shared libraries maybe?

[00:39:46]I don't know.

[00:39:48]>> I mean, I think Rust has a great Rust has a pretty good FFI story. Um, and in fact, we as part of the UPKI project have created a C library that people can call into as well. uh I don't fully understand the question but I think if the question is do I see foundational libraries that are written in C or other languages today being rewritten in Rust I think the answer is inevitably yes some of them right uh we've seen this uh a lot in the Python ecosystem so Python cryptography for example paidantic which is a scheming tool they're actually all just Rust under the hood makes perfect sense like why not have Rust back your cryptographic primitives and have a very nice way to call into it from Python and I think it's all just about using the right tools for the job at the end of the day.

[00:40:34]>> There's also one additional point uh which is AI is getting to a point where things are getting easier. So do you see like any like any possibility of using LLM to do C to Rust migrations in the future maybe?

[00:40:53]>> Yeah, I think so. I think Rust is really nicely positioned for the agentic era mostly because the compiler can be so complainy, right? Like it it almost it's a it's a free set of guardrails for your agent in a sense. The compiler isn't going to isn't going to let the agent make sloppy memory safety mistakes. So I think my experience of using Rust with agents has been very positive uh on the kind of mechanical rewrites. I think the answer is yes. I mean you may well have followed the story about bun the javascript runtime that was rewritten like a million lines of zig to rust and there was a great write up about that uh and my understanding is the core code is now shipping that runtime. So this is in the space of like a couple of weeks essentially one guy and a pile of agents rewrote a million lines of Zigg into Rust and then shipped it into claw code to be used by I don't know how many users lots right uh and sure the to start with I I saw lots of commentary saying oh my god well the first pass was horrible the rust code had loads of unsafe and it but if you read about how it was done they essentially did a mechanical file by file rewrite which necessitated a bunch of unsafe and scariness but over the course of a couple of weeks with some rules and bots of iteration, it's turned into a pretty well tested RS codebase that has all the properties of a Rust code basease you'd expect. And I think we're actually working with a university here in the UK on a project which is going to start shortly which is all about and I'll I'll post more details on this when we announce it formally, but essentially pairing LLMs with formal methods.

[00:42:26]So have LM rewrite existing C code into Rust but use formal methods to essentially prove that the software behaves the same um ideally without the abs you know with the absence of certain classes of bug. So I think it will happen. I think we've already seen it.

[00:42:42]We've seen a couple of massive rewrites.

[00:42:44]There's a there's a bunch of folks doing a Postgress rewrite in Rust which has been gaining some traction recently. Uh, I think lots of them are just experiments and I don't all I don't think we should just, you know, everyone should just down tools and rewrite everything in Rust with with LLMs because we can. But I think it puts it back on the table for a bunch of projects who have got to a point where that would have just been infeasible before, infeasibly expensive.

[00:43:07]It's now an option to them if it is done with discipline and planning and care ultimately.

[00:43:15]Do you are you like a do you use like these um like LLM approaches as part of the Rust rewrites right now or not really like is this something new to you like new to these projects because I know that they existed for a long time um like especially the the core utils one is it existed before this agentic era so I'm not sure if they actually use it but do you use with with the other like rewrites or like the new implementations.

[00:43:47]>> I mean, we use it a lot in Canon. We're like everybody else still probably at the start of a very long journey. Uh, and my I wrote about this I wrote an article for anyone who cares called the future of AI in Auntu where I talked a bit about Abuntu and AI and what's going to happen from a product perspective but also just outline our approach on it in terms of how we incentivize people to use AI. We are not going down the like measuring people by token spend token maxing madness hopefully for all of the obvious reasons which is that that's a terrible idea and does not motivate good outcomes but we are trying to equip our engineers with the best tools that we can and encourage them to basically become professionals at using them. I am incredibly optimistic about the future of engineering with agents. But that doesn't mean that I think you can just one-shot all the things and expect good outcome. I think there is still a huge need for engineering judgment, right?

[00:44:43]And also understanding the problem you're solving in the first place. I think uh much more now a good software engineer is going to be someone or an excellent software engineer is going to be someone who is more at the intersection of like grassroots engineering skills and product because writing code is going to become free in quotes. It's not free. You have to pay for inference and all those things. But it's certainly much less expensive than it used to be hiring a team of humans to do it. So given the right product brief and the right engineering rigor in structuring something, writing code is easy now.

[00:45:15]But you still have to come up with the right problem to solve and the right structure and all those things. Right?

[00:45:19]So I I think people who perhaps have done less thinking on the product side and have just been traditional kind of programmers rather than engineers will really struggle in the agentic era.

[00:45:29]Whereas those who have really gone deep on architecture and engineering principles will just see it as a big enabler for them a kind of force multiplier.

[00:45:40]There's one comment in the in the chat that is about it's a bit it's a bit a bit of a negative one. Um he says, "AI took my job and I'm unemployed. Not even Rust can save me." So do you have any um any tips for for these like new new Rust developers that is just came to this like agentic era and then they want to like use Rust and they're a bit like let's say pessimistic about the future of programming. Do you have any kind of like starter tips or any anything that you want to say to them actually? Yeah, I often hear people say things like um LLMs are going to ruin learning.

[00:46:26]I have had the opposite experience. I think you could use LLMs in such a way that you never learn anything if you just ask it to do your job for you. But I've built a couple of things with with LLM's. Uh one of which I think is probably one of the best pieces of engineering I've ever done. And it used a whole bunch of things which I've talked about and heard talked about for years but never done myself. and I use the LLM as a kind of, you know, encyclopedia partner to kind of navigate through those decisions and and learn a bunch about the trade-offs along the way. I think for people who are worried about losing their job or who have lost their job, I guess the question you it obviously always needs more context, but I would say focus on the core engineering skills. At the end of the day, the LLM is only going to ask you to write what you ask it to write.

[00:47:10][clears throat] Sometimes they go off and do things you didn't ask them to do, but the point is you have to be able to spot that, too. And so I still believe very firmly in understanding engineering principles like all of those data structures and algorithms classes and distributed systems classes that people take they're still important if in fact if anything they're more important than ever right and so having a firm understanding of the seams of software the contracts the interfaces command lines APIs all of that stuff at least with today's models in my mind still requires a bunch of human judgment and so I think that's the place to upskill It's you think of it as you you can have the LLM do the painting by numbers but you still have to get the outline in place right um >> yeah I totally agree with that I mean I think now with the with the acceleration of of these LLMs and now we use agentic programming especially in Rust I think the the the having the knowledge of Rust concepts like deeper understanding is is way more important than before because now you can write rust using LLMs but you cannot really judge the code if you don't know how the borrow borrow checker works how the lifetimes work and and so on so I think um you know having the the base understanding and u the knowledge is is way more important now >> I definitely agree with that all right so um I want to ask about like what are you funding right now and what is prioritized next in in Canonicle and what should we expect from Ubuntu going from now [snorts] so the mo the main piece of funded work we're doing at the moment is with the trifect foundation on NTPDRS that work is just starting we've been talking about it for a few months and you know the the majority of that work will start happening over the coming weeks upki um I think we're pretty much at the end of that first round of funding we would need to have you know more conversation with the Russell project about where we want to go. There's been some interest from another distribution um at least passing interest on GitHub discussions and things. We joined the Russ Foundation as gold members, which is a kind of upfront, you know, contribution of $150,000 in general to support the Russ project. Uh, and as a part of that, I get to attend the board meetings and have opinions, which I think which I'm trying to use to steer it towards the things that Colonel cares about, of course. And we also joined the Trifecta Tech Foundation as gold members, which is a kind of 40,000 euros a year commitment basically to help them with the increased maintenance load that they will undoubtedly experience as a result of many thousands more people using their stuff because we turned it on by default. So that's that's kind of see it as a bit of a almost like a portfolio of investments across the ecosystem. And that doesn't have to be all. That's just the set of ideas that we've committed to for this cycle. Um I expect there'll be more. We what the next the next big thing as a category that we'll look into I think is compression algorithms >> particularly particularly ones where we can get an efficiency [clears throat] boost because if you think at the scale of Abuntu if we could make a very common compression algorithm 3 or 4% more efficient that could actually have a meaningful impact on power consumption worldwide and so you end up with this like lovely side effect of just making the software faster and nicer to use but also having a kind of wider effect. So, we're looking at shipping BZ Bzip 2 in Rust by default this cycle and we're investigating Z standard RS zib uh and uh there's another one which I can't quite remember right now but yeah so that's the next kind of category and there there's some interesting trade-offs there. So one of them I think is Z standard is safer and does catch a certain kind of unsafe operation that the current library will allow but it's also slower and you could turn off the safety checks and then it goes faster but do we want to turn off the safety checks and so we kind of have to make some decisions about what the set of defaults are that we present to our users.

[00:51:19]So you said uh Zlib and Bzip 2, right?

[00:51:23]These >> and Z standard RS as well. Yeah, >> Z standard. Okay. And is were these like existing uh Rust projects before or you just Okay. You basically funding these projects.

[00:51:35]>> We haven't Canonical is not responsible for any of the rewrites in terms of actually taking the upstream project and rewriting it in this kind of campaign.

[00:51:43]So far up UPKI didn't exist before and we funded it, but we're not responsible for much of the code. We're working with the Russell maintainers who are writing that code.

[00:51:50]>> Yeah.

[00:51:50]>> Mhm. Okay. Got it. So packaging is is the next focus for Rust >> and we do write Rust code. We have to write we're writing we're doing some work on boot at the moment. Uh there's some work on FPGAAS and controlling them from Linux. There's there's a compositor that is getting some Rust work. It's not that we don't write Rust, it's that the these kind of big rewrites, the things that are making the headlines, they're not they're not our Rust code. We are playing the role of the distribution, which is to integrate them into Auntu and ship them.

[00:52:23]>> Mhm.

[00:52:25]So when it comes to like shipping apps in Ubuntu, snaps really come to mind because they're like very closely associated with Ubuntu. Do you see some some rust happening there or like how do you see it like the the general snap ecosystem right now?

[00:52:44]>> I mean snaps do work on other dros. I have personally run them for years on arch linux. I was an arch user for 10 years or so when I used to run snaps on there. Uh snaps can also run on suss.

[00:52:55]They can run on nyxos through a very very creative approach. um we actually employ people whose job it is specifically to make snaps and snapd run properly on other distributions. Um that said clearly mostly it's Ubuntu and I think there are a lot of mixed opinions out there about snaps and some of them are totally valid. I think canonical maybe you know years ago maybe went a bit soon on the graphical snaps and it was a bit bumpy for people and it left a bad taste. But there really is a category of software uh for which snaps are just a match made in heaven. Particularly low-level kind of systems agents, observability agents, those kind of things. It's great for browsers these days with the kind of increased sandboxing capability. It also allows if you think about the challenge that we have, which is that we release a long-term support distribution every two years that is going to be supported for 15 years. Like sometimes backporting things and making stuff work on old versions can be really hard. And because Snaps bring with them a runtime, it gives us a bit of a superpower and we can bring very very modern capabilities to very old Linuxes very easily because snaps. And so they they solve a bunch of different problems. It's been bumpy along the way, but it is a very interesting format. The build language like the way the the Snapcraft language, which is now portable across Rockcraft and Charmcraft and others, I think is laid out very nicely. And if you are interested in contributing to iuntu, that is a way to contribute to ibuntu.

[00:54:23]Like getting a snap in the snap store is contributing to ibuntu. Like you don't have to have us in the loop to have something that anybody on the other planet can install from the ibuntu app center. Whereas if you want to land something in the archive, you kind of do have to work with the ibuntu community, right? And have it vetted and all those kind of things. Um and it allows you, for example, Jet Brains. Jet Brains distributes its IDEs as snaps for iuntu.

[00:54:47]And it does that because it allows Jet Brains to take control of the release life cycle. It allows them to ship one version of for example Rust Rover that works on 2604, 2404, 2204 it as a mechanism to enable software vendors to distribute software on the world's most deployed platform. It works very well. it, but because there is security confinement, it's sometimes hard. And I don't know that we're going to be able to make that easy always.

[00:55:17]Like, it's just hard, right? Like there's always a trade-off when it comes to security. And so, there's definitely engineering work that we can do to make it better.

[00:55:24]Uh, but I do believe quite strongly in the packaging format. I like it's a strange thing to have to get religious over, but I do think it is a it's an interesting set of technologies that solve a set of problems that not many people have that it is a set of problems that we have.

[00:55:39]>> Yeah. I created one snap package I think and I got stuck because I needed some permissions to read the the kernel state or something and it was not working for me because I couldn't you know get approved or some some like permission issues. So I think that was the the most difficult thing for me as well. And um yeah but overall I think snaps are a great technology. Um, >> we're doing some work on that process at the moment. Um, it is possible to get snapped with elevated privileges, but we don't just hand that out to anybody.

[00:56:15]Like if someone shows up to our forum for the first time ever and says, hey, can I have classic consignment for this confinement for this thing that I just vioded?

[00:56:22]The answer is probably no because what you're basically saying is, hey, can you give me root on every Iuntu machine that installs this thing?

[00:56:29]Like I don't know that people would necessarily want us to say yes to that.

[00:56:32]Do you see what I mean? Whereas if it you know if Jetrains comes out with another IDE for another new lo new new madeup programming language and they come to us and say hey we jet brains would like to distribute our next IDE on a bloom and we have classic confinement because we need it for these ex for these reasons the answer is going to be yes it's just we have to the consideration is always could this application be confined and if it can't be confined and we are going to give it those permissions can we establish a level of trust with the vendor of this software like we'll have to look at it.

[00:57:03]Is it asking for those permissions legitimately or is it something malicious? Right. It's just a process of establishing trust with the vendor essentially.

[00:57:11]>> Yeah. There's one question. Is it an accepted practice to release Rustbased applications as snaps on Ubuntu?

[00:57:19]>> Yeah, for sure. Yeah, I've released it for you myself. Yeah, absolutely.

[00:57:23]>> Mhm. Yeah.

[00:57:26]>> There is uh one Rust specific question.

[00:57:29]And how does Ubuntu deal with Rust libraries that have high MSRVs?

[00:57:35]>> Uh so that is another place where snaps are great because you can install a snap with a core 26 base which is based on the Abuntu 2604 and you can install that snap on any version of Abuntu even if it's very old. And so that's how it so each it you know for each Auntu release you've kind of got the set of rust versions that are in the distribution in the archive you know apt install rust C will get you a version and that's the version you will get for the life of that version of Abuntu and so you're kind of pinned essentially to an MSRV for landing something in the DRO. Uh but snaps allow you to kind of circumvent that. So you could, you know, let's say the MSRV climbs dramatically in 10 years time, you would be able to take that snap from 10 years in the future and run it on today's Abuntu because it's a snap and the snap brings with it the modern runtime.

[00:58:22]>> Mhm. Yeah, that's pretty cool.

[00:58:25]So I think that's it from my side and thank you John for all the answers and you know just taking ownership in the in the Rust ecosystem in in Ubuntu taking ownership but no pun intended. Um but uh thank you so much. Do you have any like additional points or anything you want to share with with us before we sign off?

[00:58:50]>> Uh I do not. For those of you who want to come meet me, I'm at Ruscom in Montreal. I'll be speaking on the first day doing the keynote. Uh so come and say hi. If you want to work with us, we're hiring across a bunch of roles remotely all around the world. Other than that, thank you very much for having me and uh hope we speak again soon.

[00:59:08]>> Yeah, it was really fun. Thank you for joining our live stream.

[00:59:12]>> Thank you.

[00:59:13]>> All right, take care everyone. Goodbye.