cuda-oxide: Bringing CUDA to Rust with a custom rustc backend

[00:00:06]All right. So, uh just to show of hands, how many of you have heard of >> [snorts] >> CUDA Oxide, or rather looked it up?

[00:00:17]There's a bunch. Okay.

[00:00:18]All right, then uh how many of you have uh sort of worked with Rust before?

[00:00:24]Forget about GPUs, just Rust.

[00:00:29]Right. So, I just want to like sort of fine-tune what I'm going to be talking about.

[00:00:34]All right. So, to get started, um I'm going to be talking about CUDA Oxide. It's a project that's pretty near and dear to me, cuz you know, pretty much wrote it.

[00:00:46]Uh The idea is today, when you want to write a CUDA kernel, which is a piece of code that runs on the GPU, specifically an Nvidia GPU, there's um a bunch of options, but the most important option that you would normally have to reach for is CUDA, which is uh a dialect of C++, called CUDA C++, right? Now, with a lot of system software moving to Rust, uh be it, you know, trying to build parts of a browser, trying to bake it into the Linux kernel, trying to run hypervisors at cloud scale at AWS, or you know, in various other parts, where they're talking about um large-scale production systems, or we're talking about web application frameworks and platforms, etc. Um a lot of stuff is actually being written in Rust, and the more controversial, being rewritten in Rust.

[00:01:48]I'm not going to be talk I'm not going to talk about the second part. I'll just talk about the first part.

[00:01:53]So, today's uh world, if you were actually writing a GPU kernel.

[00:01:58]You can go ahead and do this from Rust.

[00:02:00]You can write, you know, your application, which is code that runs on the GPU, sorry, on the CPU, in Rust.

[00:02:07]Uh, but when you want to be able to, you know, get accelerate some piece of code. It could be anything, uh, that runs on a GPU specifically, what we'd call an accelerated workload, uh, you would normally have to reach for another language, and that language is usually C++.

[00:02:27]So, there's a barrier between the two, and that's what I call a seam.

[00:02:32]What happens in that space is very, very interesting.

[00:02:36]So, uh, since some of you have actually written Rust, you probably already know that, you know, you get a strong type system.

[00:02:43]You get, um, you get an ID that helps you really understand how your code is laid out, etc. So, a lot of type inference is available to you because of the strong strong type system.

[00:02:55]And then, you also need to, let's say you're sharing a bunch of different, uh, what you call aggregates or types between the host and the device. And when I say device, I mean the GPU.

[00:03:07]When you're trying to share some code between the two, you would normally have to play a few tricks because the host has a different memory layout, uh, and the device actually is different. You'd have to sort of merge the two. That's another issue. All these little things start to show up when you talk about writing GPU kernels today in a language like Rust. And the reason why you'd want to do that is because you've got all the system software that you've already written in Rust, but the moment you need to accelerate it, you have to drop down into a different world altogether.

[00:03:43]And then when you drop down into that different world, you also pull in a lot of other things, like the entire build system, the entire, uh, toolchain.

[00:03:53]All of these things become friction at when you think about it at scale.

[00:03:58]So, that's pretty much, you know, the current state where we are.

[00:04:04]And the other thing that you also might notice is if you're a Rust programmer today, the mental model of switching from Rust to C++ and back is quite is quite a cognitive load if you talk about it on a not just a bunch of people, but on a production scale. That does add up.

[00:04:26]So, just keep that in those things in mind.

[00:04:28]And I also want to talk about what CUDA Oxide is.

[00:04:32]So, it is not a DSL.

[00:04:35]It is not a domain-specific language.

[00:04:36]That means So, the the current method, if you were not using CUDA Oxide, there are other community-hosted projects. CUDA Oxide is not the only one. I do want to provide some credit to the community as well.

[00:04:57]But if you're not using any of these projects and you want to actually use just um you know, write your kernels in C++ and then write your application in Rust, you can go ahead and do that, but you will have to effectively use what I mentioned earlier, which is compile your C++ um kernels to PTX, which is what Nvidia's uh virtual instruction set is, and then, you know, pass that on to your application program to load it and launch it from the host side.

[00:05:27]So, you can already see if I have to do this in C++ and then load it and then launch it in a separate language, it's not a very cohesive story. There's a lot of things can go that can add to developer friction as well as, you know, production scale friction is what I call it, right?

[00:05:47]So, the the bet that I am making as well as Nvidia's making is that we want to actually invest in a in I want to be very cautious over here. Um a very good experience for Rust developers who are in the process of really writing out large-scale systems as well and would want to accelerate some workloads.

[00:06:12]CUDA Oxide specifically is not a domain-specific language in the sense that it looks like Rust, but it is not Rust.

[00:06:20]It is not one of those.

[00:06:22]Uh it is not what you'd call a macro that ultimately emits C++.

[00:06:28]It is not a new language.

[00:06:30]It is Rust that you would normally use.

[00:06:34]It's as simple as that.

[00:06:37]And it's real Rust. It's compiled by the Rust compiler itself and I that's exactly what I'm going to be talking about.

[00:06:42]I will spend some time talking about the pipe the compiler pipeline because all of you are compiler folks and uh I'm just a software person who happens to know a little more about compilers today.

[00:06:54]So, with that disclaimer, we will talk about the compiler pipeline a bit, but I do want to talk to you about how you you know we're bringing Rust uh to CUDA.

[00:07:06]So, uh you can think if you've ever written a GPU kernel, if you know what a you know a device global is, that's sort of what the the same uh experience you'd have when you write your kernels in CUDA Oxide. I will show you what they look like as well. That's the most important thing.

[00:07:25]Uh the other thing that I also want to say is CUDA Oxide supports currently uh the upstream LLVM back end. So, LLVM has an NVPTX target.

[00:07:36]It is supported. That is the main target that we support.

[00:07:39]We also support the NVVM target, which is Nvidia's proprietary back end.

[00:07:45]And then if you folks want to write a custom back end, please reach out to me.

[00:07:49]We can support it.

[00:07:51]Just saying, because there are other custom back ends that folks are working on that we can plug into.

[00:07:57]Uh so there's not just one way uh not one one project that can be used as the back end to produce the actual machine code. We can the pipeline is very very what you'd call malleable, which means you can easily hack it and plug it into multiple different back ends.

[00:08:16]All right. So this is what a typical uh you know Rust or rather CUDA Oxide kernel looks like. On the left you have your CUDA kernel.

[00:08:28]On the right you have your host um your host code. I'll just call it your host application.

[00:08:35]So on the right you write, you know, here's the buffers that I want to allocate on the device.

[00:08:41]And here's the kernel that I would like to launch. And here are the parameters that I want to provide. And then that's exactly what gets launched.

[00:08:50]Uh which is the the code on the left, which is the kernel that gets compiled to the for the GPU.

[00:08:57]All right.

[00:08:58]You should already be able to see some things over here.

[00:09:01]One is the simplest way to think about CUDA Oxide is you annotate anything with the kernel on top. I don't think I have a mouse here, but you annotate anything with the kernel macro as we call it.

[00:09:15]That becomes a GPU kernel that is to be executed on the device.

[00:09:21]Okay?

[00:09:22]That's the other thing. And then uh you also see some interesting things over here that you would normally not find in C++ CUDA C++ I mean is Rust is you know, there's a lot of talk about Rust's safety aspects.

[00:09:38]We want to really bring that to, you know, kernel authoring as well.

[00:09:43]That's one of the aims of CUDA oxide as well. Is how can you make it a performant as well as safe.

[00:09:51]Because that's been the mantra of of Rust on the CPU. Is you don't have to sacrifice off the three things. You don't have to sacrifice one of them, which is reliability, performance, and safety. You get all three with Rust. The same mantra is something that we want to bring to kernel authoring as well. And you can already start thinking about why.

[00:10:11]Because kernel authoring can reach out into different spaces.

[00:10:15]The number one use case that you currently see is LLMs, of course.

[00:10:20]You have large language model labs that are really going head-on trying to produce as many kernels for different different scenarios that they have. Have. But there's also other up-and-coming fields like startup labs that are building world models, robotics, autono- autonomous driving systems, etc. Where safety is a very critical component.

[00:10:43]So, there are there are conversations to be had. I'm not saying that we're all there. You can start using CUDA oxide.

[00:10:48]It's completely safe.

[00:10:50]I just want to be up front and very clear.

[00:10:53]There is some exploration going on.

[00:10:54]There are some pieces that are really useful that you can use today, which are safe. But it's not fully solved yet.

[00:11:01]We're working on it.

[00:11:03]All right.

[00:11:04]Um and that's kind of what I will spend my time on a little bit to give you a flavor for how um what that looks like. What safety means in this context.

[00:11:17]All right. So, uh here's the first thing I want to say.

[00:11:22]Um If you pretty much are writing something for the GPU, the execution semantics for the GPU are different from that of the CPU.

[00:11:33]Right? Here's a very good example of that, and this is the first one that you'll come across, which is in Rust, there's like a couple of rules.

[00:11:41]The most important one is if you have this thing called uh ampersand MUT, which is called ampersand mute, uh that's basically what's called an exclusive reference. Rust compiler is making guarantees saying that you are assured that nobody else has access to this piece of data that's being pointed to by this piece of by this reference. You have exclusive access to it.

[00:12:10]Now, if you take that concept and try to apply it to the GPU, it just breaks down, right?

[00:12:17]The reason for that is the GPU is not one program that's executing. It's actually the SIMT model.

[00:12:24]That's why the title of the talk was bringing SIMT kernels to Rust, which means a GPU kernel is you can think of that as multiple programs that are executing sort of simultaneously. I mean, the loose definition for that.

[00:12:39]Um so, in this case, you would you you would have to logically think about it as you're giving out one exclusive reference to every single program to the same underlying buffer.

[00:12:52]Right?

[00:12:54]And that's technically undefined behavior in Rust.

[00:12:59]Or what we would call unsafe Rust.

[00:13:03]Right?

[00:13:04]So, that's the first thing that you need to solve, and there is a solution for this, at least for the trivial cases, which I will talk about.

[00:13:12]Uh the reason for this is because as I said, it's undefined behavior to have more than one mutable reference to some buffer, uh because we have something called the borrow checker in Rust that checks to see whether or not that rule holds. That aliasing rule that I mentioned actually holds.

[00:13:31]And there's also a couple of things I won't get into things like send and sync traits because they're very Rust-specific, but effectively the Rust compiler ensures that you have exclusive access to a particular piece of memory.

[00:13:44]And you want to bring that same thing over to uh the GPU semantics or execution semantics.

[00:13:52]At least for the naive case, what we can do is as I already mentioned the Rust type system can be leveraged.

[00:13:59]Now, this is not very new in the Rust world. If you've done some Rust programming on the CPU, you'd probably come across this pattern. It's just being reapplied over here.

[00:14:06]What we can do is we can use the type system to define an execution semantic such that each thread could only be accessing, let's say you have a buffer, an array, can only be accessing one element.

[00:14:23]So, if each thread has exclusive access to a single element in a buffer of elements, you're still upholding the invariant.

[00:14:32]You're still good.

[00:14:34]Right? And you can do that with Rust's type system.

[00:14:37]I can basically just use this type called thread index. The way things work in on a GPU is each thread has what's called a thread index.

[00:14:48]And if you were to sort of ask for that thread index at runtime, you would get it.

[00:14:55]You can now build a type that says, "Okay, if you have this type, then you're assured that you have the index of the thread."

[00:15:04]Right? You're leveraging the Rust type system to say that, "Okay, the the type system I'm going to create a type that is represents the thread ID or the thread index.

[00:15:17]And that's only procurable when you use this particular type. And the key part is you can't construct this type.

[00:15:25]You can only get it.

[00:15:29]All right?

[00:15:31]So, that's the first thing.

[00:15:32]When you do this, you're assured a certain level of certain level of guarantee, as I'm saying this over here.

[00:15:40]The inner value, when you say let index thread index zero, you can't really construct the type.

[00:15:48]You can't really construct the type. You can only get get it at run time from, you know, your implementation itself.

[00:15:56]And that's pretty much [clears throat] making sure that you've earned one from the GPU.

[00:16:02]This again goes back to what some of the concepts in Rust where you can create these sort of types that will enforce certain behaviors for you.

[00:16:11]All right?

[00:16:13]So, if you combine that with a buffer, with a specific kind of buffer, instead of a standard array where, you know, there's no invariance to uphold, we created something called a disjoint slice. By the as the name suggests, it's a disjoint in the sense you want threads that access that buffer to have exclusive access to portions of the buffer.

[00:16:34]So, one is we figured out a way to say, "Okay, how do you represent threads very clearly in your code, thread ID specifically, and how do you ensure those thread IDs have access to very specific elements in, you know, a contiguous buffer?"

[00:16:54]I'm already giving you some uh what do you say?

[00:17:00]Uh some idea of the limitations of this uh particular implementation. It only works in certain cases. It's not a generalization.

[00:17:09]All right? But, the fact that you can do it is one step forward to say, "Okay, you can have safe programming constructs when you talk about accelerated programming. Because when you talk about accelerated programming, the foremost thing on your mind is what? You're using an accelerator, I want every drop of performance.

[00:17:31]Right? It's not really safety.

[00:17:32]But the That's the gap that we also want to build. You don't have to do either or.

[00:17:38]You can have both.

[00:17:40]Right?

[00:17:41]With con- When you start thinking along these lines, we can develop abstractions that will help us get both.

[00:17:51]Right? So, yeah, that's just one aspect that I wanted to talk about. Uh you have You need the proof to get a mutable element. Each index maps to its own elements.

[00:18:02]And uh there's also bound safety. That is, you know, if generally, you don't want a run time check. If you're going to do a run time check to say, "Okay, how big is my buffer?" So that I know exactly how many elements so that these many threads can access it.

[00:18:16]If you do the bound time check on an accelerator, what happens? You lose a bit of performance.

[00:18:22]If you're okay with that, use it. If you don't want to use that, you still have unsafe. Yes.

[00:18:32]Yeah, so you won't need You don't have to worry about this at all. That's a read-only what you say reference. This is for an exclusive ownership reference.

[00:18:41]Where you're saying, "I want exclusive access." Because the concept of Rust is what we're trying to bring over here.

[00:18:47]Uh This has nothing to do with the GPU.

[00:18:48]You can obviously write your kernels today in C++. You don't have to worry about it. They still work.

[00:18:54]But when it gets complicated, you don't know if it works or does it work the way I want to is the question.

[00:19:00]That's what we're trying to answer over here.

[00:19:02]Yeah?

[00:19:05]Okay. Now, that's just a gist of the safety aspect.

[00:19:09]Now, I also want to talk about the meat of it, which which is why I guess you're all here. How does this compiler work?

[00:19:16]What is this compiler built I mean how what's it what does its architecture look like?

[00:19:22]All right, as I said, the pipeline is a discussion that I think was covered in one of the earlier LLVM meetups as well as in some of the other talks. The author is also sitting over here.

[00:19:32]Uh we use a dependency called Plaidon, which is fundamental to how we actually do um what do you'd say compilation in using Rust Oxide.

[00:19:43]If you think about Plaidon, just think about MLIR but in Rust.

[00:19:48]That's another innovation over here, which is so far, if you wanted to work with MLIR, you have to use MLIR stack, which is upstream C++, the whole nine yards, tablegen, all that stuff.

[00:20:01]None of it is being used to build Rust Oxide's compiler pipeline. It's an end-to-end natively Rust written compiler.

[00:20:12]All right. So, uh I hope you can all see this.

[00:20:16]And uh the colors are not like messed up.

[00:20:20]Okay, that's good enough.

[00:20:21]I assume.

[00:20:22]So, what the the the green boxes are what we're building.

[00:20:29]The blue boxes are what we get for free.

[00:20:33]So, there's a considerable amount of time that we spent trying to figure out how to take best advantage of what's already available in the Rust ecosystem.

[00:20:41]We don't want to reinvent stuff and sort of spend time on on cycles on that. We'd really want to take advantage of what's available and get going.

[00:20:51]So, one of the things that you're going to see is when you actually the this is what I mentioned by saying you're not writing a DSL, you're actually writing real Rust is because we're using the Rust compiler. So, the Rust compiler has three parts to it, like pretty much what you know. You've got the front end, you've got the middle end, and you've got the back end. The back end is the de facto back end in Rust is LLVM.

[00:21:12]There are other back ends like crane-lift and GCC and all that stuff which is being built. That's fine.

[00:21:17]But the the primary back end that everyone knows or uses is LLVM.

[00:21:23]The There's I wouldn't There's no real mid-end in Rust because everything is in the front end. So, the Rust compiler, when we say rustc, we're basically talking about the Rust compiler.

[00:21:34]It's the thing that does things like parsing, type checking, um trait solving, monomorphization, all these things are actually performed by the Rust compiler, which is what you see in the blue box.

[00:21:49]All the optimizations that happen in the front end are happening by the Rust compiler.

[00:21:54]All right, that's obviously written in Rust. So, that's what the Rust compiler team or the Rust language project is about.

[00:22:01]It's basically the compiler and the standard library.

[00:22:04]If you go to Rust lang's uh GitHub repo, this is what you'd find in there.

[00:22:11]Um what we do is we've built a custom back end.

[00:22:17]All right, like we have the LLVM back end for Rust, we've built a custom back end called codegen-cuda.

[00:22:25]Now, what codegen-cuda does is very smart.

[00:22:28]We're not going to reinvent, you know, the entire LLVM back end.

[00:22:34]Unless, you know, anybody's up for it.

[00:22:36]People, we don't have access to fable 5 anymore, just saying. So, um keep that in mind.

[00:22:43]It's a It's not something that we'd want to do. So, we'd want to reuse that. What uh codegen-cuda actually does is it wraps LLVM, and then it wraps our own GPU compiler.

[00:22:55]All right?

[00:22:56]That That's kind of what you're looking at over here. You would see that codegen-cuda has two pieces. The host code is directly delegated to the LLVM back end.

[00:23:08]But the GPU stuff goes through our own custom mid end or what you'd call yeah, custom mid end is what I would call it.

[00:23:17]Where what we do is we we basically take in what we are provided by Rust, which is uh what do you call monomorphized um >> [clears throat] >> a monomorphized representation. So, what monomor- Just just for the benefit of everyone if you've not heard of this word monomorphized.

[00:23:34]Uh there are generics in Rust.

[00:23:37]If we have uh we're using that generic in two different places where instantiating them, there'll be two sep- separate copies of the generic function actually compiled into your binary.

[00:23:51]All right?

[00:23:52]So, that process is basically monomorphization. From generics, you're monomorphizing. All right?

[00:23:59]So, what we're doing over here is All right.

[00:24:08]Yeah, the rust C part actually uh uh go ahead does go ahead and uh what do you say? Compile the device and the host side code.

[00:24:20]Nothing.

[00:24:21]So, it does whatever its job is, it does exactly that job. So, it looks at Rust code and says, "This is not This is Rust code.

[00:24:28]I'm going to compile it."

[00:24:29]Right?

[00:24:30]The interception point is the most important point that happens in code gen CUDA where we say, "Okay, are you host are you host code or device code?" And how do we distinguish that? The kernel annotation says, "Okay, you're device code, so you go through this pipeline.

[00:24:44]The rest go through host code or go through the host pipeline.

[00:24:50]Right?

[00:24:51]So, now you go through the you go through that and then the mid end I will explain this in a in some more detail, but effectively, what do we do?

[00:25:00]We get that I I talked about generic code. The way a Rust Rust has its own internal IRs.

[00:25:08]Just to sort of help you out. Rust has Rust starts with you've got source code. Then you've got what is called HIR. These are very very intuitive names. High-level intermediate representation. Then there is TIR, which is typed version of HIR.

[00:25:25]Typed high-level intermediate representation. And then there is MIR, which is mid-level intermediate representation.

[00:25:32]These are pretty much the three different IRs that you pass through on your way to the back end.

[00:25:39]At each intermediate representation, you're going to be doing what you all are you know, used to.

[00:25:45]Some form of transformation, some form of analysis is done.

[00:25:50]All right? For example, Rust source to HIR, that's where the type checking happens.

[00:25:56]That's where type inference happens.

[00:25:59]That's where trait solving happens, if you know what traits are. Yes.

[00:26:05]Sorry?

[00:26:07]Does pre Pre-process Yeah, so the That's a very good question. So, pre-processing does pre-processing not take place somewhere in the pipeline?

[00:26:17]We don't really call it pre-processing over here. You saw the You see the kernel You saw that annotation kernel on top of the the GPU.

[00:26:27]Let me just go back.

[00:26:30]You see that annotation on top of the kernel called the green one.

[00:26:34]There's kernel.

[00:26:36]Yeah? That's called a a macro, a procedural macro.

[00:26:40]So, in Rust, it's pretty much what you call pre-processing. We have a build step. If you have a macro, all macros are expanded.

[00:26:49]So, that's your pre-processing over here.

[00:26:51]This kernel is actually going There's It doesn't actually look like this when it expands.

[00:26:57]The user writes it this way or rather the LLM writes it this way nowadays.

[00:27:02]So, uh that's the pre-processing step. You could obviously plug in a a a few more things, but the one-to-one mapping for pre-processing versus what happens in Rust is proc It's usually called procedural macros. And procedural macros in Rust are extremely powerful.

[00:27:22]You can invent your own languages.

[00:27:24]You can You can also speak to the author of Cléron. He's done some of that.

[00:27:29]So, uh Yes, so I was talking about the architecture of CUDA Oxide, and I said the mid-end, that part that you're seeing, that's what we control. So, all the green boxes are what we control.

[00:27:43]And that's kind of what we want to focus on. We don't want to reinvent the Rust compiler. That's a huge project.

[00:27:49]Millions of lines of code. LLVM, millions of lines of code.

[00:27:53]Not our problem.

[00:27:54]We want to talk about the pieces that actually make sense where we can tap into and really bring all these things that we've been talking about, which is performance along with safety, uh to Rust for That is bring CUDA to Rust.

[00:28:11]All right. So, the moment we we push our GPU code through our pipeline, what do we get? We get as I said, what we call We actually generate The end goal of this is we generate LLVM IR.

[00:28:23]Because as I said, LLVM has a an Nvidia target called NVPTX, which produces what is called PTX, which is the Nvidia specific uh what do you say, virtual instruction set.

[00:28:38]So, that instruction set can pretty much get compiled down to any different hardware that Nvidia produces, because Nvidia has multiple GPU families with different slightly different variations of assembly.

[00:28:52]You don't want to be dealing it dealing it dealing with it at that level.

[00:28:56]You'd want a virtual instruction set, so we've got PTX for that.

[00:29:00]Right?

[00:29:02]Makes sense?

[00:29:03]Okay.

[00:29:05]So, uh how do you do this? So, this is These are the aspects that I actually wanted to bring out because to this audience I thought this might make more sense where the way we actually work with the Rust compiler is we're not forking the compiler. You don't clone the compiler and then add a patch and then maintain the entire compiler.

[00:29:26]That's you could do that. I mean, if you're be feeling very you know, what you'd say.

[00:29:35]Super pumped and you're in a different zone.

[00:29:38]But anyway, uh the idea over here is you So, the Rust the Rust compiler team thought, "Look, Rust's becoming pretty popular. We want other backends to plug into what we have and we want to make that process pretty simple."

[00:29:54]So, the original idea, Rust only had LLVM the LLVM backend. And Rust's you when you compile Rust, pretty much, you know, only LLVM was the option.

[00:30:05]What they did is basically they looked at it and said, "Okay, we'll refactor and pull out all the LLVM specific bits in the backends and we'll create a crate um it's called the code generation secret, but we'll create a crate. Crate is nothing but a library that you guys can use so that you can build your own backends.

[00:30:26]And that's kind of what we're doing as well.

[00:30:28]We're leveraging the same infrastructure to build our own backends. And as I said, there are many custom backends that are available in the wild.

[00:30:36]You can look at GPU stuff, you can look at CPU stuff, you can look at um there's some TPU stuff as well.

[00:30:43]So, there's a lot of different backends available that you can play around with.

[00:30:48]What What does And why are we doing this again? Just to stress the point, cuz we don't want to reinvent all those things that we get for free.

[00:30:57]Which is at the bottom left corner.

[00:31:00]That stuff is not the stuff that we'd want to reinvent. We get it for free.

[00:31:04]We're going to use it.

[00:31:06]Yeah?

[00:31:08]Then, how does this split happen inside the compiler just to take one step further in there?

[00:31:13]Uh, the single source is what we want.

[00:31:17]Because you want your device code and your um host code laid out side by side. So, that if you want to share types between your device and host. For example, I don't know, you you you have a a pointer that's going to reach out to some host memory, but you want that pointer to be available in your device kernel.

[00:31:40]That means your GPU is executing code.

[00:31:43]It's going to get a pointer. That pointer the the reference actually references some memory in host on the host side. This is like a complicated jigsaw puzzle that you have to push push through.

[00:31:55]Right?

[00:31:56]You'd want to be You'd want to have that single source idea over here. Or I can give you another example.

[00:32:01]I assume you might have heard of uh, closures for the ones who've not dealt with Rust. Closures are nothing but lambdas.

[00:32:09]So, lambdas in C++ closures in Rust.

[00:32:11]Um, the idea is if you want to instantiate, let's say, a generic closure.

[00:32:19]That's very difficult if you have uh, the code the the device code in a separate file versus the host code in a separate file.

[00:32:27]Like the instantiation is just You'll be wondering, how do you do it?

[00:32:31]You need that single source concept.

[00:32:33]So, uh, that's the other important thing about CUDA Oxide. It is single source.

[00:32:38]Right? I'm going to show you examples at the end of how this runs. So, what happens? I already mentioned, you take in what the Rust compiler provides.

[00:32:47]What does What does the Rust compiler provide?

[00:32:49]It goes ahead and Okay, you've written a lot of generic stuff.

[00:32:53]You wrote a generic device kernel.

[00:32:54]Great.

[00:32:56]Now, the Rust compiler does everything that we just talked about. Gives me uh you know, code in a representation that is saying, "These are the concrete instantiations of what you've written."

[00:33:09]And then I take that and I go through the pipeline. I start to look for as I said, I start to look inside what's called a code gen unit. Basically, you we're trying to determine uh which ones are kernel code, which ones host code.

[00:33:25]Host code goes through the host pipeline. Device code goes through the device pipeline. That's what the scanning CGU means. CGU is basically code gen units.

[00:33:33]You're basically looking Code gen unit is nothing but a bunch of different uh functions that you'd want to scan through and bucket them and say, "Okay, this is one unit. I'll look through inside that to see is anyone a kernel function?" I- If If you find one, then you send it in a different pipeline.

[00:33:51]The others go through the host pipeline.

[00:33:53]Right?

[00:33:55]And then uh so, if that kernel function again calls other functions, right? Because uh a device a function that's running on the GPU can call other functions. They are called device functions. What do you do in that case? We actually transitively walk the call graph and push all of that into the device compilation pipeline.

[00:34:16]Right?

[00:34:20]Yeah.

[00:34:25]You use the same thing.

[00:34:26]Same thing. There's no And that's the whole point. Like, we were wondering whether we could do it. We can. So, we don't even have to annotate the function as a device function.

[00:34:37]If your kernel function calls that function, it's a device function.

[00:34:41]There are other reasons why you want to annotate it as device because let's say you But that's getting into the weeds.

[00:34:46]You want to do something called LTO IR, etc. In those cases, you want to express explicitly annotate something. But in general, you have a kernel function.

[00:34:54]That kernel function calls any other function, it is considered a device function.

[00:34:59]And then it's automatically included in in four device compilation.

[00:35:05]All right. So, yes, that's the host device split. Yes.

[00:35:14]Yes. Yes.

[00:35:31]So, like for example, I'll show you an example. I know what you mean.

[00:35:36]I think I'll just show you an example. It's It's exactly the way you think in Rust as well.

[00:35:41]We are encoding the the idea of a thread index in a type.

[00:35:47]And but that type information flows across everything that's considered device side compilation.

[00:35:55]Yes.

[00:35:56]Because the Rust compiler, you remember, is doing a lot of stuff. It's going to check all that for us anyway.

[00:36:02]All right?

[00:36:04]So, yeah. So, how are we doing this? Now, this is where we get into a little bit of the Rust uh specifics.

[00:36:14]We are basically standing on So, without this particular project, this would have been a lot more difficult because I was doing this before this project was around, and that was a nightmare.

[00:36:25]What happens in the compiler? The Rust compiler, a whole bunch of people are working on the compiler. I don't know, a thousand people maybe.

[00:36:31]They will keep adding stuff, breaking stuff, making it you know, whatever that it is that they want to do with it.

[00:36:39]And then, once everybody everything stabilizes, you get a stable version of the compiler, you also get the nightly version.

[00:36:44]But, the internals is such, that means all the libraries or crates in Rust parlance that make up the Rust compiler they are actually unstable.

[00:36:56]If you were to use them as a user to say, "Okay, I want this information from the compiler, I want that information from the compiler."

[00:37:02]That's not really you know, stable.

[00:37:07]You can you'd write it one day, the next day somebody thinks somebody's changed something.

[00:37:11]They because the compiler somebody pushed out a nightly version.

[00:37:15]You can't you can't keep that up.

[00:37:18]All right? So, what happens is, and I'm not the only one who's sort of having to deal with this.

[00:37:24]There's a bunch of other projects that are dealing with this. For example, there are formal verifications systems projects where they want to use whatever Rust does analysis or safety analysis that Rust does. Assume borrow checking is an analysis that they want.

[00:37:40]And they try to plug into the Rust compiler, they're going to face the same problem.

[00:37:44]The compiler keeps changing, it's unstable, you can't depend on it, you can't write a tool based on that unstable API surface.

[00:37:53]Right? And we can't write a compiler on top of that.

[00:37:58]I did it. Trust me, that's why I lost all hair.

[00:38:01]It just doesn't work.

[00:38:03]Okay?

[00:38:03]So, the the idea is we want something that's more stable.

[00:38:11]So, people have been working on this for a a long time.

[00:38:14]We have got something called Rust C public.

[00:38:17]In a nutshell, Rust C public is the public interface to the Rust compiler.

[00:38:23]These are compiler internals that you can get access to Okay, you don't get everything, but you get the stuff that you'd want. At least good oxide gets everything that it wants.

[00:38:32]All the types that we are dealing with in good oxide, every single thing. What type is this? What is the lifetime?

[00:38:40]Um you know, the what is the Is this uh Does this belong to some other or is it connected with some other type, etc. Questions like these are some things that you can easily answer because the next step What I have What I have done so far is I've let the cross compiler run and I segregated out the pieces of code that needs to be device compiled versus host compiled. That's all I've done.

[00:39:06]I've not really done the compilation.

[00:39:08]The compilation starts where you want to take in whatever Rust provides in whatever format.

[00:39:14]I want to convert that to a format that I my compiler depends on.

[00:39:21]That translation to do that translation, I need to be using something like Rust C public.

[00:39:28]All right?

[00:39:29]Hope that makes sense.

[00:39:32]Right?

[00:39:35]Any questions on this because this is like the little interesting detail.

[00:39:44]Yes. So, this is uh this is part of the project goals.

[00:39:48]It's going to be you know, the official thing because there's quite a few people depending on it at this point.

[00:39:54]And that's that works in our favor as well.

[00:39:57]So, the idea is it's not yet stable. It is being stabilized.

[00:40:02]So, to all the compiler folks over here, please help me, okay? If you can dabble in it a bit, do what you like, contribute to it a bit, it'd be great.

[00:40:13]Uh yeah.

[00:40:20]>> It is part of the upstream rust.

[00:40:22]Right? So, this is an actual crate in upstream rust.

[00:40:26]At some point, this will move to what's called crates.io.

[00:40:30]Crates.io is like your Python pip package listed in some, you know, storage area.

[00:40:38]That's what crates.io is. So, if it moves out of there, that means you can now pull it in as a dependency like a normal library.

[00:40:45]Right? It's not unstable anymore. There are plans that we're working on for that.

[00:40:50]Anyway, so the way it works is the rust compiler internals on the left, the rustc public interface is on the right. There is a bridge that has been built.

[00:41:00]There's some There's some conversion that happens between internal compiler types and the external public surface.

[00:41:07]Because we can't stop internal internal compiler development for our API.

[00:41:13]We still want them to go at the whatever speed that whatever speed that they want to go at without blocking them. So, this stable bridge is something that, you know, helps helps do that.

[00:41:26]Yeah, and if you're thinking how does that happen, and does it happen automatically? No, it's a lot of work.

[00:41:30]It happens with the work. Okay? It has to be maintained, in other words.

[00:41:37]Right?

[00:41:40]So, uh yeah, so that's the architecture of So, I don't want to bore you with a lot of detail because Qod oxide is not um How do I say this? A prototype anymore.

[00:41:54]It can do stuff that is really amazes me today because there are people doing stuff with it that I did not expect.

[00:42:01]In some cases, we are we are about 5% 8% better than the standard mature compilers, which is pretty good for where we are.

[00:42:11]So, that's one key point that I wanted to drive home, which is the compiler is designed with a design philosophy such that you can hack it.

[00:42:19]You can hack it for exactly the the workload that you're looking for.

[00:42:23]If because other compilers, when you're working with them, they have the legacy of all the other stuff that they have to support.

[00:42:30]This one doesn't.

[00:42:32]You can use it exactly to meet the need that you're looking for and not have to like riggle like tort it to meet your needs.

[00:42:42]Because the moment you deal with compilers, I think all of you probably have noticed this. You change something here, then something else works break somewhere else.

[00:42:49]Right?

[00:42:51]This is not that case because the whole idea of the compile this compiler or who doc site is you provide the base.

[00:42:58]Everything else that you want to build on it is an optimization that is not default. You can add it, you can choose it.

[00:43:05]Yeah.

[00:43:10]So, the the Rust compiler the Rust when you write Rust code source code and then before you hand it off to let's say some back end like LLVM you have to that means you're going to provide LLV produce LLVM IR before you hand it off to LLVM.

[00:43:28]From the stage where you actually have Rust code all the way to LLVM IR there's a compiler front end that's written.

[00:43:35]That's made up of a bunch of different crates.

[00:43:38]All those crates are unstable.

[00:43:40]And that's part of the Rust language.

[00:43:43]If two umbrella pieces or two pieces under the umbrella, the standard library and the compiler. That's the Rust compiler.

[00:43:51]So, all these libraries that help you get from source code all the way to LLVM IR, there's a lot of stuff that happens.

[00:44:00]All that stuff unstable.

[00:44:05]Yes.

[00:44:06]You can't count on them.

[00:44:08]There's another way to get across it if you really want to work with it.

[00:44:11]There's an option to just pin the compiler version.

[00:44:14]So, you can just work in that version.

[00:44:16]And then, once you're ready to move to the next version, you will have to like upgrade the that pin version. That's a way to do it, but yeah.

[00:44:26]Yes.

[00:44:42]That's a It's a very good question. So, just yesterday I got a PR request for O2 being included. That is you want to use your LLVM optimizer for it. We've added it.

[00:44:54]But, we're still deciding whether we want to keep it as a default.

[00:44:57]The idea over there is uh LLVM the optimizer will go ahead and do your vectorization, your SROA, all that stuff.

[00:45:07]That's not something that we want to like box you into saying because if you do all these optimizations, you can't do the other thing.

[00:45:14]You can't do this. You'll have to like worry about something else.

[00:45:18]So, we want to like get out of that way and make the compiler a lot more hackable for your use case.

[00:45:25]You can do that only when you know that you have a smaller core.

[00:45:29]The core is small, it's very easy to reason about, and then you say, "Okay, I have this particular workload, and I want this accelerated."

[00:45:37]All I need to do is focus on that workload. I'm not going to focus on anything else.

[00:45:42]Right?

[00:45:43]So, the moment you do that, you obviously end up with the core compiler and a whole whole number of optimizations that you may pick you may want to choose from, or you may not want to build your own.

[00:45:54]Yeah.

[00:46:03]Yeah.

[00:46:05]Correct. So, that's one one way of looking at it, which we have today. We have I've accepted the PR, but uh it's still something that I'm considering because the hackability aspect does lead to something more important.

[00:46:18]As I said, we don't write code anymore. We review code these days.

[00:46:23]So, given that is where we're headed, and every single um So, every single engineer over here is pretty much going to be dealing with the edge of things.

[00:46:34]You're not dealing with getting the thing functioning, working, etc. It's doing it. It's happening.

[00:46:40]You want to squeeze out that extra 10 extra 2% extra 3% of performance.

[00:46:45]To do that, you have to do a lot of things today.

[00:46:50]What I'd call gymnastics.

[00:46:52]Now, with this, if your reasoning capabilities are very clear about what's in that core and what do I need for my use case, that should help you really convert or the LLM itself converge very quickly on a solution.

[00:47:09]So, automatic kernel generation and optimization is one of our core areas that we're focusing on.

[00:47:17]Uh and that's kind of why I'm thinking about a hackable kernel compiler.

[00:47:24]Right?

[00:47:27]Okay.

[00:47:28]So, I wanted to bring this up because um since we've been talking about the architecture quite a bit, also wanted to drive home uh something that is straight from our work with the compiler where we noticed something very interesting where something that works on the CPU straight off is just a miscompilation on the GPU.

[00:47:52]That's why I call it a war story because it did take a little while for me to figure it out exactly which portion where.

[00:48:00]So, the common idea is there is this uh common optimization that's also applied in LLVM as well, which is called jump threading.

[00:48:08]You have like multiple jumps. You sort of want to duplicate that and just you know, put it in both places so that you avoid the jumps, right?

[00:48:16]Uh that doesn't work with the GPU.

[00:48:19]Especially if you're talking about things like barriers.

[00:48:22]That's specifically breaking semantics of the GPU.

[00:48:27]So, something that works on the CPU doesn't really apply to the GPU. In this case, there's no other option. We had to disable jump threading.

[00:48:36]The the the negative of that is we've so far not figured out a way to disable jump threading only for device code.

[00:48:44]It also disables it for host code.

[00:48:47]Right?

[00:48:48]So, that's an example of you know, what you can think about the problems that you can think about when you try to bring uh something like Rust to the GPU. These are This is another thing that I want to talk about.

[00:49:04]And then, yeah, sorry.

[00:49:17]Yeah, so technically it applies, but by default in Rust, jump threading is enabled.

[00:49:24]Unless there are specific cases like you've mentioned where uh again, there are very specific intrinsics involved.

[00:49:30]In those cases, you might see that, you know, I haven't looked at exactly how they disable it, but in general, by default, jump threading rustc enables it.

[00:49:40]So, if you write um if you write some normal Rust code and compile it, and then you actually check the mid-level intermediate representation passes, whatever 10 passes that they have, you will see jump threading automatically.

[00:50:04]No, yeah. So, in this case, one is we have the borrow checker. We force CPU multi's ready.

[00:50:09]Along with the fact that, you know, there are safe guarantees being offered, etc. The jump threading gets applied at the very end.

[00:50:16]Yeah, so the the flow the the phase at which you apply this is very important.

[00:50:22]You know, there is some reasoning happening beforehand where you're saying, "Okay, we've the compiler is the borrow checking has happened. We've already checked that, you know, the aliases are correct and all of that is correct. And then we ensure that um send and sync traits, all that stuff is good. So, if we've the compiler has ensured that this is correct, then I can go ahead and apply the jump threading pass. So, should work.

[00:50:52]Yeah.

[00:51:01]So, in the case of CPUs, again, what we're basically talking about is GPU the semantics change wherein where explicitly saying that if you were to enable, you know, you are sort of fuse the jumps into a single Oh, sorry. If you you were to duplicate the jumps, you're going to break semantics. In the CPU's case, the position that the Rust compiler is taking is if you've already done the reasoning, that means, let's say, in in case we're talking about a barrier intrinsic that you're using on the CPU.

[00:51:34]For example, you're saying the I'll just take some atomic instruction, right? You take any atomic instruction as well. That's already factored into the fact that here is how my type system works and here is how I'm enabling multi-threading. So, I'm ensuring that when threads communicate, if we're talking about safe Rust, and I'm ensuring that the entire thing, the entire pipeline that is being used for multiple threads to access a shared a buffer, that is safe.

[00:52:09]If you can ensure that that is safe, then when you come towards the end of the pass application, that should not be a case when you when you apply it.

[00:52:19]Because the idea is if you didn't do borrow checking and and then you asked, "Can I Can I apply jump threading?"

[00:52:26]Then I would be in a question, "Can we?"

[00:52:28]Because we'd actually end up in a similar scenario.

[00:52:31]Now, the question I have is different is how does another language do it which doesn't have um borrow checking?

[00:52:41]Right? For example, how does LLVM do it?

[00:52:45]Cuz LLVM has a jump threading pass. How does it apply it?

[00:52:49]That's a Yeah. So, Correct.

[00:53:03]Correct. In this case, the the scenario is very different because the compiler doesn't have the information.

[00:53:09]In the previous case, the compiler has the information.

[00:53:12]And then you're saying, "Okay, now I'm I've reached this point. If I've not already optimized this away such that the jumps are not there or whatever, if they're there, I can go ahead and apply the jump threading pass."

[00:53:23]Yeah?

[00:53:27]So, uh yeah.

[00:53:31]Uh so, that's how we By the way, at the bottom, that's how we disable jump threading.

[00:53:36]This is This is uh something that I guess very similar to how you do in LLVM as well.

[00:53:42]Yeah, quickly coming to the final parts, which is how does the pipeline work?

[00:53:48]What do we do? That is you saw the mid end that we control.

[00:53:53]Whatever the in the whatever is in the mid end if we can go ahead and get whatever information whatever IR that is passed from the Rust compiler we first need to go ahead and convert that to what is called dialect in my heart. As I said Pluron is basically the equivalent of MLIR except that it's completely the infrastructure is completely written in Rust.

[00:54:19]The whole idea of ops, attributes, regions, recursive IR all that stuff is fully available to you.

[00:54:27]That means you can author dialects like you would author dialects in MLIR in Pluron and then you can do exactly what you do in MLIR.

[00:54:38]Or even more in some cases. We will talk about that.

[00:54:41]But we have three different dialects.

[00:54:44]Dialect MIR, which is basically modeling Rust's intermediate representation called mid-level intermediate representation.

[00:54:51]And then Pluron LLVM, which is the LLVM dialect.

[00:54:55]That means it's a it has a one-to-one mapping with LLVM IR.

[00:54:58]And then there's the NVVM dialect. NVVM is basically stuff for intrinsics.

[00:55:03]All the intrinsics that you want to emit, this is where things get, you know, a little curly with the barriers and stuff where uh all the intrinsic stuff is being modeled by this dialect, which is dialect NVVM.

[00:55:17]These three dialects simply make up the good oxide compiler pipeline.

[00:55:25]Right? Apart from, you know, the other things that we already use, which is the Rust compiler and the LLVM back end, these three are what we have built to ensure that we take Rust code and then we create LLVM IR.

[00:55:42]Makes sense?

[00:55:43]So, the first one models rust intermediate representation.

[00:55:46]The second one models LLVM.

[00:55:49]The third one models So, the second one you can just think of as an LLVM dialect from MLIR, which you already know.

[00:55:55]The The third one you we already know, which is an NVVM dialect from MLIR.

[00:56:01]The only thing that doesn't exist the equivalent doesn't that doesn't exist today, there's no MIR dialect in uh upstream LLVM. Sorry, upstream MLIR.

[00:56:13]Okay?

[00:56:14]That's what we have here. Yeah.

[00:56:28]Yeah, that's a that's something that, you know, exactly how you would do it in MLIR.

[00:56:37]But, I would say this, uh the Pluron project is actually some um the right way to say this is we are working on a bunch of different things.

[00:56:48]The legalizer's already part of the infrastructure that's available to you as part of Pluron. So, you would be able to use that.

[00:56:56]But, to your specific question, uh I'll actually let uh you know, you can probably have a chat with the Pluron author at the back uh after this call after this meeting.

[00:57:06]So, he'll be a better person to actually answer this very specific part. But, I would say this much, if you're talking about optimizations as such, we have a number of optimizations that are already part of Pluron, and there are more that are going to be added.

[00:57:20]Uh I hope that sort of helps with that question.

[00:57:26]>> How does something like fusion does the fusion define the semantics based on the type having ownership >> I wouldn't say so. I haven't looked at it exactly, just to be very clear.

[00:57:46]But so far, I don't think anything like that has been a problem.

[00:57:51]But we can talk about it. I think I would certainly want you to have a chat with him.

[00:57:56]So, these are the three dialects that we use as part of the infrastructure that's available to us in MLIR.

[00:58:05]Now, um as I said, our pipeline is going to be the base pipeline. We don't want to be immediately talking about um what you'd say our compiler has all these optimizations, so we're going to be go really fast, etc. We're looking at a very different design, and that's kind of why the compiler pipeline is pretty bare-bones.

[00:58:29]The small core with as many optimizations as you want, but apply them for your specific accelerated workload is the design path that we're trying to go and see how that works out actually. But yes, so the only pass that you would see right now, which is default, is mem to reg. That's it.

[00:58:49]That's kind of what we That's all we need to ensure that we can model dialect MLIR dial- dialect MIR.

[00:58:58]Sorry, MIR in our um play on dialect, which is dialect MIR.

[00:59:07]So, what works today?

[00:59:11]Pretty much everything.

[00:59:14]If you find something, if you try this out and find something that isn't working, it's a bug.

[00:59:18]All right? So, please feel free to raise a PR.

[00:59:21]Right? So, ownership semantics, borrow checking, uh every little piece works and I couldn't I obviously can't fit everything here because all of the rust all of the language semantics sorry constructs cannot be something I can include in one slide.

[00:59:39]There's a lot more that is included in the slide and one last point would be right now we're supporting Ampere.

[00:59:47]So, there are different families of or architectures that we support in Nvidia.

[00:59:53]Cuda Oxide is currently targeting Ampere and above.

[00:59:57]Ampere uh Hopper and Blackwell are the three architectures we support. There's going to be stuff like there are Rubin and whatever comes next.

[01:00:07]All right?

[01:00:10]So, I'm I wanted to show you some demos which I will show you.

[01:00:14]But just to give you a heads up, this is what you'd call uh a full-fledged kernel that's written in Cuda Oxide today.

[01:00:22]Uh this is just about a small snippet.

[01:00:24]The actual kernel is some I don't know some some 300 lines or something.

[01:00:29]But the key point I wanted to point out was this is what you call the speed of light uh today which means state-of-the-art that is provided by Nvidia's own cuBLAS library which is if you if you take that as 100% we're somewhere at 60% with zero optimizations.

[01:00:53]With a plain vanilla pipeline, we haven't done anything.

[01:00:57]We haven't even applied O2.

[01:01:00]Okay?

[01:01:01]Yes.

[01:01:05]So, yeah.

[01:01:06]So, this is what you'd call you see all these blogs that write that talk about generalized matrix multiplication the core primitive that is used in LLMs massive matrix multiplications.

[01:01:19]So, if you were to run this on a black level GPU, you would get something around 1500 tops.

[01:01:27]Okay, flops. T flops.

[01:01:30]Tera flops.

[01:01:31]Here we're getting something around 870 tera flops.

[01:01:38]Right?

[01:01:41]Yes.

[01:01:44]Yes.

[01:01:53]So that's exactly why I said you should start talk to again the Pluron author because um I told you we talk we talk we use something called procedural macros in Rust and they're very powerful.

[01:02:04]So we use the same thing to actually uh pretty much build a lot of infrastructure in Pluron.

[01:02:11]Uh that's kind of what we use to effectively model the entire thing. You don't have to write your ops and your attributes and your everything in one language and then, you know, sort of build them and get it in C++ all that stuff.

[01:02:26]Everything's in Rust and everything's in native Rust. So procedural macros, if you work with them, there's basically a stream of tokens.

[01:02:34]It it it's a a macro is something but takes in a stream of tokens and then produces another stream of tokens.

[01:02:39]That's it.

[01:02:41]And uh the Pluron uh project does this really well and really intuitively and that's kind of why we've used it in the Oxide project.

[01:02:53]Okay, so uh Yeah, I'm going to show you a demo of this.

[01:03:00]Yeah, all right.

[01:03:08]So the safety aspect that I showed you was in the beginning where we talked about the thread index and the disjoint slice.

[01:03:14]Some portions of that are used over here.

[01:03:16]The other thing is you're also uh if you look at this kernel which I will try and show you but don't don't abuse me for that because it's a pretty lengthy piece of kernel.

[01:03:26]Uh you will see other Rust constructs like you will see builder type patterns.

[01:03:31]You will see things that are usually not associated with a kernel.

[01:03:37]I mean you wouldn't see all these quality of life improvements in the kernel. So all those all the rank all the the benefits that you can squeeze out of a strong type system at least the the low hanging fruit stuff is all incorporated.

[01:03:53]Right? So for example, we have type state patterns built into the kernel itself which means if you want to enable a barrier in a certain in a you want to put the barrier in a certain state but you have to go through a sequence of steps in an order to be in the legal step in the legal it you can encode that in the Rust type system.

[01:04:16]Or for people who do not know about this the simplest example that I can give you is I don't know if you work with embedded systems because that's where I come from.

[01:04:24]You want to enable an LED there's like a bunch of different states. It's on off and there's an intermediate state as well that people don't talk about. Okay? That exists. So you need to like go through those states but you can model those states in the in Rust type system and say that if you ever transition from one state to another state and that previous state was an illegal state it is a compile time error not a run time panic.

[01:04:56]Make sense?

[01:04:57]This is stuff that you get with the Rust compiler. So patterns like that builders pattern type state pattern and a whole bunch of other patterns that are available to us.

[01:05:05]Trade off.

[01:05:06]Yes.

[01:05:13]Yes.

[01:05:27]Right. So, the question was uh I mentioned that you know, in some another use case we actually got more performance from my my computer's on, so my computer's on this this thing blacked out.

[01:05:43]So, the question was uh there was another uh example that I gave where we're actually seeing we're actually seeing uh better performance from the good oxide compiler better than, you know, more mature compilers, etc. I am very glad that you asked that question because that's not something that we did. It is a community Let's say it came from the community.

[01:06:10]Somebody actually added an optimization.

[01:06:12]Somebody actually did what we exact what we were talking about, which is add a bunch of very target-specific or targeted optimizations for their own work workflow, which work load, which in this case was FFT.

[01:06:27]So, the person works on some biology-related stuff. He's trying to do some fast Fourier transforms, get the best performance out of it. He does not want to use cuz he's written a lot of stuff in the Rust world.

[01:06:39]He wants to use the kernel. He wants to use good oxide um for this, but we don't have any optimization passes because we don't use them.

[01:06:49]You can plug in your own optimization pass. He plugged in what is called an FMA pass, which is fused multiply add, which is primarily what you need for something like the FFT.

[01:07:00]Right?

[01:07:01]So, and plus he turned on uh the optimizer as well.

[01:07:07]He turned on optimizer plus he did this, so he saw a performance shoot by 8%.

[01:07:13]And, you know, you could probably look this up. I think he he put this on LinkedIn and shared it with me, so that's how I know.

[01:07:20]So, we didn't do it. This is what I mean.

[01:07:23]Like that, there are many examples that have been brought to us.

[01:07:32]Uh okay.

[01:07:41]Hey, so the TV's working.

[01:07:47]Okay.

[01:08:09]And then Is your screen working?

[01:08:23]Yeah, should I share my screen first?

[01:08:26]It's working.

[01:08:27]Okay.

[01:08:37]Okay, where did that go?

[01:08:40]I can't show you that.

[01:08:44]You hold this?

[01:09:03]And it's still loading.

[01:09:10]Oh, yeah, it says one.

[01:09:15]Okay, so we're here. So, I quickly wanted to show you a couple of demos, get you excited, make you, you know, part of the cult. So, but so um yeah, so this is just an example of, you know, the kind of performance that we're already seeing uh with CUDA Oxide.

[01:09:37]Now, the other thing I wanted to show you was I'll have to put this down, but I'm just going to show you a quick demo.

[01:09:53]But, uh I hope you can see this.

[01:10:05]All right, so the way this uh sort of works, we have our own build tool. You just say cargo oxide run, and let's just say we want to build something like a vector addition.

[01:10:18]You can go ahead and just launch the kernel, and you should be done with it.

[01:10:22]If you want to see what that looks like, so I already showed you an example of this.

[01:10:31]You would write your kernel, which is this is your kernel. It's inside a module called the kernels module, and then you annotate it with the kernel attribute or macro, and then you have your own main function.

[01:10:47]This is your host code that's actually doing the launching of the kernel.

[01:10:51]Right?

[01:10:52]So, this is the most basic version of how you would write a kernel, Everything in one file, single source.

[01:10:59]The The host code has has visibility to what the device as in the device has visibility what to what the host code has in terms of types and etc. Right? So.

[01:11:20]Sorry about that.

[01:11:21]Yeah, sorry about that.

[01:11:22]I hope you can hear me now. So, just to repeat that again.

[01:11:26]Um what you saw previously was you've got device code which is annotated with the kernel macro.

[01:11:37]As you can see here. And you can see this is actually a safe kernel.

[01:11:43]There's no unsafe word.

[01:11:46]And usually you should have something like unsafe over here.

[01:11:49]But that's not there. Because this is the most trivial kernel that you can write. What you would normally call this is an embarrassingly parallel workload.

[01:11:58]Right?

[01:11:59]So uh that's the most trivial version of it.

[01:12:04]Now for the most uh interesting version, I'll have to switch to a different machine. So, I'll have to put the mic down.

[01:12:12]All right. So, in the interest of time because uh my machine decided that you know that particular demo is not happening.

[01:12:22]I'm just going to walk you through what that kernel looks like which is the matrix multiplication the the generalized matrix multiplication that I was talking about.

[01:12:31]We just bring that up.

[01:12:39]All right. So, if you actually looked at So, sorry.

[01:12:48]So, if you uh go into the code outside repo and you look under examples, you'll find gem_soul.

[01:12:58]Soul just stands for not so s o u l but speed of light.

[01:13:03]Uh which is which is the terminology that Nvidia uses for state of the art.

[01:13:07]Uh here what happens is if you go and look at this kernel, I have like five different versions of it, each one adding basically exercising different parts of the hardware so that you get better and better performance.

[01:13:21]If we did have, you know, uh the numbers, I'd be able to show it to you. But effectively what the kernel looks like is something like this.

[01:13:33]So here's the first one. This is the tiled version of it.

[01:13:37]What you would normally see is you would see that, you know, we allocate some static shared memory, which is effectively what you would call um um the the most um low latency memory that's baked into your GPU, which is where you'd want to put stuff if you want to access to some of this um some of this information so that, you know, you don't have to go and reach it all the way into what you call global memory or HPM.

[01:14:07]Uh and then you initialize your bar- barriers or you basically declare them.

[01:14:13]And then you start setting up the entire chain.

[01:14:17]All this stuff is very GPU specific programming stuff, which I not really don't want to really get into.

[01:14:24]But I do want to show you a couple of things that I talked about, which is let's say you wanted to do things like, you know, there's a descriptor that you need to hand to some uh piece of your code to say, "Okay, you want to go ahead and run this um do something, you need to provide it a pointer to a descriptor."

[01:14:43]The way you construct that is very manual and error-prone.

[01:14:47]But now with you can probably do that in a very functional style, number one. And this is called the builder pattern.

[01:14:54]You can basically set certain values uh in a very type-driven functional manner.

[01:15:03]All right? That's one. Then there's other areas over here that we talk about where we say you can do things like type state pattern, etc. which uh okay. The other thing that I can also talk about is intrinsics are all typed.

[01:15:18]Because when you add an intrinsic to the Rust to good oxide, we actually have infrastructure that will help you to use types as much as possible so that you can't go wrong with its usage.

[01:15:32]All right? That's another thing. That's again baked into the compiler.

[01:15:37]So, all these pieces are things that are available to you in the compiler.

[01:15:42]Uh Yeah. So, for example, I don't know if I can show you this and if this is here.

[01:15:57]All right. Yeah, I just wanted to show you the type state pattern as well, but that example's not available.

[01:16:02]Yeah. So, effectively speaking, this is you know, as you can see, this is not a prototype as such.

[01:16:09]This is something that you can really use to write complex kernels.

[01:16:13]You can pretty much write any kernel.

[01:16:15]Right now, the only thing that we've not yet fully supported is uh being able to write what you'd call networking-related kernels. That means the GPU directly wants to do something with uh networking, which is called collective operations.

[01:16:34]That means you have a bunch of GPUs, all of them want to talk to each other, share information, do a bunch of different compute.

[01:16:40]Uh we've not yet finalized on a bunch of APIs for that, which is something that we're working on.

[01:16:47]All right. So, that's uh in a nutshell, what I wanted to talk about as far as CUDA oxide goes.

[01:17:00]Sorry, I couldn't get the the demo running because I just can't log into my machine for some reason. VPN's not working.

[01:17:07]Um Somebody asked a question about should we just uh is CUDA oxide suggesting that we start writing every single kernel in Rust?

[01:17:16]No.

[01:17:17]That's not going to happen.

[01:17:19]Uh, right now you can already write all the kernels that you've already written in C++. Still work.

[01:17:26]Right? You can use Rust kernels and C++ kernels together because we have an option wherein we can do this at link time, wherein you can write your Rust kernel that calls into a C++ kernel. I'm talking about a Rust kernel GPU kernel calling into a a C++ GPU kernel or vice versa. All right? Both options are available today and we do that through a technology called LTOIR, that's available to you as well, which means the existing NVIDIA libraries that you know and love, which is things like NCCL, which is called nickel as well.

[01:18:02]Uh, cub or cuFFT.

[01:18:05]All of this stuff, it's not like you have to stop using them.

[01:18:09]It'll just work.

[01:18:10]You can augment that with Rust if you're working in a Rust environment.

[01:18:16]Right?

[01:18:17]And then finally, the status is the full compiler is fully sound and working. Not found, but fully working.

[01:18:25]Uh, all the features that you'd normally expect in Rust have are available to you on the device side as well.

[01:18:33]I want to put a caveat over here.

[01:18:35]All the meaningful features in There are things like dynamic dispatch, etc. that exist in Rust that don't really make any sense at this point. If you need it, please reach out to me. I can go ahead and we can go ahead and discuss that for the use case that you have.

[01:18:53]And then, as I already mentioned, experimental stuff is this idea of a hackable compiler.

[01:18:59]We're also working on a whole bunch of rich optimization.

[01:19:03]Uh rich rich optimizations that A are available through our mid-end through Pleroma as a dependency, as well as community source optimizations, as well as first-party ones.

[01:19:14]But, the idea is the core design is create a central core, which is very hackable, that will enable any kind of workload to be accelerated very quickly.

[01:19:25]Right. And that's the the link to the repo.

[01:19:31]Uh and yeah, you can find me at the handle Alpasha. That's my GitHub handle, as well.

[01:19:37]So, please feel free to reach out to me if you have any questions about this.

[01:19:41]Yes.

[01:19:41]>> Uh so, you mentioned that some network stuff doesn't work. Does that include really and multi-GPU questions?

[01:19:48]>> Yeah, so that's the part. So, we have not It's not that it doesn't work. So, just to clarify.

[01:19:55]We have experimental prototypes that already work. The the thing that we're figuring out the the APIs that we want to expose, because that's where Rust will really help considering, you know, there's a lot of concurrency involved.

[01:20:10]There's a lot of common concepts that you would find in the CPU world that are applicable over here, as well.

[01:20:18]So, stay tuned for that. We have a lot more to share on that.

[01:20:21]>> So, it's mostly tested on like single GPUs.

[01:20:24]>> Yes. Yes. Yes.

[01:20:27]Yes.

[01:20:28]>> So, uh how do you handle a case where multiple blocks could write to the same memory.

[01:20:33]Is there a way to do that, JP?

[01:20:35]>> Uh that's a good question. So, you've seen the trivial case.

[01:20:40]Uh we're using the type system to sort of, you know, help where we can reason about it.

[01:20:46]If you have a scenario where you have cooperatively, you know, working to produce a result.

[01:20:52]That means threads are cooperatively working to produce a result.

[01:20:55]The generalized safe abstraction for that is not something that we've worked out yet.

[01:21:02]There are some ideas. There is some exploration.

[01:21:05]Uh the What I want to say is it's not So, we have a whole page for this.

[01:21:10]Before, you know, I actually say it's not possible, let me also point you to We have a book.

[01:21:20]Okay? So, that book uh has some AI images.

[01:21:26]So, don't worry about that. But, we have a very specific page on safety.

[01:21:31]All right? I would recommend that you read go through this page for extended context, but just to answer your question, the generalization is possible. There are some hard problems that we need to solve.

[01:21:41]Those hard problems do have solutions.

[01:21:44]Uh specifically because we're using Rust and some of the guarantees that Rust's compiler offers in the con- either be it via the borrow checker or be it via the type system.

[01:21:56]But, we need some extensions that are not yet what you would say part of Rust.

[01:22:01]So, that's a whether we want to do it within Rust or want to extend it is a question.

[01:22:08]Let's put it that way. Yeah.

[01:22:09]Yes.

[01:22:14]It's a very nice question. That's a PR that's opened up. We will absolutely support it. That will be something that The the reason again is there already exists a bunch of existing APIs.

[01:22:26]There's some recent work that's been done on the graph stuff that is really good. We we're thinking sort of build on the new stuff.

[01:22:33]Uh take the newest stuff that we're already working on.

[01:22:37]So, yes. The answer is the short answer is yes.

[01:22:43]Right?

[01:22:45]Uh so, yes. In case, you know, the what do you say? The GitHub repo has like a lot of stuff that's going on.

[01:22:54]The examples are, you know, not something that you want to start with.

[01:22:58]I would recommend the book.

[01:23:00]One is I wrote it. Please give feedback.

[01:23:02]Two.

[01:23:03]Took a little bit of work back then. So, when I wrote the book, there was no LLM.

[01:23:08]Cloud 4.5 was just beginning.

[01:23:10]Took some work. Nowadays, that's not the problem.

[01:23:13]So, yeah, please provide feedback. We've tried I've tried to cover as much as possible.

[01:23:19]Uh there's everything that you could think of. Getting started, installation, and also how to use the compiler.

[01:23:28]All right. So, that's kind of what I wanted to cover. Any questions? We'll take it or yeah.

[01:23:36]Okay.

[01:23:37]Right. Thank you very much.