Rust's God Mode

Added: 2026-05-12

[00:00:00]If someone tries to convince you that Rust procedural macros are complicated, show them this one. It doesn't do a whole lot, but it is only a few lines of code. Rust procedural macros take a token stream as input, and they produce a token stream as output. This particular one is just hot potatoing the input to the output. It's not a whole lot, but it's a start. Let me back up for a second. There are three types of Rust macros. The first is a derived macro. And a derived macro will automatically generate an implementation of a given trait for the annotated strruct. In this case, I have a derived default on this thing strruct which is going to automatically generate an implementation of the default trait for the thing strct. The second type of macro is a function-like macro and everybody knows the vector macro. It's going to create a vector with all the elements that you pass in. That is a function-like macro and you can invoke these pretty much anywhere that you'd otherwise be able to invoke a Rust function. The third type of macro is an attribute macro. A good example of that is the cfg macro and the cfg macro in this case will decide whether or not to include the subsequent line based on whether the given feature is enabled. In this case that's the do print feature.

[00:01:11]Now there are two ways to implement R macros. The first is declaratively the second is procedurally. And declarative macros are generally considered to be kind of the easier way to implement macros, but they are a little bit more limited in what they can do. With procedural macros, you can do pretty much anything you want. The other limitation of declarative macros is that you can only implement function-like macros. So macros like the vector macro.

[00:01:37]If you want to implement a derive macro or a attribute macro, you have to go the procedural route. In this video, we're going to talk about procedural macros and we're going to talk about implementing function-l like procedural macros. Those are kind of the simplest and the good news is that once you understand the concepts there, it's pretty easy to pivot to something like attribute macros or derived macros. So, let's go ahead and look at that simple macro that we saw earlier, that nothing macro. Delete all this. So, I'm going to do nothing and print line hello world.

[00:02:10]And again, let's see. Import that macro.

[00:02:14]Again, this is just going to hot potato the input to the output. So, the compiled binary here is just going to wind up with a print line hello world.

[00:02:24]Let's run it and verify that's the case.

[00:02:26]Sure enough, we get a hello world, right? But what if we want to do something a little more useful and interesting? Well, it turns out we can do that with only a few more lines of code. Say I want to be able to measure the runtime of a specific block of code.

[00:02:39]So I want to be able to do this time this and pass in print line hello world and I want to tell me how long that took to run with this macro that one. Let me run this and show you what the output looks like before we dive into the code.

[00:02:53]So we get runtime.elapsed 68 micros. So this print line hello world took 68 micros to run. Let's take a look at the code for this macro.

[00:03:04]So this is a little bit more involved but it's still only a handful of lines of code. What's actually going on here is that we're prepending the input with this line of code and we're appending the input with this line of code and the elapse code is printing the elapse time since that instant was created. Token stream implements from stir. So we can actually create a token stream from a string using the parse method. That's what's going on here. So I'm calling parse.

[00:03:34]We're converting both of those lines of code to token streams. And then we're creating an iterator from the first line of code. We're chaining the input onto that iterator. Chaining the elapsed code onto that. And effectively we're sandwiching the input between those two lines of code. And then we're doing a collect. So this into iter is going to give us don't worry too much about the types right now, but calling into iter on a token stream is going to give you an iterator of token trees. We're going to look at that type in a second. And then token stream implements from iterator for an iterator of token trees.

[00:04:09]So you can collect back into another token stream. That's kind of cool. So that's what's going on here. We're collecting back into a token stream and then returning that value. That's how the time this macro works. Pretty straightforward. But there's a few problems with this macro. Number one, it's not what we call hygienic. And hygienic means it makes some assumptions or it may impose in some way on the developer's code. In this case, it's creating an identifier called runtime.

[00:04:34]If the developer has an identifier called runtime in the block of code that they pass to the macro, that's going to create a problem that's going to shadow our original runtime code. Uh the second problem is that we have Rust code in string slices here. So, we're not getting any syntax highlighting when we're writing our Rust code. The next example, we're going to solve one of those problems. Before we dive into that code, I want to thank boot.dev for sponsoring this video. Boot.dev Dev offers what I truly believe to be the optimal approach for improvement. Learn about a topic, do an exercise related to that topic, then have your solution automatically validated. Every time I personally use this approach, I don't know how else to say it. I start to feel like I have this unfair advantage. Every course on boot.dev embraces this approach. Learn, do an exercise, then validate your solution. On top of that, they gify the entire thing. There are achievements, experience points, and a leaderboard. I'm not actually number one. I just edited the DOM node, you know, to see what it would feel like.

[00:05:34]The course catalog includes topics like C, Git, SQL, Docker, cryptography, and much more taught by what I would refer to as celebrity instructors. Something that surprised me is that they actually offer all course content completely free, and you can try interactive features in some of the lessons. Go to boot.dev and use promo code code to the moon for 25% off your entire first year.

[00:05:55]Thank you again to boot.dev for sponsoring this video. This is the same exact macro implemented using the quote crate. And the quote crate solves that syntax highlighting problem among other things. The same two lines of code are here. We have our instant creation and the printing of the elapsed time. And we're interpolating the input in between those two lines of code. Much cleaner than chaining together a bunch of iterators. The other thing we're doing is immediately converting to a different token stream on the first line of the macro. There is a crate called proc macro 2 that is pretty much used in every Rust macro that you're going to see. And proc macro 2 is a drop in replacement for proc macro token stream.

[00:06:39]Why do we need a drop in replacement?

[00:06:40]Well, proc macro token stream is only available in limited contexts. It's only available in proc macro crates which can make it really hard to test and so it's very common to immediately convert to a proc macro 2 token stream right off the bat. The quote macro generates a proc macro 2 token stream as well. So we actually have to convert that to back to a regular token stream before when we return the value from the function.

[00:07:07]Additionally, there is a trait called two tokens in the quote crate. Any implementation of two tokens is something that can be interpolated into the quote macro and there is an implementation of two tokens for proc macro 2 token stream. You can see if I remove this input here which is shadowing the original input parameter.

[00:07:26]If I comment that out we get an issue here we say quote two tokens is not satisfied for proc macro token stream.

[00:07:32]So there's no implementation of two tokens for the original token stream.

[00:07:38]Put that back. Let's go and try this out in our main. So the name of the macro is the same. The difference is crate name.

[00:07:45]Let's try this. And we can see we get the same type of result. 79 microsconds to run that. So that is the quote crate and the proc macro 2 crates. There is another very frequently used crate for procedural macro development called sin.

[00:08:01]That is for the specific case where you know the input to your macro is going to be valid RS code such as when you're writing an attribute macro or a derived macro for something like a function macro. If you're creating a domain specific language that may not be the case and the sin macro may not make sense. So you may or may not need the sin macro. But the quote crate and the proc macro 2 crate pretty essential for procedural macro development. But what's inside this token stream? Up until this point we haven't really looked inside this token stream. We've been either hot potatoing it straight to the output or we've been just sandwiching it between two other lines of code. What if we want our output from the procedural macro to depend on some of the details of what's in that token stream? So, let's take a look at what the token stream looks like. I've written this macro that actually prints the contents of the token stream. Don't worry too much about this code. I just wanted to show you this these four lines. When you get an iterator from a token stream, each element is of type token tree, which is an enum. And there's four variants of the token tree enum. Ident, punct, literal, and group. And we're going to see examples of all those in a second.

[00:09:08]So the way this macro works is I'm going to delete this. I can do token stream print and I can pass in whatever input I want. I can do ABC, 1 2, and then some other stuff, right?

[00:09:22]It does not have to be valid rest code.

[00:09:24]Let me make a use statement for that.

[00:09:26]Delete that one. Let's have a look at the output for this. I'm going to run that. Make this a little bigger. So you can see the first token is A and it's of type ident. B is ID as well. C is short for identifier. Comma is punctuation like you might expect. Anything enclosed in square brackets or curly braces or parentheses is going to be of type group. And group is the only recursive token tree. So a group can have child token trees as well. The other three variants don't have child token trees.

[00:09:59]Group does. And so if we dive into the group, we can see we have a literal, a punctuation, and another literal. And then we're back to the original level, which is all punctuation from there. So that's what a token tree looks like or a token stream looks like. A very, very primitive a. It's not a Rust A. It's a very primitive a that can be adapted for different use cases. If you want a rust a from the token stream, that's when you'd use the sync crate s. Okay, so now that we know what the token stream looks like, let's try doing something a little bit more involved. I want to create a macro hash set that works similar to the way the vector macro works. I wanted to create a hash set from all the elements that I pass in. So, instead of making a vector, I want it to create a hash set.

[00:10:48]What's that going to look like? Okay, first things first. Let's start off simple. Let's just create let's just return an empty hash set. We're not going to worry about the input quite yet. Right off the bat, we're going to use that quote macro. And this is important. I talked about hygiene earlier. Uh, one of the tactics you can use to make sure your macro is hygienic is to put the resulting code in its own scope. So, I'm going to create another scope here. And I'm actually going to create that the new hash set in that scope. So I'm going to do let set equals. And then the other thing you can do to ensure that your macro is hygienic when you're referring to other modules or and other strrus you want to use fully qualified paths to do so. So I'm going to do std collections hash set new.

[00:11:35]This way it doesn't matter whether the developer has already already has a use statement for hash set. If they do that's fine. If they don't that's fine.

[00:11:42]Either way, this is going to work. If I assume that they already have a use statement for hashet and I don't use a fully qualified name, it's going to break if they don't have that use statement. Right? So, I'm going to create the hashet and then immediately I'm going to return it. It's going to be into there. I do need to bring in the quote macro. Okay, so we're creating a hash set. We're ignoring the input. That's fine for now. Let's just make sure this works.

[00:12:10]We have to assign this to something and import this.

[00:12:19]And it's going to ask for a type. It has no way to infer the type currently.

[00:12:23]That's going to change in a second. But for now, I'm just going to specify a type set.

[00:12:33]And I'm going to import it again. Later, I won't have to do that. Give this a run. And we have an empty hash set.

[00:12:40]Perfect. All right, we're on our way.

[00:12:41]And just like that, it's been three full days my time since I created that hashmap. Camera was overheating and then it was a weekend. And yeah, here we are.

[00:12:50]But let's keep going. As we did before in some of the examples, we need to convert the input token stream to a proc macro 2 token stream. So, let's do that.

[00:13:02]And then we're going to create a vector elements.

[00:13:05]We'll assign this a value in a second, but ultimately what's going to happen is we're going to interpolate this elements vector into our quote macro by using this special syntax. So we're going to do pound per pen and then underscore set and then and then close pn asterisk. What this is going to do is repeat whatever's inside the parenthesis for each element of the collection that's interpolated in here.

[00:13:33]In this case, that's the elements vector. So, what we need to do at this point is build up this elements vector.

[00:13:38]And each element of the elements vector needs to implement two tokens so it can be interpolated into the input to the quote macro. Now, remember the input to the macro is a comma delimited list of literals. So, in this case, the first token is going to be the one, the second token is going to be a comma, third token is going to be a two, so on and so forth. So, to do that, we're going to do a filter map operation.

[00:14:04]We're going to see what token is. And if it's a token tree literal, we're going to return sum of that literal. If you're not familiar with filter map, the closure that you pass to filter map, if it returns sum, that's going to be included in the resulting iterator. If it's none, it's going to be excluded from the resulting iterator. So token tree literal is going to be the type if the developer passed in a quoted string or a number. Otherwise, if it's a comma, it's going to be punctuation. We don't care what it is if it's not a literal.

[00:14:35]So, we're just going to do underscore.

[00:14:36]And in that case, we're going to return none. And we're going to do a collect at the end here.

[00:14:44]Import token tree. Oh, we need to import proc macro 2 token stream.

[00:14:52]Okay, that should do it. It's important to note that this macro only accepts literals. The developer can only pass in numbers or quoted strings. It doesn't even support variables. You can't pass in a variable identifier to put the value containing that variable into the hashet. Obviously, you can implement a hashet macro that does that. The code for doing that is a little bit more verbose, a little bit too long for a YouTube video like this. So, we're implementing kind of the poor man's version of the hashet macro. Oh, one more thing. We forgot to make set mutable. So, let's test this out here.

[00:15:26]And we get our 1 2 3 4 elements as we expect. We can also remove the type now that we get the the type inference from those insert statements that this compiles to. The compiler can infer the type of the hash set based on the things that we're passing to those insert method calls. So we can remove the type.

[00:15:49]Note actually that we're not enforcing the punctuation though. can actually go and remove these commas as long as there's a space between the the numbers.

[00:15:58]Um, this is still going to work. So, you can see this still works. We could, if we wanted to, if we go back into the implementation, if we wanted to, we can update this match block to actually check that there's commas between the input elements, but I'll leave that as an exercise. That would make the code a little bit more verbose. There you have it. That is a quick introduction to Rust procedural macros. These are some very very basic examples, but they serve as a good foundation for taking things to the next level. Maybe going to attribute macros or derived macros. The basic concepts are pretty much the same.

[00:16:32]There's some differences like for an attribute macro, you have two input token streams instead of one, for example, but the basic idea is the same.

[00:16:40]You're taking some tokens as input, producing some tokens as output, and of course, the output needs to be valid Rust code. If you like this video, definitely check out this other video on Rayon. It is a very, very easy way to speed up your Rust applications if they are compute heavy. Thank you again to boot.dev for sponsoring this video.

[00:16:58]Thank you all for watching and we'll see you in the next one.