Building ChatGPT in Rust: Part 3

[00:00:07]Hey everyone, welcome on in. Happy Friday. We made it to the weekend, you guys. We're going to be continuing our GBT that we're building in Rust today.

[00:00:16]We're building in Rust and this we need to do a lot of things. Don't worry, there's there's plenty of things that we have to do before we are successful here. We need to continue building. We built yesterday. Where did we do yesterday? So, we built the initial the initial network. We got the initial network set up, which is very satisfying. I'm very very happy about that. What we need to do next is start pushing data forward and seeing what we can get as the output and then we're just [music] going to incrementally build from there. I think that's the plan. I think that's a good plan. I like that plan. Where are we right now? So, where did we leave up here? All right.

[00:00:55]So, run. Let's see where we're at here.

[00:00:58]Okay. Oh. Oh, right. Right. Right. Okay.

[00:01:02]So, we built our forward here which includes our tokenizer. We added ourselves a dictionary.

[00:01:09]Right. Wait, wait, wait. Where did we get the dictionary? Oh, did we make it inside here? Did we build the tokenizer directly here? Where where did we do this? So, this is the transformer and then we're pushing it through a forward.

[00:01:22]And I'm seeing tokens on the output.

[00:01:26]So where where is that happening?

[00:01:31]>> Oh, dictionary right here. Okay, >> so the transformer uh right here. Okay, that's good. Are we saving it? We're saving it right here. Okay, so let's put this in let's put everything in the right order. Tokenizer embedding, positional encoding, projections, blocks, and then output projection. We are missing norms. We're missing norms. So, it's to do add layer norms, which we need in different places. We need multiple layer norms. I need to remember where we [music] put cuz we put a lot of we put like four layer norms in or maybe it was three. Let's find out.

[00:02:14]So, CD desktop uh CD pi was it transformer transformer pietorch main? No, transformer.

[00:02:25]Okay.

[00:02:26]>> Where did we put the norms? Norm >> no norm. Layer norm. Layer norm. It's called layer norm. Right.

[00:02:34]>> Where's our layer norm? Layer. Did I I know I know we have a lot. Why can't I find a single one? We should be able to find a [music] single one here.

[00:02:45]>> Okay. Where are we at? Where are we at?

[00:02:47]>> Where are our This is like find find the adventure here. Where is our layer norm?

[00:02:56]Hold on. Wait. Hold.

[00:02:58]Positional swiggloo dictionary transform. Wait. Do we have the GP? This isn't the right one. Okay. This uh custom transformer. There we go. And now we go. Layer norms. All right. So, we had three layer norms. We got to add those. And where did we do these at? On the forward initially. Okay. So immediately on the forward. So before the projections. Okay. So we need a layer norm before the projections. So right here layer norm before projections. Okay. And then we needed where was the other one? Then layerm two. [laughter] Okay. I remember this now. All right.

[00:03:46]So we had a little bit of extra things that we had to take care take care of here. All right. So the attention after attention we need a norm and then we needed a norm after that. The thing is here's the thing this is this is important. We needed to combine the attention with the original inputs.

[00:04:08]And the original input is going to be tokens.

[00:04:14]So when when we're and let me think let me think of how to do this cuz we're we're architecting this a little bit differently. It's just a smidge different.

[00:04:24]>> So [music] we have to figure this out.

[00:04:27]>> Uh where was this at? So what are inputs here? I think these are obviously if these are the projections then the inputs are going to be the uh everything the positional encodings. So, embeddings and then positions, right? I think that's it. I'm pretty sure that's it. Okay. So, if that's the case, then we can simply just All right, let me let me let me just copy and paste this really quick into our so I have this as reference.

[00:05:00]Here we go. All right. So, let me paste this over here. Hey, Bonupi. Good to see you. Happy Friday. Welcome on in. Howdy.

[00:05:08]We are We're continuing on our adventure here. Building a Let's see. Get rid of that. Oh. Oh, nope. Not that either. Did I not copy and paste this? Probably. I probably didn't. [music] There we go. Okay. Paste. Whoa. I forgot that.

[00:05:25]>> There we go. Okay.

[00:05:27]>> You're really the first one here. Is chat really empty? I know. What happened? Where did all chat go? Where are my stuff bugging? No, you're really You're really the first one here. I just been coding. I've just been coding away.

[00:05:39]>> I don't know what happened. I don't know. Uh, good to have you here though, Vanzupi.

[00:05:44]Okay, so transformer, [music] where are we here? So, I need I need to I uh one sec. Yes. Okay, right here.

[00:05:54]Okay. Tokenizer, embedding, positional encoding, and we need a layer before. We know that part. However, was there something else after? There was. There's the two other layer here. Post attention.

[00:06:06]Okay.

[00:06:07]So, we need two other layer norms here.

[00:06:10]>> There we go.

[00:06:12]>> There we go. Okay.

[00:06:14]>> Let's get rid of this here. There we go.

[00:06:18]Okay. All right. So, that's for later.

[00:06:21]So, we're good there. You're fully prepared to blame a colonel misconfiguration for the emptiness of chat. You were [laughter] I love that bun.

[00:06:33]Uh, good news. Good news. everything that you've been doing. You're you're good. You're good. You're good to go.

[00:06:39]You're good to go. Bonzupi, you're doing it correct. You are on the right path.

[00:06:45]>> Okay.

[00:06:47]So, let's see here. Let's see here.

[00:06:48]Where are we at? Let's go a step further. So, here's our tokens. Let's do our embeddings now. So, I want to do embeddings. Let's say [music] embedding. Embeding. Yeah, the embedding is going to be tokens embedding. Let's return the embeddings. Let's see here.

[00:07:04]You've been fine-tuning a custom kernel configuration and all sorts of weird stuff is broken now. Wait, I'm kind of curious. What are some of the kernel configurations that you've tuned? Oh, what are they? What do you tune him? I'm curious. Let's see if we can do this. Is that going to build and run? You never know. You never know. All right. So, this is the original Pi. Let's see. Let's call this the Python transformer. Hey, Hari. Good to see you. Welcome on in. Hello, bro.

[00:07:34]Good to have you here. Happy Friday. You made it to the weekend. Transformer.

[00:07:40]Okay, then we want to go back to our build, which is right here. Okay. See if that runs. Okay, we got a problem immediately. It only took a second.

[00:07:50][laughter] Only took a second.

[00:07:56]All right, bro. Where are you from? I am from Seattle, Washington, United States.

[00:08:00]Thank you for asking. Good to have you here. Uh we're currently building chat GPT in Rust.

[00:08:06]Uh let's see. Tokenizer length value is used here after move. Uh oh. Do wait.

[00:08:15]But it's Hold the phone. Do I really do I need to just can I just It's just It's just an I64.

[00:08:23][laughter] It's an I64.

[00:08:26]You're from India in Kerala. Kerala. All right. Welcome on in. Good to have you here. Happy Friday. Let's see here, you guys. All right. Uh, transformer block.

[00:08:39]This is going to be a fun coding day. I think we're going to have a good coding day today. I am I'm I'm here for it. All right. So, you said in our Let's see here. Let me I need to open up another file now. Source the file of the to Did I already have the token? I don't have the tokenizer here. See, look. It's a length. It's a parameter. It's right there. It should be perfectly fine. I don't know what's going on. [laughter] It used it was already used. That's crazy. Everything recompiled the kernel CN V LLVM instead of GNU C compiler as well as the Nvidia kernel modules. I also replaced the GNU core utils with freeBSD Chimera Linux core.

[00:09:22]You are building something. What are you doing over there Bonzupi? Happy Friday.

[00:09:28]Hey J bro, good to see you also from Carerala or uh Carala Carerala. [music] Good to have you here J bro. Welcome on in. All right, what are we doing here?

[00:09:39]What? So my length should been perfectly fine. And [music] also why is it saying that it does a oh on the embedding uh embedding embed. So when I use it that's value used here after move. So I mean do you want to copy it? We're just going to copy that. All we have to do is say copied, right? I think that's what it is. We just said copied, right? So, where is that at here? Let me close this. And then disabled everything.

[00:10:06]Didn't use include AMD. Apparently, stuff you do you got to use it because boot frame buffer driver isn't working causing a a screen that is not there.

[00:10:18]Okay, sir. Hurry. Sounds good. Yep.

[00:10:21]Hurry. That's the plan. That is the plan. That's what we're doing right now.

[00:10:25]Uh my problem is here. This is where the problem is right now. And but why though? Why is it copied? Can I do that?

[00:10:35]Is that is it is it going to is it going to be happy if I do that? [music] No. Wait. Oh, wait. I 64 is not an iterate. I No. All right.

[00:10:44]Uh what what did this want to be? What is it? It's an I64, right? Yeah, it's an I64. It's right there. Moab link.

[00:10:53]So, oh, can we say let vocab equals here?

[00:11:02]Vocab there. Uhhuh. There we go. Almost there. Your kernel binary is 40 megabytes. Whoa. Hey, Squirrel Sniper.

[00:11:10]Good to see you. Welcome on in. Happy to have you here. Friday. Good Friday.

[00:11:14]Bunch of legacy harder modules. Disables your Wi-Fi drivers. Oh, you couldn't use any of your peripherals.

[00:11:21]Oh, that is rough.

[00:11:23]That should be fine right there. That should be fine. Vocab, right? Vocab.

[00:11:29]Embedding. Oh, apparently that did the trick.

[00:11:34]Wait. [music] Self. Help remove the arguments field not a method.

[00:11:43]Is it? Oh, we have to say forward. I know. I I know what's going on here. All right. I messed it up. Wait. No, wait.

[00:11:50]No, no, no. Not here. Uh down lower right here. Okay. Forward.

[00:11:56]All right. Let's try that. Let's try that. Let's see if that did the trick.

[00:12:01]Expected a address to a tensor. All right. So, we got the actual tensor there.

[00:12:09]All right.

[00:12:11]OH, LOOK AT THAT. WE did it. [laughter] Oh, hey, that's our first output from our rustbased chat GPT model. Oh, look at that. All right, we're making it work. Hey, Ruchet. Good to see you. Hey, man. Welcome on in. H happy Friday.

[00:12:29]Greetings from across the pond. Hey, Stephen. Hey, class. Squirrel snipers.

[00:12:33]It's good to have you here. Thank you for for visiting us today on Friday.

[00:12:38]Hope you're having a good weekend start here pretty soon. Okay, so I'm really happy about this output. Look at that.

[00:12:44]Oh, it's so good. That's so great. Okay, so we did our first. Can we do our second? So, this is going to be our embedding. Now, we need to do our positions. Let positions equals uh which is going to be a tensor. And we're going to say self.positional encoding.forward dot embedding. There we go. Uh-huh. And we're going to return positions little little steps at a time. Little steps at a time. Okay. Okay. Okay. Uh oh, wait. Right. Right. Right. Address.

[00:13:16]We're going to pass it as a reference.

[00:13:20]Hey, what is this? That looks like a problem. Uh-oh. Wait, no. Fair. What happened? Okay, we're we're going to Oh, called unwrap. I didn't call unwrap anywhere. Where did I call unwrap? Where did I do that? I did that right there.

[00:13:38]All right. Did I'm embedding right here.

[00:13:41]Okay. Self embed. Oh, wait. Maybe maybe this right here is not what I thought it was going to be. So I need to figure out what this is. Is this actually a tensor or is it an optional tensor? Do you guys know? I don't know.

[00:13:56]I don't know. Early boot frame buffer isn't working. So it can decrypt the Luke the Luke encrypted drive because it never gets the password prompt. Uh, Bonsupi.

[00:14:09]Bonzupi the That sounds like a major problem. It does. You can get it. I think you can get it. How's it going there, Samara? Welcome on in.

[00:14:18]Congratulations. Oh, thank you. Hey, Snot. Good to see you. Welcome on in.

[00:14:21]Happy Friday. New York City voice sounds OP. The OP New York City. I think New York City is okay. I think they have pretty good OB. Hey, Epic Blox. Good to see you as well. Happy Friday. Hello.

[00:14:34]Oh, heard you need a binary if you learning machine learning. Is it true?

[00:14:39]You don't need Well, not really. You don't need to know binary. Bonzupi. Oh, wait. Epic Bloxow. You don't I don't think so. I is It's helpful maybe. Is it? I don't even think binary is much involved in AI. It's mostly coefficients. It's all coefficients [laughter] and [music] weights and other, right?

[00:15:00]It's all numbers. Binary is numbers as well.

[00:15:04]Uh, what do you what tokenizer are we using? Hey, we built our own. Yeah, we did. You want to see it? You want to see our tokenizer? We built it a couple days ago. Here, I'll show you. Let me go by source tokenizer right here. Right there is our parser. And then we've got our encoder decoder right there. We've got our own padding, [music] special padding token right there. Right. So, we can ignore things if we don't want to.

[00:15:32]Right. Here's our encoding. And then we didn't do batch on the output also. Do we need to do batch here? No. Maybe.

[00:15:44]Yes. Okay. To do we got to do to-do batch. Got to do batch right there as well. [music] Using custom kernel satisfying. Oh, it's so good. It's so good. VPN speed power strong. Wait. VPN [music] speed power strong. Power in. What do you mean? Well, what do you mean?

[00:16:03]Nyx OS allows you to just boot into whichever version of the kernel you want. As long as you don't clear the unknown good kernels from your cache, you can roll back and work with it. Oh, I didn't even know about that. That's nice. How's it going there, the forts?

[00:16:16]Yo, bro, good to see you. Welcome on in.

[00:16:18]Happy Friday. Yeah, this is our this is our our custom tokenizer and we built it right here. You guys, we wrote this on stream live on streaming guys. Yesterday was tokenizer. Now continue with GPT.

[00:16:32]You got it, Ruchi. Exactly.

[00:16:35]You got it. Can I suggest you a better tokenizer? Uh, there's definitely going to be a better tokenizer.

[00:16:42]You are right. Are you doing bite pair encoding? Hello, Mr. Bum. Hey, Erdom.

[00:16:47]Good to see you. Happy Friday. Good to have you here. Welcome on in. All right.

[00:16:52]So, here here you go, you guys. Yeah, absolutely. Amzad, tell me tell me your favorite tokenizer.

[00:16:59]Tell me what you like. What's your favorite tokenizer? All right, so we have a problem here. Can we say does this return? What is embedding? What is this return? Do you guys know? Do you guys know what this returns? Let's see.

[00:17:14]What is the forward return on this?

[00:17:16]Let's find out. All right. So, Google uh not that uh Google paste uh forward.

[00:17:27]What returns?

[00:17:29]I think we can say that. There we go.

[00:17:32]There we go. Okay.

[00:17:34]All right. What do we got? What [music] do we got? What does it return? What does it return? Uh. All right. Uh, you are you coding without AI? Hey, dog.

[00:17:43]Yes, we are. We are coding without AI.

[00:17:45]Absolutely. You got it. Well, I mean, sort of. I mean, obviously we just kind of started using AI just right now. I love it. We're looking things up. We're looking things up. Is it a tensor?

[00:17:57]Is it a tensor? That's what I'm asking right here. No, I need I need to see what it returns. Uh, TCH Rust. What returns?

[00:18:06]All right.

[00:18:08]Completed PyTorch basics today. Hey, very nice. All right. Implemented neural networks. You did it. Good job. Good job.

[00:18:16]Read this full man because yesterday was tokenizer. Now in continue of GPT. Now we are in multiverse of Steven [laughter] Ruchi. Yeah, I see what you're talking about there. Yep. Use tokenizer. The tick tokenizer with GPT encoding. Ah, the tick tokenizer. That's going to be the one to use, right? All right. Here, I'll put it on our to-dos. I'll put it on our to-dos here. All right. Where is this at? Here. Is there a main? All right. Let's go over here. I'll make a new one. All right. Use tick tokenizer with GBT encoding. All right. Amzad Hussein.

[00:19:02]Thank you. 8118. Okay, there we go. All right, we got it written down. It's written down now. Of the bite stream remains identical. Oh, when tokenizer encounters a typo like computer, it splits the word completely into different subwords. Thank you for the heart, you guys. Yep. Bite level model process. Squirrel sniper. Yes, 100%. I see what you're doing there. Oh, hey. Uh there there is no way. There is no way.

[00:19:28]How's it going? Happy Friday. All right you guys, for those of you who are new here, my name is Steven Blum, CTO at PubNub. We raised 134 million series E 1 billion devices. That's what there is no way is talking about in chat right now 68 patent. A fan of GNU. Oh, the GNU.

[00:19:46]The GNU. [music] Uh how can you talk live with captions?

[00:19:50]I know, right? It's pretty amazing technology, isn't it? [laughter] Hey, I know [music] free BSD. Oh, Bonzupi. Okay, so you guys Yeah. So, you asked how we raised this is what obviously we have revenue and [music] we have a growth rate of revenue and then you say, "Here's our growth rate. Here's our revenue. Would you like to invest in this company?" And they say, "Okay, that's what it is right there. It's really easy." I mean, obviously, you have to get the revenue parts the hard part. Once you have that though, you could do venture capital. And we had a growth of so we've got over a billion devices right like these mobile devices we are an API communications provider that allows in-app communication things like talking to your doctor on your phone tele medicine right ordering food on demand delivery that the economy that you press a button on your phone and something comes to you that's us we do that communication the top 10 apps in the app store already have built-in app communication that's what we do you guys that's what we you. You're so positive.

[00:20:55]Keep on going. I admire you. Oh, thank you so much. I appreciate it. That's really nice. What you're doing sounds like it would be of great use. Ooh, yeah. How much profit? A a lot of profit. A lot of profit. You've got to have a lot of profit.

[00:21:11]Lots of profit, you guys. You have good dentist. Yes. Overall, yep. Overall opposites. Thank you. Appreciate it.

[00:21:17]Yep. Been through many orthodontics.

[00:21:20]Lots of orthodontics, you guys. Let's see. The Pixstruct render text to pixels and process it through a vision transformer and eliminates the tokenization flaws entirely. It natively handles layout. Oh, squirrel sniper. I like what you're talking about. I like that idea. That sounds very satisfying.

[00:21:40]It takes all the text on the screen and it converts it into a 2D matrix, right?

[00:21:45]Or essentially, right? or some sort of some sort of along the line creating [music] pixels turning into pixels so the AI can understand what it's looking at visually rather than through tokens.

[00:21:57]A like that is a really good idea. That is [music] so smart. I like it. Samara friends. Oh, friends indeed. You got it.

[00:22:08]You got it. You guys, have you uh see uh tried KB caching? Oh, Amzad. Yes.

[00:22:14][music] Uh we we looked into it. We have not tested it out though. We've not [music] tried it yet. And KV caching, tell me if you know about this. With a KV cache, can you help me help me help me learn this? When you're inputting a token into the model, do you input it one token at a time, right? And then you sort of collect the tokens in a cache and the projections already have the cash. You project over and then you pass it into a concatenation on the cache, right? Is that what that is? One token at a time. Do you know it's gen Genius idea? It's the matrix.

[00:22:52]Oo, the matrix. Yeah, your country. What? Your country. Uh, I don't know what that is. Some I don't know what you're saying over there.

[00:23:02]Okay. So, let's continue, you guys. I need to figure out why it it broke. So, what happened here? So, we got a uh it didn't work. Some reason here. Let's find out. Let's see if we can learn more. Let's see if we can learn more about this. It panicked on an unwrap error value. Expect a tensor from argument number one indices have one of the following scalar types. Oh, long. Oh, but got float. Oh, we can fix that. I know how to fix that. Hey, here we go. Okay, you guys ready? All right, so our tensor type was of float. Our tensor type needs to be a different dype. It needs to be long.

[00:23:45]It's got to be long, you guys. All right, let's let's keep going. Let's keep going. All right, here's a problem.

[00:23:50]You run into it like this. Expect the tensor from argument indices to have the following scaler types. Long int but not float.

[00:23:57]So, we have [music] to turn it into a float or a long. We got to turn it into a long. That's the answer right there.

[00:24:03]All right. So, how do we do that? When we do the forward, we need to Well, we're throwing Oh, okay. So the linear layer I need to pass in a D type here.

[00:24:16]How do we how do we add the data type here? Do [music] you guys know we need to do that? We need to set the DT type on this linear layer. Not not this linear layer on the embedding. No, not in the bedding on the Wait, hold on. The tokenizer on my tokenizer. Okay, so not in here. We got to go. We got to go somewhere different, you guys. All right, let me get the next song going quick.

[00:24:46]Here we go. Excellent. Excellent. How do you write bold to-do?

[00:24:52]Oh, good question. Right. So, you see how there's a to-do here? Right here.

[00:24:56]That to-do on the screen right there.

[00:24:58][music] To-do. It's automatic. My text editor does this for me. To do. And if I if I type it wrong, it won't to do it, right? So, it's got to be all to-do like that. Right. Right there.

[00:25:11]Got to make it a little bit extra.

[00:25:13]Right. Little bit extra. Okay. I need to open up the tokenizer source.kenizer.

[00:25:21][music] And when I return a tensor, I need this tensor right here to be what do you call it? LA a from tokens it. Uh, it's already an I64.

[00:25:37]So, [laughter] thank you for the hearts, you guys.

[00:25:40]Appreciate it. From Slice, it needs to be. It already is I64.

[00:25:46]It already is an I64.

[00:25:49]It's already that way. Okay.

[00:25:52]So, [music] let me see here.

[00:25:55]Not a crime. Just world's love. Just the love, you guys. Thank you for the hearts. Appreciate it. Thank you so much for the reactions. All right. So, we got our transformer. [music] And then this is on the forward here is where we're having the issue right here on the embedding right on the embedding.

[00:26:12]It's got to [music] be token. So this problem here. So let me see. Let me turn that off [music] for a second. Oh, two embeddings. Oh, right. I see. Okay.

[00:26:25]I'm doing I'm applying it wrong. That's what it is. I'm applying it wrong. Thank you for the hearts, you guys. Oh, thank you for all the reactions. This is so good to have you here. You guys are so nice. Here we go. Okay, so outputs look good. What we [music] need next? You've seen recent video of Nvidia CEO sign 1080 Ti with emotions. This was the best I know of 1080 Ti. I didn't see it.

[00:26:52]Route. I did not see it. I'm curious about it though. So when we apply our positional encodings, we have to do that [music] differently, right? So we have to with the embed. How do [music] we how do we do this? Um I would like to do this. I really would. [music] Uh because in order to do this properly, we have to do how did I do that here? Let's go to five.

[00:27:20]Okay.

[00:27:22]Position.

[00:27:23]All right. So we get our positions.

[00:27:27]There's our positional encoding. All right. So, we need to do something like this.

[00:27:31]All right. Let's bring this back over here.

[00:27:36]Uh-huh. Uh-huh. Uh-huh. Close [music] the tokenizer.

[00:27:40]Set. Paste.

[00:27:42]Go. Hide all [music] these things here.

[00:27:45]So, because they're not they're not valid Rust syntax.

[00:27:50]Okay. So [music] with our interesting interesting okay so what we could do is we could create our own positional encoding [music] module that has an embedding and then we could do that here. Let's just let's just do that. All right. So let's do strruct and we don't need to make this public positional encoding uh strct positional encoding and then we're going to have our embedding and that is going to be identical to this embedding here. Where is it? Right there.

[00:28:34]There we go. And is it all that what we need? Yeah, I think that's it. And then and then we just imple say new uh function new my Oh, hey. How's it going there, Joe? Good to see you, Joe.

[00:28:53]Happy Friday. Welcome on in. How's it going? We're we're doing good. Thank you for for uh having you here. Thank you for stopping on by and saying hello.

[00:29:00]Good to have you here. Okay. new uh let's see we don't do we need do we need information yes we do we need the dimensions all right dims as i64 uh and then max tokens here we go equals 5,000 more than we need a lot more than we need also this needs to be an i6 is there Is there a way to do defaults in Rust? You guys know?

[00:29:36]Wait, are you? Oh, Deero. No, I'm I'm just a regular man. I'm a a man in my 40s. I'm old. I'm old. Hey, Bruce Lee.

[00:29:44]Yo, bro. What's this? We're building chat GBT and Rust. Rust is a computer programming language that is very very fast and rather complex in many situations. We're combining the most difficult things I would say in computer world.

[00:29:59]>> [music] >> Rust and artificial intelligence. Those two things combined very difficult.

[00:30:05]Very difficult you guys.

[00:30:08]All right. The transformer initialization block. You need to initialize a dedicated embedding layer for positions. Yeah. Setting the maximum sequence length and the hidden dimensions of Yes, that's exactly what we're doing. Squirrel sniper. Yep. Nice work. Keep it up. Thank you, Bruce Lee.

[00:30:23]Appreciate it. Appreciate you. Thank you.

[00:30:28]We have a very fun adventure ahead of us here. All right. So, uh, is there a way to default assign values in a function?

[00:30:38]Can you do that? I don't know if you can. I think you should be able to. I don't know how to do it. Let's see. All right. Here, let's do this really quick.

[00:30:46]All right. Google set default values in args function args function args rust see if that works.

[00:31:03]Oh no you could do something else. You could do something else. There's some other approaches you could do.

[00:31:11]Common outline logic generates sequential range tensor sequence length passes it through a positional embedding layer unsqueezes it to the match the ba batch dimension. Yes, we got to match the batch dimension. Yep. Adds directly to the token embeddings. Yep, you got it. It's a It's a addition. It's a real simple addition.

[00:31:32]Let's see how we go. Option optional string. Okay. Oh, okay. So, we'll just do an option. Here we go. So, this will be option option N I64.

[00:31:51]And then we say let max tokens I64 equal to max [music] tokens. unwrap or we set it to 5,000.

[00:32:07]Boom. Like that. Is that the way to do it? Pass position through embedding. Add batch dimension position embedding equals. Uh-huh. Unsqueeze zero. You got to unsqueeze. Unsqueezing is going to remove. It's going to remove one of the dimensions. Combine token semantics matrix additional combinations equals embedding. Yep. Squirrel sniper, you got it. Hey, the Xbox, how's it going? Good to see you. Happy Friday.

[00:32:36]Thank you so much for your chat message there. Xbox 720.

[00:32:42]I like that. I like that Xbox 720 cuz they had the Xbox 360.

[00:32:47]Okay, so I think that's it right there.

[00:32:50]We need to uh wait. here [music] uh self and then we say return [music] self like I like to say return I know we don't have to I just want to I want to return self all right max tokens dims wait what do we do did do did do did do did do did do did do did do did do did do did do did do we call do do we want to yes no no no no this is not what we want we don't want to do it this way we want to do embedding here we go embedding heading is going [music] to equal this. Yeah, there we go. Like that.

[00:33:30]There we go. Perfect.

[00:33:32]Excellent. Excellent.

[00:33:35]And this will be max tokens. What? No, this is wrong. Is this right? No, I got this has to be dims max tokens.

[00:33:43]Like that, I think. I'm pretty sure.

[00:33:46]We'll call this positions.

[00:33:48]Positions.

[00:33:50]There we go. Excellent. And I see we've got that there though. We are just going to not have it. We're not going to do that. Do we need We do. We still need the positional encoding though. So we have to do this differently. We need to say uh positional encoding new. Let me say dims v [music] uh max uh just dims. That's it right there.

[00:34:22]Dims, right? I think that's it. [music] That's the end. That's the answer.

[00:34:25]That's it.

[00:34:28]You're You are a manual tester getting underpaid salary fresher but doing an experienced people work although easy still underpaid. I deal with deployment in the pipeline. Oo. So [music] continuous deployment right. CI/CD continuous integration community.

[00:34:47]Postman for functional validation. Oo yeah, manual validation with Postman.

[00:34:53]Nice. And JMeter. Oh, the Java world.

[00:34:57]Hey, you're in the Java ecosystem.

[00:34:59]Are you doing positional encoding via cosign and sign? Oh, thank you for asking. I'm going to do the chat GPT2 approach where we're using an embedding for our positional encoding cuz it's it's easy.

[00:35:13]It's the easy option, right? the GPT2 [music] that approach it allowed the positions to get learned automatically.

[00:35:21]So we are using an embedding which is weird I know right it's weird why would you do that well it works it works so we just wrote our positional encoding and then we needed uh function forward [music] and then we need to have inputs it's going to be a tensor and then [music] we need to do something here return inputs for now we have to do something else here to do add position S.

[00:35:53]Okay. All right. Where are we at here?

[00:35:55][music] So, I just created a positional encoding. We created a new one which dims max tokens. The input will be right. So, we need an we need it to support the input of the dimensions which is going to be correct because the embedding layer is going [music] to give us the output of dimensions. Perfect.

[00:36:18]Perfect. All right. That looks [music] good to me. Oh, wait, wait, wait, wait.

[00:36:21]Uh, yeah. No, no, this is fine. Okay, so we need one. We need something else here. [music] We're missing a thing. We need to add in the vss, which is a [music] and path. Is it an and path or just a regular path? It's an ant path.

[00:36:35]Okay. And then we [music] need to pass that over here. VS like that.

[00:36:43]Perfect. Okay, that looks good to me.

[00:36:46]And then we can do a forward pass [music] on this.

[00:36:50]Looks good. All right. Forward.

[00:36:53]Excellent. So, let me see if this builds.

[00:36:58]Uh, see, deactivate.

[00:37:01]Deactivate source. Uh, V info iron bin. Activate.

[00:37:08]And then I wanted to run. Can I run without here? Let me grab this here really quick. Let's grab my warnings. Okay, here. Let's just let's just let's just get the bigger screen anyway. Uh, struck is not supported trait implements. Uh, struck. Oh, did I do did I type it wrong? Did I type it wrong?

[00:37:33]That looks good to me. Is that right?

[00:37:36]Yeah. Strruct. And then I name whatever whatever I want to call it.

[00:37:42]Strruct. Okay.

[00:37:44]Uh, is this good? Hold on.

[00:37:51]Am I doing something weird here?

[00:37:56]Okay, compare the two.

[00:37:59]That looks fine.

[00:38:01]I don't see anything wrong with that.

[00:38:03]Maybe uh Oh, okay. Let me see here.

[00:38:07]So, what is this here? Traits [music] or impulse struck. Consider moving the strruct to to a nearby module. I don't want I want it in line right there.

[00:38:16]>> Coolest coder.

[00:38:17]>> Hey, Sergio. Thank you for the $1.99.

[00:38:22]Steven is the coolest coder. Oh, thank you, Sergio. Sergio.

[00:38:27]Sergio. Thank you.

[00:38:31]The [music] Thank you. Here we go. Thank you, Sergio. My gratitudes to you. Thank you so much. Good to have you here. Happy Friday. teach me from scratch what you've done. Oh, uh, Dom, thank you again, Sergio. That's so nice for you. I I was not expecting that. Thank you.

[00:38:50]What a great Friday present. Thank you.

[00:38:53]Teach me from scratch. Uh, what do you want to uh There's a lot to teach.

[00:38:59][laughter] That might take a few days. What What are you looking for? What do you want to learn? What are you interested in learning? All right. So, uh, let me see [music] here. All right. Let me pull this up higher maybe.

[00:39:13]Does it want this to be higher? Like what is this here?

[00:39:18]Here we go. Oh, wait. Let's grab this here. Positional encodings. Bring it all together. Oh, I put it in the weird spot. That's why.

[00:39:27][laughter] Whoops. I'm just putting stuff wherever I want. That's what's going on.

[00:39:33]Hey, Ankit. Good to see you. Hey, Stephen. My first exam was just perfect.

[00:39:37]Wish me luck for the rest. Hey, congratulations on your first. You'll do great on the rest. Good luck. Good luck.

[00:39:43]You'll make it. You'll make it. You're going to do great.

[00:39:48]All you got to do is persevere.

[00:39:51]Got to persevere. Just keep on going.

[00:39:55]Keep on going and you will make it.

[00:39:56]You'll make it. All right. So, that will fix that. Now, we got a new error. Uh, let's see here.

[00:40:05]Is that Let me double check. Let [music] me double check. All right. Got a transformer, a pub. Looks good.

[00:40:10]Implement the block. Yeah. Uh-huh. We get our attention. We did a lot of work so far. This is great.

[00:40:18]All right. Uh, looking good here.

[00:40:22]Syntax is good. Strruct positional encoding.

[00:40:27]Yeah, right there. Looks good to me.

[00:40:29]Okay. All right. All right. You guys help me figure out why it's not doing what it wants to do.

[00:40:38]Use of undeclared type positional encoding. It is it's I used What do you mean undeclared type?

[00:40:46]I I declared it. It's right there.

[00:40:50]Start consider. So do maybe what maybe what's going on is I've got some weird syntax somewhere. So let's let's see if we can fix that really quick. This is line 53.

[00:41:06]Okay. Uh oh.

[00:41:11]Right. Okay.

[00:41:16]Got to do that.

[00:41:18]Excellent. All right.

[00:41:21]Figure things [music] out here.

[00:41:24]How about that now? A little bit better.

[00:41:27]Do we need to Let's see. Let's Let's move this up higher. I I suppose maybe it should.

[00:41:36]This is weird. Oh, wait. Is there transformer? Oh, hold on. One second.

[00:41:45]All right. Well, that's a problem.

[00:41:48]All right. One second. I I must have like a a random semicolon somewhere. All right. All right. How about now? How about now? Are we better now? Did we do it?

[00:42:00]All right. Yeah. Okay. Unexpected. Yeah.

[00:42:01]Semicolon. Yep. Hey, Dorail. Thank you for saying hi again. Appreciate it. Good to have you here. Thank you for clicking the hi button.

[00:42:10]Okay, let me just close that so I can see this a little bit better.

[00:42:15][music] Yep. Yep. Looks good to me.

[00:42:18]Happy. Oh, wait. Right here is the [music] problem. No. Oh. Oh, I broke it.

[00:42:26]This is where our problem [music] is.

[00:42:27]Right here. Where did our implement?

[00:42:32]Where did our implement go?

[00:42:36]I must have I must have I must have got too excited. I got too excited right here.

[00:42:43]This is going to be our transformer block.

[00:42:48]Did I delete it by accident?

[00:42:52]Oh, here.

[00:42:54]Wait. So, I already got my transformer block here. What is this? Uh what is this forward for then? [music] What did I do? I I made it weird. I made it really weird. Okay. Encoding transformer itself, right? Oh, so the transformer.

[00:43:15]Okay. I see. I see. I see.

[00:43:19]I move stuff around very oddly.

[00:43:24]I don't remember doing that. All right.

[00:43:25]We're going to fix this up. We'll fix this. Also, there doesn't need to be a Yeah, there does right there. Okay, that is what we needed [music] to do.

[00:43:35]Okay, how about that? Are we better now?

[00:43:40]Uh, a little bit better. Expected nothing but found a tensor. Oh, right.

[00:43:45]Okay, so right, right, right. This is on line 98. Okay, right here we will return a tensor.

[00:43:56]Fix that. Okay.

[00:43:59]All right. Transformer block redefined.

[00:44:02]Uh, I must have done something real weird.

[00:44:07]All right. Transformer block.

[00:44:12][music] Transformer. Where's our transformer block? Oh, wait.

[00:44:18]Look, it's it's doubled up.

[00:44:21]This is weird.

[00:44:23]All right. Uh, however, that worked. All right.

[00:44:29]Okay. We're making we're a little bit at a time. This is great. All right.

[00:44:32]Positional encoding arguments I 64 option is missing. Well, yeah, cuz it's optional.

[00:44:38]It's an optional value, right?

[00:44:43]Do we say none? I guess we just say none. Uh, none. Okay. All right.

[00:44:54]Uh let's see here what you done as system architect in your day and to today life at loyal. Oh your previous org mentioned in LinkedIn and tech lead [music] as Microsoft. Uh just it sounds complicated and it it really is just application [music] development is all it was. It was application development specifically web app development. Back then there was no React, there was no nextJS.

[00:45:20]There was no Angular.

[00:45:23]There was there was barely any JavaScript frameworks. There was no backbone even. [music] That was way before then. So we had to write all the code ourselves and wrote our own libraries from the front end. And all the browser vendors were were fragmented. You had Mozilla, you had Fire Mozilla Firefox, Internet Explorer, Netscape, Opera, and Google Chrome was just coming out, [music] right? web in Safari, we had to write code that worked differently in each one of those browsers back then. So we had to create frameworks that have scaffolding and then [music] we had to make sure that the backend was performant and scalable before you know no one knew about sharding.

[00:46:08]No one knew about sharding. Sharding is where you take data and split it across multiple servers. So this is all stuff that we had to invent at the time like how do we how do we get data and there's different types of sharding right do you do it by [music] alphabetically so all every every element that starts with the letter A lives on this server over here and then you had to do routing you route specifically right and then what about data aggregation cuz you couldn't do that [music] anymore so we had to invent map produce we had to do all those things C++ yeah there you go I like the C++. Remember when Mozilla and Mozilla Firefox were separate web browsers?

[00:46:48]Yeah. Uh-huh. They were different. They were different things. I do you guys. It was a whole different world.

[00:46:57]All right. Positional encoding.

[00:46:59]None.

[00:47:01]There. Boom. All right. We fixed it again. Hey, [laughter] we eventually get status. Hold on. Let's Let's Let's save this. All right. Get add. All right. All right. Get commit.

[00:47:14]We did uh [music] positional encoding imple done. Okay. Good, good, good. Are you coding in Rust? Yes, I am. Gastron Gaston, hey, how's it going, Gaston?

[00:47:29]Good to see you. Happy Friday. We [music] are coding in Rust. We're building chat GBT in Rust right now.

[00:47:33]That's what we're doing.

[00:47:38]Arjun Arjun, I'm so sorry you got rejected. That's terrible.

[00:47:44]That's terrible. No, they shouldn't reject you. Why' they do that? That is an awful That is very awful.

[00:47:53]That is so not appropriate. They did it.

[00:47:57]The thanks for your lives. I enjoy them.

[00:47:59]Hey, Gaston. Good to have you here.

[00:48:02]Thank you so much. I appreciate you. I appreciate you a lot for being here. All right, let's see if we can do our positional encodings here. Let's see if we can do it. I'm going to see here.

[00:48:13]Let's see if we can make this happen.

[00:48:15]Positional encodings embedding return our positions. There we go. Or positioned positioned encoded. All right. No. No.

[00:48:25]Uh [music] run the build there. Okay.

[00:48:28]Good.

[00:48:30]Embedding.

[00:48:31]Uh what what hold on. What does this want? What is this asking for?

[00:48:39]Why is it saying that specifically? Uh, yeah, that should be fine, right? I think that's good.

[00:48:47]Oh, wait. Oh, wait, wait, wait, wait.

[00:48:49]Okay, where's my forward? Here we go.

[00:48:51]Yeah, it's a tensor right there. It's a tensor.

[00:48:55]Do we want to [music] say and tensor? Is it going to be happier if I do that? No, it doesn't care. Okay. Uh, positional coding embedding forward.

[00:49:07]Oh. OH. OH, NO.

[00:49:12]I see. I see. Hold on. Yeah, I defined it here though. Do I have to say pub?

[00:49:20]Why does it want me to do that with embedding?

[00:49:22]That's that's my that's that's my like it's it's right here. Yeah, I know it's there. I just want to access this function.

[00:49:32]Okay, so we already got the new that part works perfectly fine.

[00:49:37]So if we just do a quick I'm I'm going to go backward really quick. You got to go backward and forward. Okay. Expect a tensor found and tensor. Okay. Uh oh, right. Yeah, of course. Okay. Okay, that works. Now, here is a problem.

[00:49:56]Positional encoding. It's going to give me a new positional encoding. And I only want to access the forward method directly directly on it. Uh why do you develop chat GPT when it's already developed?

[00:50:10]Domu we want to do it to learn. We want to become experts at this right. If we build the technology that's hard to build along the journey we learn a new path, right? We learn a whole bunch about it. So being able to rebuild it is going to give us a new advantage, especially [music] in the world of AI that can write code for you. Building it ourselves by hand is going to give us a deeper understanding of the architecture [music] and design. It allows us to be able to get to the point of innovation.

[00:50:43]We'll be able to innovate. That's the way. Thank you for the hearts.

[00:50:46]Appreciate it. Uh so nice for you to do.

[00:50:49]Appreciate you. I see I see those floaty hearts. This is what it looks like on my screen when you guys do that. I see those floaty hearts. Thank you. All right, that's awesome, Stephen. Ah, Sergio. Hey, thank you. My channel dope, Sergio. My channel dope. Your name is great. I like your name there. All right, so we got a positional encoding.

[00:51:07]We get a new version of it. Let me [music] build this here. Positional encoding. Yeah. Yo, hey, how's it going there, Viet? How the Viet AV guy? Good to see you. Welcome on in. Happy Friday.

[00:51:21]Thank [music] you for the reactions, you guys. All right. So, we get our position one encoding. We got our embedding here.

[00:51:27]And then we return our embedding. We create a new one. And then we do a forward pass. That's it right there. And it doesn't need to be public cuz I'm already inside my public location. You got stuck in data structures and algorithms making logic every I drive.

[00:51:44]Oh, Maverick. Yeah, there's a lot of fun adventures there, aren't there?

[00:51:50]Wish you could learn code. don't have time or money to Hey, don't Hey, mean I didn't I spent zero dollars. I spent zero dollars on learning. It's just time. It's all available for free on the internet. You can you can do it. You can do it anytime you want. It's right there. Although, you don't really have to anymore because the AI can write it for you. I know, right? What we're doing right now.

[00:52:15]However, I feel like I've got an advantage by knowing how to do all this stuff. Sir guide me make logic start simple [music] use python right open a terminal window type in python type 1 + 1 that's the best starting journey right then you can say a= 1 a + 1 let's say a + 1 is going to equal 2 right a + 10 is 2 right b = 100 a + [music] b start there start with basic expressions in variable assignment Uh the Viet AB guy pronounced correctly first try. Yeah. [laughter] All right. We try our best. We try.

[00:52:58]Intermediate, sir. Um intermediate. Oh, so you already know how to program. What are you asking me for? What are you doing? What are you saying over there?

[00:53:09]You already know how to program. All right, that's great.

[00:53:12]You're You're on the right path. You just got to keep going. Thank you for the hearts, you guys. Appreciate it.

[00:53:18]Okay, thank you so much. I want So here's what I want to do. I want to figure out why this part this part [music] right here is having it's struggling. It's struggling. See, I can access new perfectly fine. However, when I try to access the for Oh, that's what it is. That's the part that I was missing. Whoops. All right. Little small details. You're like, "It's right there, Stephen. It's right there. I didn't know. I didn't know." Okay. So [music] now let's confirm that we still build. Yep. Okay. Then we're going to return [music] the tensor of our positions.

[00:53:56]Let's try again. All right.

[00:54:00]And embedding spec [music] tensor found and tensor. Oh, wait. Really? Really?

[00:54:04]Line 71. All right. We can we can do that. Sure. Why not? I assume. Is this the first time that we've asked for a tensor in our data? So, can we just say the and tensor that? How about that? Inputs expect. Oh, right, right, right, right, right, right, right, right, right, right, right. Um, you know, you know what? I suppose we could just try that.

[00:54:30]See if that works here. Boom. There it goes. Right there. That's what I'm looking for. All right. Perfect.

[00:54:39]Let me Let me save. We got We got to save it. We got to save [music] it. Get commit.

[00:54:44]It took It took us a a smidgen of a minute here. Fixed PE uh uh strruct. There we go. Push domain.

[00:54:53]Good to go. You're stuck in making logics with new questions like patterns, printing, and data structures and algorithms problems. So you need to think about how to retrieve how you need to store and retrieve data. Right?

[00:55:07]That's what DSA is mostly about. DSA [music] data storage and retrieval and you can store data quickly and you can retrieve data quickly. In order to put them together, you have to create specific structures that match for that. So, for example, a hashmap. What you need to do is create a whole bunch of buckets with addresses in them that allow you to store data. You can either do this quickly or slowly on the right side. So, you can have a slow write but a fast read.

[00:55:37]>> [music] >> And you can create just a basically a hash. It's I mean there's a whole bunch of different patterns. My favorite is the hashmap. It's a really good one. I also like B tree because it's going to be low memory.

[00:55:50]It's going to be a little bit slower lookup though. So if memor is expensive, the B tree is going to come back the cell balancing red uh red black green tree. [music] There you go. Where to start learning machine learning and AI? Oh, there we go. So, that's the better question. All right, you guys want to start learning AI and machine learning? So, you go to do this. Do this. Go to Google, type in scikitlearn.

[00:56:18]This is a really good starting point. I like scikitlearn. [music] It is machine learning. This is standard traditional machine learning. It's got all the algorithms built in. It allows you [music] to work with the algorithms directly and then it lets you learn how they how they work basically then you can start building your own right I think this is a good start to try someone else's first so you see what the input data should be and what the output data should be I recommend scikitlearn here you go [music] even write code to ask me optimistic that wait even write code that they ask me to [music] optimistic Well, it it takes a lot of time. It does. It just takes effort, memorization, rinse, repeat. Here's the best thing that's really important. Got to check this out. All right, you ready?

[00:57:08]You have to do it yourself without help.

[00:57:10]And you have to fail. You Wait, wait, hold on [music] a second. Wait, that doesn't make any sense. Why would I want to fail? You don't really want to fail.

[00:57:18]You need to fail because that failure is what's going to help you learn. [music] You have to seek out failure. Seek out the hard things. Fail. Learn what doesn't work. And then you'll eventually find what does. The part that you found and you learned on your own for what doesn't work is a 100,000 times more valuable than the thing that itself that does work. It's so helpful.

[00:57:46]Sorry. Optimize. You got to optimize.

[00:57:48]Oh, understood. Yeah, optimize. Uh, which is what AI is. You're optimizing, right? You're creating an optimization problem. You're optimizing. Thank you, Maverick. Hey, absolutely. Hope that helped a little bit. Just one tip. One tip for you there. One little quick tip.

[00:58:06]Okay.

[00:58:08]So, I think we're at the point now where we can add in our positions.

[00:58:13]However, we're not doing this as a batch. So, that's going to be a problem.

[00:58:17]Just a little bit of a problem. All right. Uh, let's see. Let's see here. I [music] need to figure this out. All right. So, we got our shape positions and then [music] we need to add them and return them based on the positions. Okay.

[00:58:37]All right.

[00:58:39]I see. I see. All right. So, if we can do at least this one part here, I'll be [music] happy. If we can solve this one bit, this one bit. It's the one thing today. If we can do this, we'll [music] be in a good spot.

[00:58:52]All right. So let's see if we can create a All right. So [music] let positions equals tensor equals tor tch a range and then we do input. We need our shape. Let the uh there's sequence sequence and we didn't do badge yet.

[00:59:17]What is this project? We are currently building chat gbt inside rust using the rust programming language which is very challenging. I think it's a hard problem. Is it react? No, [laughter] that's not this is not react. No, this is rust. This is rust. So colon rust.

[00:59:38]Uh see heart.

[00:59:42]There we go. It is rust. Rust [music] programming language.

[00:59:47]There we go. Okay. Sequence batch. How do we get the shape of the inputs?

[00:59:57]Inputs.shape.

[01:00:00]We do that.

[01:00:03]Shape 2. How do How do we do that? Do you guys know? All right, let's find out. All right, Google Rust Tensor TCH shape batch sequence and features.

[01:00:22]How do we do that? Let's find out here real quick, you guys. Rust. There you go. You got it. Rust. That's it. No, not that. I don't want to reshape. I want to get the shape itself.

[01:00:34]No, I want the shape. No, I [music] dotshape.

[01:00:39]I want the dotshape.

[01:00:42]Yep. Yep. Tensors size 1 2 3. Okay.

[01:00:49]Can I do it the way I did it with Python?

[01:00:53]Can I do [music] something like uh so batch, sequence features equals [music] uh tensor.shape shape size.

[01:01:05]I want to do something like that. This is what [music] I want to do. I can't.

[01:01:09]Yes, I can. I know I can. Yeah. See right there?

[01:01:13]So, that's what I want to do right there. Expected 3D tensor, but got a 2D.

[01:01:19]All right. So, then I have to unwrap it is what I have to do. So, I have to do a slice [music] and then unwrap.

[01:01:26]Is that what I want to do?

[01:01:28]Can I do that?

[01:01:31]I think I might be able to. This looks a little I mean this. Which one's better?

[01:01:36]I think this one's better. If you're 100% sure 3D, you can exactly use [music] the exact index.

[01:01:43]Or we could do this. I like this. Here we go. I like this. Let's [music] do this.

[01:01:49]Rust is difficult. Yes, it is a lot more difficult. [music] It's a lot more difficult because the compiler is very strict. It's like, you wanted to do that thing over there. You wanted to do that.

[01:01:59]No, you can't. You can't. No. No. No.

[01:02:01]You can't do that. Oh, you wanted you want Okay, if you want to do that, how about you try this instead? The compiler will do that, right? Look at here. See, it's like this. Look, it's always this.

[01:02:11]It's like, oh, you wanted to do this.

[01:02:12]How about you try this? And then you try that. And it's like, oh, nope, that doesn't work either. You want to try something else now? And then you try it again. And then it's like, nope, that doesn't work. Yeah. So, it's like this in a loop. It's in a big loop. It's like you're fighting. You're fighting with it. It's so much fun. It's a lot of fun.

[01:02:30]It's like a big adventure, you guys. All right, so size. All right, so let's see.

[01:02:37]Yes, please. Let size equal inputs size.

[01:02:41]Okay, and then we're going to say sequence uh and then features, right? Equals uh la. Okay, as a tupole size zero, size one.

[01:03:01]And let me think about this for a moment. So, we've got I would love I would love to can I think we should start off immediately with the batch cuz if we don't then we're going to hit some problems later down. It's going to take a lot of effort to unwind it. So, we should probably start with batches.

[01:03:24]Go. Uh, what is it? Golam. Golan Mustafa. Hey. Right. Hey, welcome on in.

[01:03:29]Good to have you here. Cowshake. Hey, you only use R and Python. Hey, that sounds like a lot of a Sounds like a lot nicer, doesn't it? Using Python and R is a lot more fun. It is. It's a lot better. Serio, I know, sir. I know, right? Sergio, here. We're going to do this. We're going to say BS and features. Now, one second here. Let me go back to five here. One second. I want to go to this really quick. Where I built something I was really proud of two weeks ago here. Right here.

[01:04:05]What is V? So, batch sequence and then V.

[01:04:13]Do you guys remember what I meant as V?

[01:04:16]A vocab.

[01:04:18]vocab. Ah, this is the decode. Okay, so these will be logits.

[01:04:27]This really should have been an L.

[01:04:30]L for logits, right? It really should have been an L.

[01:04:34]However, for since this is not an output, this is not logits. This is features. So, we're going to have we're just going to have to call it features.

[01:04:43]Size one, two, and size two. Looks good there. Python is easier than Java. Yeah, it is Maverick. I love it. Hey, just keep driving. How's it going there? The best way to describe riding a motorcycle is like trying corn with mayonnaise.

[01:04:58]What? For the first time. It's life-changing.

[01:05:03]I've ridden I've ridden on motorcycles before. I've driven motorcycles with the clutch, right? You got the clutch and the gas at the same time. Hey, Quantify Quantum. Good to see you. Happy Friday.

[01:05:12]We made it to the weekend. Your company hires for internship. Maverick, we I tried. I tried and there was nobody who wanted to do it this year. They're like, "We don't want to." Like, "Well, why not?" Well, we don't we don't usually when there's an intern, right? When there's an intern on the team, we need to be able to work with them collaboratively, and that's going to take away from our ability to be productive. I'm like, "Yeah, of course."

[01:05:39]And so, like, is there a way that you can work with the intern to allow you to be more productive? And nobody thought that was going to be the answer. Hey, Nea, how's it going? Stephen, how much happier are you with TCH? I'm so much happier. Oh, it's just so much better.

[01:05:53]Nea, thank you so much. Nea here, everyone. Nea is here in the chat. Nea recommended TCH. We were using the burn library. [music] And wow, TCH is so much better. Lib torch, you guys. Lib torch is the way to go. I love it. I love it.

[01:06:11]Okay. So we are going to do this will be our shape one positions size a sequence then I guess if we only need a sequence if we only need a sequence.

[01:06:29]All right. No good. You need an internship. I know data science. Python is very common daytoday language which you I love Python. I use so much P. Look here's how much I love Python. I've got a Python reference right here in code comments. That's how much I love Python right there. You're looking for one, I'll let you know in our Discord. If we do end up opening one, I'll let you know.

[01:06:53]Yep. There are other companies other than PubNub. If you ever go to qualification portfolio, you would get one. No. Yep. There are a lot of companies out there, right? There are like Nev was describing. We've got I've got my own I've got my own company here, PubNub. We're a data communications provider. I call this an API for communications, [music] multiplayer gaming, on demand delivery.

[01:07:15]If you're building any sort of app where you need uh one phone to talk with another phone, that's what that's what my company does.

[01:07:23]That's what I do right now. We're building AI.

[01:07:27]You're taking a massive machine learning course. 2 hours and 23 remaining. That's a big one. That's really big. That's really big.

[01:07:36]You miss the kind of brainstorming while coding me. Oh, you swear. I know you hate the dependent on AI. I know it's a whole different world now when we used to collaborate back and forth with each other and come up with really good ideas. Do you remember that? Cheers, fellow. Thanks for your time watching a camping video before sleeping. All right. Good to have you here. Yep, I know what you're talking about. That was now we talk with the AI to get our ideas across, right? We sure do. We sure do.

[01:08:03]All right. torch arrange size. Let me do this here. [music] Let me think about this. I don't know if that's going to do what I want it to do.

[01:08:13]Let's find out. Let's find out. It's f No. No. Um, not build run.

[01:08:21]There we go.

[01:08:23]Not found in torch. Okay. Pattern cannot match, right? Because it's not a [music] 64x 64, right? Uh, let's see here.

[01:08:35]How about we just say size one?

[01:08:42]All right. A little bit extra there. All right. So, a range. How do we get a range in torch library? How do we do that? All right. So, let's copy and paste this here.

[01:08:55]All right. Uh, Google TCH Here we go.

[01:09:03]Uh, Stephen, fun question. How much tokens estimated did PubNub use? If you guys are measuring the [music] usage, uh, I can tell you it's we're spending a lot of money. I can tell you that number of tokens though, I don't know. Um, so millions at least, if not billions, we definitely have to be in the billions by now. billions of [music] tokens. Why?

[01:09:30]It looks like you're trying uh typing an iPad. Oh, right here. An iPad. Oh, yeah.

[01:09:35]That's what the iPad because this is a Mac and I have the I have the keyboard up [music] here because you guys asked for it. You're like, "Hey, I want to see what you're typing in your keyboard."

[01:09:43]What are the keystrokes? So, when I type in the keystrokes, you can see what I'm typing, right? You can see what I'm typing there. Hello. I want to start my freelance. Can I help you with that? Oh, hey, Gabby. Oh, Gabby. Okay. So, check out Upwork. Upwork is a good place to go. This is a contracting freelancing platform. [music] It might be a good place to go.

[01:10:09]And what you want to do is you want to stay on top of all the new posts.

[01:10:13]Apply to [music] each new post that you want and target them cuz usually people don't reach out. They don't reach out to you. You have to reach out to them.

[01:10:24]Hey Gabby, thank you for saying hi. Uh peace loving soul. How's it going there?

[01:10:27]What is it you're doing? We are currently building a chat GPT in Rust.

[01:10:31]The programming language Rust is a rather difficult programming language.

[01:10:35]However, it's very fast and performant with the benefit of safety, concurrency safety, memory safety, and type safety.

[01:10:42]How much money is it for per engineer?

[01:10:46]Uh we're spending like maybe well it's it's like 200 to a,000 per month. About that per engineer. How about that? 200 to a,000 per month. Nea, good question.

[01:10:59]Good question, Nea. All right. Did I get what I wanted here? So, I want the A range refers to a function in torch. Yeah, but is it a thing? A range. Start end options. Options kind and device.

[01:11:14]Okay.

[01:11:16]Okay. So, we have to do this a range start.

[01:11:22]Okay. Yeah. All right. Got it. Okay.

[01:11:28]So, torch a I don't know why it's called start. That's really weird.

[01:11:34]Start. End.

[01:11:36]All right. What do we want? So, we need to see how what the sequence is. Right.

[01:11:42]So, how long it's going to be and do we want it to be a range? Is there a different thing? I think a range is correct. It's not line space. It's [music] a range.

[01:11:52]Start and end. Okay. So, we're going to say zero to the size.

[01:12:01]And then our options are going to be a float. We want float.

[01:12:11]We want float like this. And device.

[01:12:13]Here we go. Okay. There we go. Like that. Okay. There we go. Perfect. And did we define device here? We're going to need to We're going to need to grab the device. We need to grab the device.

[01:12:26]All right. Device is going to be a device.

[01:12:30]There we go. Very good. Oh, how was the day? So good. We We did it. Fixed the frame buffer problem. Nice. Gunzupi, good job. Thank great. Hey, how's it going there Rust?

[01:12:45]Rust Sock, good to see you. Thank you for saying hi. Can you make series on game making? Oh, we did. We actually did. We'll make another one in the future. We will. We did. Yep. On our on our YouTube. We sure did. YouTube over here. Wait, no. Uh, Steven, [music] here we go. Yeah, it's around here somewhere.

[01:13:04]Let me see. Playlist. Okay. Did we do a playlist? Here we go. Here's the game right here. So, yeah, we did that right there. We did. We're going to do another one. How's it going there? Nimzo switch.

[01:13:17]Oh, wait. Nimzo witches. Nimzo. Hey, Nimzo. Good to see you. Welcome on back.

[01:13:23]Good to have you here. Nimzo. Witch.

[01:13:27]W. No. Witz. Nimzo. [music] Witz. Shish. I'm trying to see your name.

[01:13:33]Uh, kind of game is it? It was a sort of a cat-and- mouse game where the AI would chase you [music] and it would get better every game, every round. It was a very basic game. It was very basic.

[01:13:42]Don't get me wrong. It was still pretty fun though. Call me Kunai. Auno. Kuno.

[01:13:48]All right. All right. Nim. Hold on. Let me see it. At nim equals kuno. Okay. Kuno. Gotcha. Gotcha.

[01:13:58]Kuno.

[01:14:00]All right. Float device. Do we want this to be No, we want this to not be float.

[01:14:05]We want this to [music] be long. We want to be long. And then self-device. We need to say device. Yeah.

[01:14:14]Is a the device. All right. We need to add the device over here. It's going to be a device.

[01:14:22]There we go. And then [music] when we initialize the positional encoding, we have to pass it in. We got to pass in the device. And uh hold on.

[01:14:32]We want that to be where? Path dimensions device. No, I think I want to do it differently. I want to do this over here. Like that.

[01:14:43]Perfect. Okay. Path device dimensions.

[01:14:47]That's what I want. All right. Path device dimensions, right? Yep. There we go. All right. Good. Good. Good. Good.

[01:14:56]Like it. I like it. We're doing good.

[01:14:58]Yesterday here, if you remember, you were still working on Rust GPT version.

[01:15:02]Yep. That's what we're still doing. It's going to take us a while. We'll be doing this for a few weeks. So, we're going to build this. We're going to make chat GBT and we're going to build it in Rust and we're going to be successful. That's what we're going to do.

[01:15:15]You're doing coding in a fun way like your style of coding. Can you give me tips on it? Oh yeah. The the the tip is you just dive into it without any preparation. [laughter] That's the tip you do. And then it's like an adventure. You're like, "Okay, now I got to do this. I got to do this."

[01:15:30]And make no [music] plans. No plans.

[01:15:32]Just little increments at a time. Just dive right into it. No plans. I mean, we have a big overarch plan, right? The ultimate plan of building chat GPT, right? We got to build the chat GPT.

[01:15:44]So, that's the plan. Uh, can't chat help you? Yes, it can. And we don't want to use the AI because it's going to slow us down our learning. It's going to stop our learning. Claude code. Yeah, we could. Yeah, we could. We could. Kunel.

[01:16:00]Kunel. Yes, we could. Kunel is DSA is em now. What is [music] imp? What is imp?

[01:16:09]Okay.

[01:16:11]All right. So, we did the good part here. And then [music] I'm just going to hide this for a second cuz I don't know if that's correct yet. And I need to see if this builds. All right. It does.

[01:16:22]Good. By the way, any YouTube channel you studied with. I didn't study with a specific YouTube channel. I just went around and looked at everyone who was building something on YouTube and I saw how they built it.

[01:16:37]And then I then I took their idea and then I reimplement it my own way is what I did. And then I reimplemented again.

[01:16:44]And then I did it again. And then until I could do it from memory. That's what I did. IMP is important. Oh, DSA is important now. Oh. Oh, got it. Okay. IMP is important. Uh, it is not as necessary as it used to be if you want to be successful as a software engineer.

[01:17:03]Heypo. How's it going there? Good to see you. Good morning. Welcome on in. Happy Friday. When did you start your company?

[01:17:11]Uh 10 uh over 10 years ago. So almost uh uh like 14ish years ago, right? 14-ish years ago. Yep. [music] My company, PubNub, you guys. I'm not I like how you know how to ask that question. Yep.

[01:17:25]Started it a few years. Like quite a few years ago now. 14 years. [music] Hey Derek, good to see you. Happy Friday.

[01:17:35]Welcome on in. Good to have you here.

[01:17:38]We're continuing where we left off yesterday. We're going to see if we can get positional encodings with the batch going. So, I do want to do batch. I do.

[01:17:47]Although, turn around to get to positions. We don't need to worry about batch here because this isn't a batch.

[01:17:54]So, we don't need to bother with [music] that.

[01:17:58]We don't need to bother with it. Let's see if we can at least [music] get that to work.

[01:18:03]Uh, do we have something missing here?

[01:18:07]Okay, let's try that. A range not found in torch.

[01:18:13]So, we need to copy that. And we do need help. So, we are asking the AI for a little bit of help [music] here.

[01:18:24]Uh, how was your employee life back then? [music] It was fine. It was ordinary. You had to learn to work with people differently, right? You had to be a contributor, an IC, individual contributor, and you just worked with the team. You we worked with the team collaboratively. [music] You built, you reported in, and you talked to people.

[01:18:44]Now, system design is more important than DSA. Oh, yeah. You're right, Kunol.

[01:18:48]Yes, you got it. I'm with you on that.

[01:18:52]Absolutely. Absolutely. You got it. You got it. All right. Torch kind device tensor a range.

[01:19:01]Okay.

[01:19:08][music] Okay. Is that right?

[01:19:14]Variant associated function. Okay. So, we need to see what that is. All right.

[01:19:18]So, yeah, there we go. That's what I'm talking about. Okay.

[01:19:27]Good. Good. Good. parent associated with constant not found in kind. So is it what is this? Uh I need to see what is uh in64. All right, fine. int 64.

[01:19:44]That's what I'm looking for. All right, we're making our way, you guys. This is looking good. AI is going to program anyway. Yeah, it does. Yep. Peaceloving soul. Uh-huh. Yeah, you're right. So, does it make sense to do things the old way? It does. If you want to learn it, if you don't really care, then you don't need to do what I'm doing right now. I care though and I want to learn it. I want to be an expert. Hey Lun, it's good to see you. Happy Friday. Yeah, welcome on in. Good to have you here. You're you're on time. You're on time. We've got plenty of time. How can I join your company? Is there a place for we uh there is we do have open rolls. So, PubNub careers, right? to click on our careers page.

[01:20:29]We've got stock, right? Flexible time off, which is basically unlimited time off. We call it flexible time off so that way you don't take advantage too. I mean, you should I mean, take the time off that you need to take out. It's it is an unlimited time time off policy.

[01:20:45]Health plans, full health coverage, wellness, uh, and other fun things. And so, we have we just filled our AI roll.

[01:20:54]We do have a Rust core engineer role that just opened up on our You plan to watch any of the World Cup games? Yeah, I probably will. Yep, I probably watch a few with Sam. I will.

[01:21:08]Let's see here.

[01:21:10]AI is right now in the growth phase.

[01:21:12]Yes. Uh-huh. Will change per token using vibe coding. Uh, it's going to be different. We're in a rapid iteration cycle right now. It's going to be very different. Oh, loons. Thank you. I'm going to pin your your chat there.

[01:21:27]Let's see. Uh what? Let me get back to my code here really quick. All right.

[01:21:32]Did that that build? Yes, it did. Okay.

[01:21:34]So, it was tensor A range the whole time. Tensor A range. Okay. Don't need that.

[01:21:42]Looking good here. So, [music] we got our size. We've got our options here. I'm going to I'm going make this a little bit easier to read.

[01:21:52]See if that still builds. Does it build?

[01:21:54]Yeah. All right. Looking good. See, all I do a little bit of adjustment and it makes it a little bit easier to read.

[01:22:00]There [music] you go. What about hackathon? Oh, yeah. I didn't mention it. That is there. We do hackathons. We create space for teams to explore exciting ideas that solve real problems.

[01:22:08]It's a part of the thing we do at least twice a year at least. And these are not like small hackathons. These are like, you know, a half a week or to a whole week.

[01:22:21]right now. A buffet system of AI. Use as much as you want. It is. Yeah, it's a buffet of AI right now. You got it. You sure do. At the stadium. No, Derek. Just on TV. On TV.

[01:22:34]When is it? Oh, these are internal company hackathons. We do do external hackathons. In fact, we just had one. We did. It's on our Discord, you guys. So, hackathon. We did. We just had hackathons. And Lun, who is here, won.

[01:22:50]Let's see. Where is it at? Do we have our hackathon winners here? Yep.

[01:22:53]Hackathon winners right here. We did.

[01:22:56]So, we will have more hackathons in the future. This one was just a few weeks ago. So, just last week or a week ago or so. So, it'll be a little while before.

[01:23:06]Which team you're supporting? Um, I don't have Which country is really what you're asking, right? What country am I supporting? I I don't have a specific one that I am rooting for. I don't I know. Wait, what? How can you What?

[01:23:21]WHAT'S GOING ON THERE? I know. I don't know. I I like to software engineer you guys. I see the sports on the TV and I go back to programming.

[01:23:32]I can see your passion here. Best wishes, peaceloving soul. Thank you.

[01:23:36]Thank you so much. All right, so we got our positions and then we should add these in. So, how do we do that? How do we add? We just do a simple add [music] right there. Positions embedding.

[01:23:46]Uh-huh. Yep. Did we get that? So we need to throw it to our pos. So we say how do we do this? Uh or embedding. Yes. Or self.betting positions, right?

[01:24:01]Uh let's see. Uh range. We do range.

[01:24:05]Yeah, we'll do range. That's better. Uh [music] range.

[01:24:10]So then we say let embedding equals embedding. [music] There we go.

[01:24:16]And then I think what we could do is we don't need to unsqueeze it right now.

[01:24:22]Though we should get into batches here.

[01:24:24]We need to make sure that we're doing batches cuz we want to be able to train this with millions of tokens per second.

[01:24:32]Who's your favorite player? Uh what about Messi? Does that count? How about Messi?

[01:24:40]Yeah, I heard that's a player. Lionol Messi.

[01:24:47]Uh, that's it. I mean, I other other I just know that as a a player. [music] I don't know really much more than that. I can't claim that I know. I cannot claim you guys. All right. So, positions. Yes.

[01:24:59]Yes. Yes. Yes. Yes. Uh, inputs. We need to do plus embedding. I think we could do that.

[01:25:08]Uh, hold on. Undo.

[01:25:11]No, it didn't like that. Okay. Field not method. Oh, we have to do forward.

[01:25:19]We go. Expected and tensor.

[01:25:23]Fine.

[01:25:26]There we go. All right.

[01:25:29]Plus embedding. Wait, how about positions?

[01:25:34]So that way we can read this a little bit easier in the future.

[01:25:38]That should do it. Hey. Oh. Oh. Oh.

[01:25:41]Okay. We uh hit an error here. Uh size of tensor 128 must match the size of 5,000. Uh tensor B 5,000.

[01:25:51]Um did we need to do something different?

[01:25:56]Those are the dimensions. However, on the shape size of things, we need to I guess we need to slice [music] it, right? We got to slice it up, bro. Mr. Chat. Oh. Uh, I missed your chat. Sorry.

[01:26:12]Uh, Kunel, Kuno, what' you say? What' you say? It's like we are are right now in a Buffett buffet system of AI, right?

[01:26:21]Oh, if uh here, I'll show you. One second.

[01:26:25]So, this is all I see.

[01:26:29]Uh, zoom in if you don't mind. I don't know if you could see that or not. That might be hard for you to read. But basically, uh, this is what I see here. Bro, miss your chat. It was like we were in a buffet of systems. Honestly, air right now company will change overnight. Yep, I'm with you on that. If you sent another chat, that means that it didn't show cuz of uh Google [music] might have said nope. Sorry.

[01:26:55]If import dependence, APIs, SDKs, and important for apps, what are the essential files of components needed for an AI agent? Uh the most essential is function calling tool use, [music] right? So you want to do function calling tool use, right? Function calling tool use is what you want. Hey Rox, what's up? Hey, good to see you. Welcome on in. Happy Friday.

[01:27:19]We made it to the weekend. You're you're [music] on time. You're on time.

[01:27:24]You're good. LinkedIn account and you want connect. Sure, Kuno. Absolutely.

[01:27:28]Yep. Um right here. Uh I'll send this on our Discord. I'll send you a link on our Discord. So under link share there you go on link share there you go on discord let me know and I can connect with you right now [music] like let me know as soon as you connect and then I will open up the browser and I'll I'll accept I don't have a LinkedIn account what Bonzupi are you serious you never have one should I have one I mean if you're a professional if you're a professional I think typically you want one right you typically want want.

[01:28:05]How do we slice this?

[01:28:08]Or how do we do a slicing here? Cuz this works. We're going to get successful compilation here.

[01:28:15]Uh what if we unsqueeze?

[01:28:19]I don't think that's going to do it. All right. Well, plus positions.

[01:28:27]Yeah. Oh, no. Error. All right. So, we'll see what are shape errors here. Uh la size of tensor 128 the same as before. Okay. So let me see here needs to [music] get the so oh I see zero. It's got to be [music] zero cuz this is not a batch.

[01:28:57]How about that? No. All right. 128. It's it is okay. So, it says 5,000. [music] However, we've got the dimensions here. That's the token's output. That's the max output. [music] So, that'll be the embedding.

[01:29:16]Do we need to transpose this? What do we do here?

[01:29:22]Un unsqueeze zero.

[01:29:26]Try that. I don't think that's going to do it. All right. Where [music] is our problem? Still the same thing as before.

[01:29:32]Yep. Same exact same thing right there.

[01:29:35]You sent the request. Oh, all right. All right. Let's do it. Let's do it. You have an idea about coding. Should I skip or keep watching? Genuinely uh asking uh uh parake. You know what? Whatever.

[01:29:47]Let me know what you're interested in.

[01:29:49]Let me know. How does slicing look in Rust? Sergio. We're going to find out.

[01:29:53]We're going to find out, right? We're going to make it happen. We're going to make it happen. All right. Before I do that though, let me see here. Let me pull [music] this up. We'll do a new window.

[01:30:04]Zoom out here. Go back to code.

[01:30:08]Okay.

[01:30:10]All right. So, let me go to network. I'm pulling it up on another screen. Uh, Kuno, I found you. I found you. Here we go. Accept.

[01:30:22]All right.

[01:30:24]Hello. Here. Let me zoom in. zoom in if you don't mind if this is going to zoom in or not. Are we going to zoom in? No.

[01:30:31]A little bit zooming. [music] There we go. Okay.

[01:30:35]All right. Here we go.

[01:30:38]Uh my screen. [music] All right. Here.

[01:30:42]Uh hey. All right. So, hello. Hello. All right. We're connected now. There we go.

[01:30:48]We got connected.

[01:30:50]Okay.

[01:30:54]Been doing literally most of your life.

[01:30:56]I'm really good at doing custom Linux deployments. Oh, very nice. I know how to configure most distros for most use cases. Bonzupi, nice. Any suggestions regarding build my first home lab server. Ooh, Barrento. All right, this is a Banzupi will give you some good tips, I think.

[01:31:15]Probably. I'm probably sure Bunzup is pretty good at that. Yep. Okay, so I know this is a shape mismatch is what we've [music] got here. So I want to how do I get the size? So let me say positions [music] do size uh print ln exclamation point.

[01:31:39]Uhhuh.

[01:31:43]All right. So I need to see what's going on here. [music] Here we go. All right, let's take a look.

[01:31:53]Let's take a look. Uh, that is 5x12. See right there? That's what I'm talking about. It's 5x 128 because that's the input shape. So, it should perfectly match the embedding. It's like so good.

[01:32:09]Where's the [music] 128 come from?

[01:32:11]Where's the 5,000 come from?

[01:32:14]So, positions positions of size. What about inputs?

[01:32:20]Let's do the same thing with inputs.

[01:32:23]Inputs.

[01:32:28]All right. This will be input size.

[01:32:29]Inputs size.

[01:32:34]And then we do position size.

[01:32:38]Positions size.

[01:32:41]All right. See what we got. What do we got?

[01:32:43]Uh, let me let me not print the other things out here.

[01:32:50]Print hide that.

[01:32:54]Okay, easier there. All right, there we go. Okay, so positions [music] is 155,000.

[01:33:03]However, we don't Okay, so what if we Okay, I see. I see. Did I Did I do this right?

[01:33:12]So, so is that is that correct? [music] So, uh, unsqueeze one maybe. No, I think it's got to be unsqueeze zero is what it's going to be.

[01:33:24]515,000.

[01:33:26][music] No. Here, let's try this.

[01:33:35]Let's take a look again. All right, that is 5 by 5,000 and it needs to be five by one.

[01:33:45]No. Um, what is this a range? So, our a range.

[01:33:52]What's going on here? Um, so it is it is a correct that is correct on this dimension.

[01:33:59]It's just that this 5,000 should be Does it need to be 128?

[01:34:07]Uh, I don't remember. Rest arrows are so ugly. I know, Derek. Right. 25 years in tech. Damn, I know. We're experienced, Kunel. Yep, you got it. You got it. So, you creating the positions on max sequence length or input sequence length? I'll show you. Um so we're creating an a range which is just you know number 0 through 5 right so basically creates a simple range we'll do the range [music] here as well range range so you can see a range it should be five really simple right we should also print out the range range print so you can see uh it's b basically 0 to 4 which is basically that's the number of sequences, right? [music] So, we need the number of sequences.

[01:34:59]Um, and then the output.

[01:35:02]Let me think this through. I I Does it need to be 128? [music] It might need to be 128. Does it though?

[01:35:09]So, it's currently 5,000, but it doesn't need to be.

[01:35:16]Let me think. Uh, let's see. Okay. Where we [music] at? Where we at? Okay.

[01:35:26]Let me check. Let me check my calendar here real quick, guys.

[01:35:30]Uh la.

[01:35:33]Okay. All right. All right. Good to know. [music] Good to know. Okay. So, we do have a plan here. We have a plan. I would really love to get this solved before we [music] wrap up today. This positional encoding would be really nice if we can do just this. Thank you guys. Thank you for the hearts. Appreciate it.

[01:35:54]V. Uh, Venge. Hey, Videnge. Uh, here I am. Uh, here you are again. Welcome on in. Good to have you here. This is going to be great. This is going to be great.

[01:36:03]Okay, so I have everything I need. [music] I just need to get in the right shape. So, we've got our embeddings. We've got the right [music] input side shape. Right now, we need Is that good enough? Is what we have right now work?

[01:36:21]5 by 5,000 isn't enough. No, [music] it's not. So, it's going to give us a an error here. 128 does not match 5,000.

[01:36:29]Right.

[01:36:30]So, how do we deal with that in this world?

[01:36:36]Right. So, this is what we're trying to match up here. Oh, that's okay. I I see what the problem is. You guys see what the problem is now? I didn't know what the problem is. I figured it out, I think. Maybe. Have you tried code forces? I have not. Kunal, I have not.

[01:36:54]Uh, Vanch, what's your plans for today? Okay, [music] here we go.

[01:37:03]Very new to tech. My college is about to start next month. Oh, Kuno, that's really exciting. You're on to a new adventure. It's a new adventure. Get ready. Everyone's going to cheat with AI. They are.

[01:37:15]Don't let them. Don't let them. Okay. I very much [music] want to fix the thing that I uh broke. Here we go.

[01:37:28]Boom. Right. Right there. Little bit of that. Little bit of that. Right.

[01:37:33]Now, oh, look at that. Okay. Did we fix it? We sure did. We did our positional encodings. I know it's not batch yet. I know it's not batch. We need to add the batches. We got to add the add batches.

[01:37:48]It just took a little bit of debugging.

[01:37:50]Look right there. How do you debug [music] in modern computer science? You just print. You just print it out. You print out all the things until you get what you're looking for.

[01:38:01]You have to study calculus again. Oh no.

[01:38:04]Hey, Retro. Hey, how's it going? Sup.

[01:38:06]Sup. Good to see you. Welcome on in.

[01:38:07]Happy Friday. We made it to the weekend.

[01:38:09]We made it, you guys. We made it. Hey, Foosball. How's it going there? Debug DBG exclamation point. Wait, hold on.

[01:38:18]Wait, wait. Foosball. Foo balls. Foo balls. Foo balls. I want to know about that. All right. What is that? What is that? DBG exclamation point range. Is that a thing you can do? Is that a macro that you can do? Uh, let's find [music] out.

[01:38:38]Oh. Oh, look at that.

[01:38:42]Foo ball. Thank you.

[01:38:46]I needed the debugs macro this whole time. Thank you so much. I've been using print. You know why? Because I've been using Rust since it basically came out a [music] few uh a little while ago, right? In 20 was it 2017 is when I started using Rust and I don't they might have had the debug statement back then. All I knew was about print. All I knew was fooballs 42. Thank you so much.

[01:39:11]Oh my goodness.

[01:39:14]Do you watch the Prime? I have Kuno. I Yes, Kuno. Yes, I did. In fact, I watch all almost all of the primer. I do. I do. I watch a lot of primer genen. Oh, that's some secret, Marco. It's a good one. Oh, thank you so much. This is great. Oh, look at that. All right, we're going to debug again. DBG exclamation point so I remember this positions.

[01:39:36]And then we're going to DBG inputs and try that. Yeah, there we go. Range.

[01:39:42]Okay. See if that looks good. There.

[01:39:46]Here we go. Here we go. Uh oh, wait. It wants the address. Really? All right.

[01:39:53]Here we go.

[01:39:55]Look at that right there. It's so clean.

[01:39:58]Oh, that's fantastic.

[01:40:00]Oh, Skinwalker official. Good to see you. Happy Friday. Welcome on back.

[01:40:05]You're making quantum app for assembly language and binary language mixed. Ooh.

[01:40:11]Ooh.

[01:40:12]Are you simulating quantum or are you using a real quantum processor?

[01:40:18]I mean it's returning map part looks like a function or am I wrong? Macro needs to be installed. No. Oh. All right. So in Rust you can define a macro which is just a function and they I don't know what makes them special.

[01:40:33][music] They're just usually denoted by an exclamation point. So see the exclamation point. See all these [music] bright up yellow exclamation points here? Those are just macros. Let's see here. How about this? Uh let's see.

[01:40:48]Google define Mac row in Rust. Simple.

[01:40:53]Let's do a simple one, like a really simple, easy one.

[01:40:58]Okay. See how to [music] do that. I don't typically do this. Uh my macro, right? Macro rules. My macro. Oh, see this is why I don't do it [laughter] right here. This is why because it's a I believe it's like mostly compile time [music] logic, right? Predefined function and macro. Yeah, exactly.

[01:41:18]Exactly. Kumal, you got it. Oh, that's what I missed. Rust. There you There you go. You got it, Retro. You got it. Rust is the answer.

[01:41:31]Boom. Boom. Boom. We did it. This is great. This is great. All right. So, we're done with our debugging now.

[01:41:38]However, I'm going to leave it there [music] because I want to convert to batch. Let's do batch real quick, you guys. So, uh let's leave this here. And we should get into batch mode, which is going to be array of strings.

[01:41:58]So, this needs to be a ve a ve of strings. Uh let's see. batch vec.

[01:42:10]All right, you guys. The right here.

[01:42:12]This is what's going to cause it right here.

[01:42:15]That's going to be different. All right, we need to open up the main again and update it. So that way we're going to do data data. We're going to do data data.

[01:42:26]Let's see.

[01:42:30]Let me say an address data data. Can we do that? Is that something that we can do? All right. And then we're going to just return a tensor. We're We have to We have to change everything now. We have to change it all. Everything's needed. We're adding a dimension. You sell my app to Google company.

[01:42:52]Skinwalker. Did you really?

[01:42:55]That's amazing.

[01:42:58]Biolearn.

[01:42:59]I've seen Python C build different than that. You got to specify every bit of information there, bro. I've got the book about it in C. Uh-huh. Never finished it. Advertisement on stream is Oh, you got an ad. Oh, no.

[01:43:15]Oh, wow. You got an ad. What was it like? What was the ad? Or is that too revealing? Finally. Hey.

[01:43:26]All right. All right. I'm going to get the batch going here. The batch, you guys. All right. So, is that correct? Is that part look does this look right?

[01:43:33]Does that look right to you guys?

[01:43:34]Dictionary to No, not here. Uh, this is wrong. I did that wrong. Whoops. Here we go. I got to do this lower forward right here.

[01:43:48]So, we will do this as a vector of strings. There we go. Perfect. Okay, that seems good. That's it. That's all we got to do right there. Vector strings. Use ad blocker actually too.

[01:43:59]Oo, really? That's one That's one way to do it. One way to do it.

[01:44:06]Okay. [music] It was a It was not a good commercial is what you're saying. It was not a good ad. It was not good. Okay. All right.

[01:44:14]So, that will take us into a ve of strings.

[01:44:18]Now, I need [music] to hide these guys here because they're not going to work anymore. We have to redo them. We have to rebuild them.

[01:44:27]All right. Right here. This is what needs to change.

[01:44:32]Let me before I [music] do that, I'm just going to say uh return return from uh tensor from slice.

[01:44:44]We're going to say 1 2 3 like that.

[01:44:47]Okay. See if let's see if [music] that builds. Expected vex string found string one. [music] All right. Uh let's see. array. Should we do ve or array? Oh, I [music] guess we could do.2vec.

[01:45:04]So, here's the thing. Which is better?

[01:45:06]Do you guys know [music] in Rust if I'm going to be doing this like efficiently?

[01:45:10]If it's going to be efficient, do we want to use an array or a vector?

[01:45:17]All right. Which is better? I'm of course going to ask Gemini.

[01:45:22]I will ask Gemini.

[01:45:24]See and percent here. Okay, let's go back to Jiminy. Okay, where do we do this? [music] Uh, is it Jiminy?

[01:45:34]Okay, what better vec or array [music] for uh TCH and batching performance?

[01:45:46]All right, let's find out. I I actually don't know between the two. I'm not sure which one. I think it's going to be vector. Okay, performance breakdown.

[01:45:53]effect. What you have right now is already good. Heap allocated dynamically resizable array. Although it's not, we're not going to resize it though. Oh, fixed array. Array stack allocated and fixed at known size. Yeah, at compile time a view continuously [music] catch Rust requires the size to be known at compile time. Oh, do we know? Well, we're going to have the batches, aren't we? I don't know. I don't know. Which one do we want to do?

[01:46:23]Do you guys know have the same notification sound? If you know the size, use array. That's a get foo ball.

[01:46:30]Yep. If we know the size, well, we are going to know the batch size, right?

[01:46:34][music] Cuz we're going to train it. So like batch size uh 256 or 512. So we will [music] know that. So that could be the array. However, when the user I suppose it's going to be fixed on the input side. So and for inference the batch size is [music] going to be one.

[01:46:52]So, if that's the case, we could just use array. We'll just do array. Use vec optimize later. Oh, yeah. Okay. All right. Fball. I like I like see what you're saying there. All right. Vec optimize later. Sounds good. Sounds good to me.

[01:47:05]Okay.

[01:47:07]Optimize later. Optimize. Optimize. You got the right. Never early optimize. You got it right, Fub. You got it right.

[01:47:16]Never early optimize.

[01:47:20]Okay.

[01:47:21]What you find less frustrating, rust or tor? Uh, pi torch is definitely a lot easier to use. We have to deal with all the other like we're combining rust and [music] and AI together. Challenging.

[01:47:32]Very challenging.

[01:47:35]Uh, see, nope. Nope. Nope. Two vec.

[01:47:41]There we go.

[01:47:43]Let's try that. Let's try that.

[01:47:48]Okay. Text not found. Oh, right.

[01:47:52]Uh, this is going to be our batch.

[01:47:56]Leave that alone for now.

[01:47:59]Go. Okay, there we go. Vec first.

[01:48:03]Optimize later. We did it. Hey, Steven, can I have mod, please? Bonzupi.

[01:48:08]Bonzupi. Okay, Bonzupi, you should you would There are no other mods here, so the answer is yes.

[01:48:18]We are hitting our mod limit on the channel. We got a lot of mods though. We [music] do we do we're we're starting to give them out a fewer. So, Bonzupi, because I know you cuz I know you cuz you come here very often.

[01:48:32]It's going to happen right now. There you go. All right, Bonzupi, you got it.

[01:48:37]You're good to go. You're good to go.

[01:48:43]Okay.

[01:48:45]So, let's see if we can get this uh return [music] tokens.

[01:48:51]All right. Tokenizer. We're going to do our batch. Batch. Here we go. This is our input. Batch input.

[01:49:00]Steven. Dude. But I don't see them very active [music] though. Oh, are they?

[01:49:05]Wait, really? How's it going there, Joey? Hey, Joe. Good to see you. Joy.

[01:49:10]Joy. Joy. Hey Joy, welcome on in. Good to have you here. That's why I asked because there are no other mods right now. I'm Bon Zupy. Thank you.

[01:49:20]You're here. Oh, GCA. Oh, GCA, I didn't see you. I didn't see you there. Uh, I keep your stream running while I code myself. Nice. So, I don't often interact with you. Hey, GCA. It's so good to have you here. Thank you so much. We're going to keep coding. Let's keep coding together. Let's keep going. Let's keep going. Okay.

[01:49:38]All right. Uh la. We need to get our tokenizer to deal with batches. [music] Now we have to upgrade our tokenizer.

[01:49:45]That's going to be easy.

[01:49:49]Maybe [laughter] we're going to find out here. Let's open up our tokenizer. All right. So, new and our encoder right here. Okay.

[01:50:00]So, this is going to be our batch.

[01:50:03]So, how do we how [music] do we do this? We need to process this a little bit differently.

[01:50:12]So, we need to map over this as a batch.

[01:50:16]Hey, Mark Leman. Good to see you.

[01:50:17]Welcome on in. Happy Friday. We made it to the weekend.

[01:50:24]J bro, it's a lot of responsibility.

[01:50:26]Uh-huh. It is because then you then you have to like pay attention a little bit maybe and like see if I mean there's some there's some expectations. There's some although not too much. It's it's casual like GCA for example just you know hanging out. That's also fine. That's also fine. Okay. So you guys [music] I need to take this and convert it to a batch. So this is going to be a ve of vectors.

[01:50:55]Right?

[01:50:56]Is that how we want to do it? A vectors.

[01:51:01]Then we got our parser and our parser is going [music] to return a vector of strings which is fine. We don't need to do anything beyond this [music] part. I think that part's fine.

[01:51:18]Right. Okay. So the parser has to stay as a parser.

[01:51:23]All right. How do we want to do this?

[01:51:26]We need to iter and then filter map and then iter inside the filter map in order to get this right. So if I were to do this and will it be a ve? No, it's got to be a it should be a tensor slice from two and it'll be a ve of vectors. Ve of vectors, right?

[01:51:54]something like that. And then we return uh let's see here. We got to go a step further. So this is going to be uh sequence, right? Sequence.

[01:52:11]This is going to be tough. This is going to be tricky. Have you ever created malware? I did, Nathan. I did many years ago. I did. Yeah. And it it worked. It did the trick. And it circumvented the security. It did. I did this when I was uh in college. It was in college when I did that uh many many years ago. [music] Don't worry, I didn't break any. I didn't break anything or or you know, anything crazy like that. I just built malware and it did the thing I wanted it to do. It circumvented some security is what it did. And I got what I wanted.

[01:52:46]That's [laughter] what I I got what I wanted.

[01:52:49]Okay.

[01:52:51]So, filter map. Uh, I am being vague on purpose because I don't want to get in trouble [laughter] because it happened so long ago. You guys know you there's it's hidden now. It's secret. It's lost in time. It's lost in time.

[01:53:07]Sequence uh filter map. Here we go.

[01:53:17]Filter map. Okay. I need to do some indentation here.

[01:53:23]Just a smidgen. Just a smidgen of indentation.

[01:53:27]There we go. Uh then we're going to bring this down here.

[01:53:33]Self.

[01:53:34]And then let's do this. Uh this also needs to be a new line here. It map filter map.

[01:53:43]And then I want to college site security. No, this was out.

[01:53:49]This was not college related. This was something outside of college. It was outside of college. Um filter. This will be uh not C. But maybe this is a token.

[01:54:02]Is this token? Yeah, this is a token. I think this will be token. Or how about how about how about word? Let's do word.

[01:54:11]I think word is better.

[01:54:15]Okay, the word.

[01:54:17]and then close.

[01:54:21]I don't know if you could do this. We need to collect this.

[01:54:27]Collect uh and then we need to make this a ve a [music] ve. Okay.

[01:54:35]Can can we how can we do this? How how do we do this? Um [music] can it be multiple lines? Do we have to do this? This might this might this syntax. We're gonna find out if this works or not.

[01:54:50][music] If you want to call that Oh, didn't answer the question anymore. It's quicksand. I know, right? Oh, will be.

[01:54:56]Uh-huh.

[01:54:58]You got to be careful. You got to be careful. Heard somewhere you once once in the internet. It's forever. It's true. I know, right? The good news is I I did I did go on the internet though, but that was many years ago. That was like a many years ago. It was so long ago now. You might never find it. I'd be very surprised if you found it though.

[01:55:19][music] I would be. Good luck, you guys.

[01:55:21]Good luck.

[01:55:24]Let's see here.

[01:55:26]Word collect [music] ve of vex. Is that good enough? I don't think that's going to work.

[01:55:35]Expected answer found vec. Yes. [music] Okay. Ve of strings encoder batch found.

[01:55:43]Didn't did I do this [music] right? Oh, batch vec like that. [music] Oh, we got to put our parser differently. We [music] got to do our parser differently here.

[01:56:00]So, we got to say batch.

[01:56:04]um here. [music] All right, let me let me let me think about this really quick.

[01:56:11]So, let's undo all this really quick.

[01:56:13]Let's bring this back here. Okay, [music] so this is the original and then we need to wrap this around with a batch. So, let me copy and paste this. [music] Okay, see here.

[01:56:31]Uh we need to say batch [music] dotitterfilterm map [music] sequence.

[01:56:43]So this be a batch iter filter map sequence. Basically [music] we need to do all of this text. How about Oh text.

[01:56:55]Here we go. Now now we got it. Now we got it. [music] Good evening. How's it going there soy?

[01:57:01]Good to see you. Welcome on in. Happy Friday. Are somewhere there? Uh, okay.

[01:57:07]Good to have you here.

[01:57:10][music] Oh, okay. Uh, where are we now?

[01:57:13]So, we got our vectors batch it filter mount text. Happy Friday to you as well.

[01:57:19]Thank you. Thank you. Thank you.

[01:57:23][music] You wouldn't make your own malware in Python. I made it in C. So I made it in C is when I [music] did it. Oh uh worm like in C or compile exe or whatever and send it. There you go. That's the way to do it.

[01:57:37]Uh la batch iter filter. We get our text and we say text [music] parser parser.

[01:57:45]Yeah, this is what this is what we need right here. Okay, we need the whole thing here.

[01:57:50]Just going to cop paste this. And that'll be text indent collect.

[01:57:59]And [music] then we delete this. There we go.

[01:58:04]Will that work?

[01:58:06][music] Is that something that will work?

[01:58:11]I mean that should work. [music] Slice two. Okay.

[01:58:18]uh value of option cannot be built from iterator. Okay, but I filterm mapped it all. I filterm mapped all of it. This is supposed to auto un unwrap uh options, right? Filter map un unwraps [music] options.

[01:58:36]Uh let's see here.

[01:58:38][music] See that really quick. Okay.

[01:58:42]Uh okay. Value of option cannot be built from an iterator over elements of type i64.

[01:58:49]Option cannot be built.

[01:58:51]All right. Uh from iterator I64 is not implemented for option is implemented for it. Option underscore.

[01:59:00]Okay.

[01:59:02][music] Vec string item is and string.

[01:59:11]Uh, did I do this wrong?

[01:59:16]Oh. Oh. M.

[01:59:21]This needs [music] to be a veabex back. No. No. This is a ve. That's fine.

[01:59:25]That's fine. So then we get a a batch iterate. [music] Oo. Hey, Son. Wait. Uh, Son. Wait. S Aani. Svakani. Silvakani. Thank you so much. I appreciate it. That was very nice for you to do. Become a member there. I was not expecting that. What?

[01:59:47]Oh, very much. Thank you so much.

[01:59:50]Wow. My gratitudes to you. Hi, you joined the right channel for software engineering. We're currently writing some Rust. A little bit of struggle there. Keep up the good energy. Oh, fball. Thank you. Might want to think about switching to software and gaming development on Twitch. Oh, people who want to look at programming content would find you more easily. Oh, hey. How do we do that? How do we do that? All right, we could we could do that. What is it currently set to right now? [laughter] Uh, let's see. It says science and technology.

[02:00:23]So, we can So, I'll do this. All right.

[02:00:25]Software and game development. [music] There we go. Boom.

[02:00:32]Done. Okay.

[02:00:35]I think we did it.

[02:00:38]Okay, stream updated successfully. All right, there we go. Foo ball, thank you.

[02:00:42]Appreciate it. Hey, we're learning new things every day, you guys. This is great.

[02:00:48]Learning new things, you guys. You're welcome, sir. Thank you. That was so nice for you to join. I appreciate it. I very much appreciate it. Okay, so let's keep going for a smidge longer here. Uh, just a little bit longer and then I have a bunch of things that I got to take care of. However, we're really close. I would like it if we could solve this here. And I might grab some help. I might grab some help from the AX. I don't know specifically why this not [music] doing what I wanted to do. Batch iterate filter to get our text. Oh, our text is going to be uh a ve of strings. Can we do this?

[02:01:27]I don't know if we can do that or not.

[02:01:30]Is that something that will work? I don't think that'll work.

[02:01:33]Uh, it did not work. Uh, expected found mismatch.

[02:01:48]Is that not what it should be?

[02:01:52]Oh.

[02:01:54]Oh, no. No. No. It's supposed to be a string.

[02:01:58]That's supposed to be a string. Okay.

[02:02:04]There we [music] go. There we go.

[02:02:07]Uh, the way back machine archives a lot of stuff. I know, right, Bonzupi? I know the shutdown kernel module. Oh, just kidding. You done? You didn't do it.

[02:02:16]Okay, good. You didn't do it. What did I do if I forget to compile the shutdown kernel module? Oh, the kernel module for shutdown. Yeah, right. Shutdown R. Now, don't press enter. Don't press enter.

[02:02:30]It'll end stream.

[02:02:32]Oo, that was scary. I shouldn't have done that. That was too risky.

[02:02:38]That was risky.

[02:02:40]No one better on the internet teaching nowadays, Stephen. Ah, he's really awesome. Whoa, MC MC G, Lucky Luke, thank you so much. That's nice for you to say. I appreciate it. We're having a lot of fun adventure right now. What?

[02:02:53]Does it really want me to add an amperand layer? Does it really? Like seriously, you want me to [music] do that? Really? So this is this is correct. This is a text because we're just iterating over the batch. We're mapping and we're getting the text.

[02:03:07][music] We're returning we're returning a parsed array [music] of tokens of of words, right? This is an array of words or vector of words.

[02:03:22]Basically, [music] we're iterating over that. And each of these is a word. So let's change this over here [music] to word.

[02:03:30]like that.

[02:03:32]And let's close that really quick. Okay.

[02:03:36]What's the progress boss?

[02:03:38]We did our positional encodings. We did.

[02:03:41]We got it working. Uh-huh. We got it working. And so now what we're doing next is we want to [music] start batching. And if I can get this one batch here, then I'm good. Shutdown r depends on shutdown.co. Even does it?

[02:03:56]Well, it might have shut down my [music] computer. I've not I don't usually use it. I usually just leave my computer running. Think we can do something really creative with AI? Yeah, we can.

[02:04:06]Ernum. Yeah, we can.

[02:04:09]That's kind of like the idea.

[02:04:12]All right. So, All right. So, we got a different error now. I 64 is not implemented, is it? Where is it? Here. What line? 54.

[02:04:22]54. Here. [music] Yeah, it is. It certainly is because we've been doing this.

[02:04:29]All right, [music] since we do have to get going, you guys, I'm going to get a little bit of assistance here. All right, do a new See here. Uh uh fix [laughter] you didn't see anything. You didn't see it.

[02:04:49]How does voice changing models work? You get an input and output, right? So you could do an encoder decoder if you want.

[02:04:57]And what you do is you have your input which is going to be your voice and then you could do input as text as well.

[02:05:04]Right? So let's see encoder decoder. So in this is what we're building right now. We're building a transformer. So we could basically do a voice [music] changer. So we've got our input voice and then we've got our basically our uh input alsoish our target [music] for what our output's going to be. And that could be the modulated other voice, right? So you've got your target voice, which is going to be the whatever effect you want, like reverb. If you want to add reverb, and you want the AI to learn reverb, you encode your original voice, and then you also reverb your voice, save that as a wave, and then make that as the input here.

[02:05:44]So then you got on both sides, you got the inputs and outputs.

[02:05:47]So that's how I do that.

[02:05:50]Okay, let me see. Back over here. Close.

[02:05:54]Did we [music] Where's our Gemini at?

[02:05:57]Did we do that here? No. Let me close some of these [music] windows, you guys.

[02:06:02]Few too many windows are open.

[02:06:05]All right. Don't need that anymore.

[02:06:07]Don't need that. Here we go. A range.

[02:06:10]Right. Fix. Okay. Here we go. Okay.

[02:06:16]Is this not what I wanted? No. Where is the thing that I want? No, this isn't it. Close that. Oh, where did our Where did I go? Is this it here? Yes. Here we go. Okay.

[02:06:28]So, what did we do wrong?

[02:06:30]See, it looks like we got it right. Uh, see how far off were we?

[02:06:40]Encoder.get tokenize everything. Yeah. Yeah, that's what I wanted.

[02:06:49]What's going on here?

[02:06:52]So, you just uploaded a screenshot of your last Nick's build and the Rice channel for Discord. Oh, I want to see really quick. Rice.

[02:07:02]What do we got here? What we got? Ooh, look at that.

[02:07:07]Colonel.ny.

[02:07:09]Very nice. Ooh, is that Machai? Is this Machai? I like this. This a great really good one. Hey, it looks Machai. Smokai, I like your colors. Very nice. Oh, maybe this is standard colors or [music] something. I don't know. Very nice. Fire and ice. Uh, snowflake.

[02:07:31]Very nice, Bonzupi.

[02:07:34]With my CPU pin at 100%.

[02:07:37]Oh, I didn't notice. Check that out.

[02:07:39]Yep. You You pegged it. You pegged both of your cores. Oh, wait. You pegged every core. You pegged all of them. All six. How'd you do that? What? What would you do to do that?

[02:07:55]What are you running to? That's kind That is a skill on its own. That is a skill on its own to be able to peg every single core 100%. Nothing wrong with using AI. No, but look, it says that we did it correctly.

[02:08:07]It says that we got it right. This is why I was like, "Okay, I got to go to AI, you guys. I got to go to AI."

[02:08:13]Uh, so how about this instead?

[02:08:16]So return tensor from slice address 1 2 [music] 3 or sounds good.

[02:08:26]Looking good to me. Okay. See if that builds. See, it doesn't like it. It doesn't like it even though I did it right here. [music] Are we sure?

[02:08:37]Yeah. Okay. So, here's what it says that it should be.

[02:08:42]Flat tokens.

[02:08:44]Do we [music] want to do that of back?

[02:08:47]No, we don't want a flag. We want a We want a multi-dimensional batch.

[02:08:53]Make flags. Oh, there you go. J8. Yep, that'll do [music] it right there. Yep, that'll consume everything. Yeah. So, you have six cores. You could said J6 kernel.nix where you can able enable it.

[02:09:11]See, did you J8 some Did you do J8 somewhere or or or you just saying that in general? Yeah, on the make. Yeah, got it. [music] It's parallel. Got to parallelize the the what do you call them? The objects. Hey, donut dude. Good to see you. Welcome on in. Happy Friday.

[02:09:29]Welcome on in. Uh I'm good to lad to have you here. We're trying to figure out what went wrong with our code. Uh flat tokens.

[02:09:39]See, I did it all correct. This is all correct, you guys. It flat. Oh, so we don't collect it. What if we don't collect? So, that's that's something that was different. They didn't collect it. All right, we'll [music] try flat tokens and see if that works better.

[02:09:58]See if that builds. [music] Uh, no. Collect is Oh, it really does want us to have that there. Okay. Well, it didn't hear.

[02:10:09]Obviously, this could be wrong.

[02:10:12]Loving the music. Hey, don't dude. Nice difference between reshape and view. So, uh yeah. So, view will make sure that it doesn't change the underlying linear structure.

[02:10:28]So, when you change the view, you're basically just changing metadata.

[02:10:33]Reshape does the same except for if there's a scenario like if you already transposed if you've already transposed.

[02:10:43]You cannot change the view because it it's like it's trying to like you're really mixing it up at that point. So you have to do a reshape.

[02:10:54]So if you do transpose and you want to change the view, you have to do a reshape. Uh that's it. However, if you you can change the view as much as you time and it's free. View is what you want to use as much as you can often as can use the view because it's fast.

[02:11:08]Reshape [music] will try to change the view. If it can't, it will change the underlining data structure which requires another malic potentially.

[02:11:19]You have eight cores. It's just hiding.

[02:11:21]Oh, got it. Okay. So, J8 and you're using all eight cores and it's not showing them all here. Got it. Okay.

[02:11:28]Understood.

[02:11:34]All right, we got to get going, you guys. I have about 0 minutes left.

[02:11:41]Btop. Oo, Btop window too small. Yeah, Btop will hide things when it want.

[02:11:45]Yeah, it will. All right, I do got to get going, you guys. Um, I really wanted to fix this. I think what we'll do, the plan for tomorrow is we'll do the batches. We'll get batching done on the system so that way we can proceed [music] onward, right? We can proceed onward cuz I think for the most part we've this is like one of the most challenging parts I remember when I converted my non-batched batch specifically in Rust because now we have to do this weird nested thing which we are right here for and it is hitting a problem. Uh value option cannot be built for item 64.

[02:12:27]Uh so we here.

[02:12:32]Let me just grab this really quick. See you later all. Yep. All right. Had so much fun you guys. Thank you so much for joining. We'll be back tomorrow on Saturday for uh early morning stream again and uh around 8:00 a.m. or so. And we will continue with our batches. We'll do our feed forward and then we'll continue building our GPT and rusts. All right, Cartum, thank you guys. Sergio, have a great day. Foo ball, thank you.

[02:12:57]Foosball fooballs, thank you guys.

[02:13:00]Sergio Erdom Dar, hey Stephen, I have always a question like this. Any AI model have reasoning I always have a question like how any [music] AI they kind of kind of they're emulating reasoning. It's not real reasoning, right? It's pretend. It's pretend. Uh, Son Connie, Silon Connie, thank you so much for becoming a member today.

[02:13:21]Appreciate you, J bro. Google recommend video similar to my dream. 1,000% sure I never searched or talked about it.

[02:13:28]Strange. J bro, I've had the same thing.

[02:13:31]Have you guys [music] ever had that where you had a dream and then you wake up the next day and then you see it in a video? Have you done that before? I've done that a few times. Yeah, it's crazy.

[02:13:41]It is strange. All right, bye guys. Have a good rest of your day. We'll be back tomorrow. More software engineering, more coding, more Rust, more AI. Thank you for joining everyone. See you tomorrow, Dar. Bye everybody. See you tomorrow. Thank you so much.