This video demonstrates implementing a reinforcement learning library from scratch in C, covering the autograd engine for automatic differentiation, the policy gradient Reinforce algorithm, and a snake game environment. The implementation includes defining variables, matrices, and computational graphs for forward and backward passes, with the agent learning to navigate a snake through rewards and penalties.
Install our extension to search inside any video instantly.
Coding RL library in C from scratch
Added:Hi everyone, I hope you're doing good.
In this video, we will going to implement reinforcement learning library from scratch in C. So I hope you have some understanding of the reinforcement learning. How the agent loop works and how the agent takes action in the environment and environment gives the report to the agent and that how the agent learns and get feedback uh and also have some understanding of the machine learning and stuff. So yeah, I think u if you have these all the prerequisics then let's get started. I think this is a very good video because here we can actually go very deep and implement things from very scratch in C and uh we will implement the policy gradient reinforcement reinforce algorithm and um we will implement the model from scratch. We will implement the environment RL environment from scratch and also we will not implement these things but also implement the autograd engine which is like a computational graph of the gradients where that's how neural network trains.
So yeah, let's get started. U there's some of the things that uh I already defined which is like base.h which is like some types here which I have already defined and this files contains some random numbers which will help us to use it in further video. Uh so here are some things I want to mention first like this is the video you can probably see in the screen this video is really good video uh and I think this person actually taught really good about like how to build the machine learning library from scratch. So this is kind of an inspiration that I took and okay let's go and implement the RL library from scratch. So yeah, if you want to check out this video, please check out that too. And uh pretty good video. So let's get started here. Uh I think first we can uh let's define the autograd because that is one of the fundamentals right now and then we can uh go for like defining model and then our our environment. So let's get started.
Autograd C. Um okay. And uh here we have this u what we need. Um so as you probably have the understanding of like how the model works where you have like forward pass and then backward pass. So which requires a lot of which requires a graph uh to go from one direction and then also in the reverse direction and that graph is what we're going to be define here like this the autograph which computes the auto differentiation stuff uh in the backward pass. So here we will define the graph of the forward pass and then in the backward pass when we are going that backward pass basically creates the traversal graph and identify variables uh within that and then computes the gradients and that's why we call autograd which is like auto differentiation stuff. So yeah uh let's go and define the variable first like what is variable you know variable is basically the which is struck it could be and uh it will contains index q32 index and then will have the number of inputs you know and that number of inputs is will be like a number It's also a var so inputs because one variable can contains the inputs of it. So let's suppose I have a + b equals to c that c is also where a is also where b is also where v variable is basically the parameter you can say in model. So this is what it is.
It will have flags you know uh U32 flags and uh this V inputs basically will have number of inputs and then it will also contain so basically this V inputs is the number of inputs number of inputs it will contain a basically the inputs it can have like one input or two input so because like for Some of the operations like relu soft max requires just one input but for the operations like addition subtraction that requires two inputs which like could be a and b and any kind of variables. So yeah I think uh what else we need like we need value which is like a what which will be a matrix. So here we will create a matrix.
We haven't defined the matrix yet. So but we'll we will define later and uh a gradient. So gradient is also matrix right. So these are the things. Uh what else we doing in a variable can you think of um where can have like columns stuff? I guess not really but a matrix can have.
So we can define in the metrics. So yeah um you can define create metrics which will have 232 rows 32 columns and um and then we have it it will have like a data right and that data is could be 32 okay let's define first this thing have this and then a data I So we can have like a U32 data or I can say like F32. Yeah, we can now this is like uh the actual definition like the the initialization of the stuff we can define the definition here also so it won't get confused that defined earlier or later.
So type def I hope it is recording. It is recording I guess. Yes. So, so right we can check it if this is working or not. Go to terminal C run that. So this is very confusing you know when we are using Python and C at the same time Q32 will come from nice. Uh then we have like strruct bar type. So this is the bar we have defined.
But like you know we also need what is the type of this bar because it can have like addition bar it can have soft max bar it can have like different operational types of bar. So with what type of value it is? So bar type and this can have like this bar type can have um the types which is basically shape. It can have bar function shape. This shape you know this this function basically will help in shaping this v type and this bar type have can have like war funk back forward function and backward function. So then it will have a good definition of v type like what kind of function it has, what kind of backup function it has and obviously the name. So it will have a name, right?
And that's it. That's uh that is the bar type and also the type. So now we also need to define the function here which is like this function and stuff right. So after our definition of v we can define or like there's not nothing needed for that because like say here v p32 and uh function which will have like the 32 columns 32 rows. This will have just one. Okay. Now this is we have defined it will have forward backward shape type and stuff four errors which is like type this should be I guess uh should be this should be a pointer.
Got it. Uh the type will already be defined by the name here. So I guess that's it. And uh that also requires like how many inputs it has this um this bar will have. So we can also define like E32 number of inputs right it is working. Um we can now we need to define some of the definitions here which is the v it will have like v soft max you know that will have me arena okay I haven't defined the mem arena right so I'll just what I'll just do is like I will also include the arena thing which is arena is basically which defines the memory location allocations and stuff and for this kind of project.
So I was just going to go and I have defined the arena here. This is the arena.
So it will help us to actually create the arena and stuff. And uh arena.h will have the all the definitions.
So yeah uh in autograph we can just like import it like this. Now this soft max will have me arena now which ask for space and stuff model state model and then v input actually just input I will define the model state so we'll get the arena from the arena itself and uh I'll define the model state or we can define it first let's not put Great big deal. So truck model and uh in this model we have you know model state essentially model state uh in this we'll have like you know a lot of things uh which help us to build the model the one of the things that I actually getting in my mind which is advantages.
So advantage require in reinforcement learning. So we will store the advantage here and this is basically going to be flo 32 right and uh this is the advantage and it will this contains a lot of things like number of bars and number of bars would be again this U32 and it can have it can also have a lot of different things I guess model state not models it can have you I guess that's it we will innovate it later I think we can just find what are exactly required for us so this is the model state uh we can define it here again type dev straw basically we can just define it here like that bar soft max bar arena arena state model where input this we can copy it right for the where add now this v a b this again subtract I guess we also need type of it which is like the v type I'm guessing v type type subtraction is done now maximum this is the matrix multip application you know and model state reinforce and art we'll get like a pro probabilities and art stuff. So let's run this after this draft. Okay nice now you know we also need like backward pass and forward pass thing. So void will basically you know which is where soft max forward which basically going to forward the soft max and that require these things like me arena and all the stuffs inputs and this will compute the I guess this will compute the soft max and stuff yeah so we don't need bar type we don't need this and uh yeah why we are just doing this is again going to be you know same same going for the backward so wide soft max backward although like here we just need back don't need like because we we're not computing anything we're just computing gradients and we already have like the uh calculation stuff so we are going backward and then using this var we can just get the output of it and we can compute the gradients. Got it? V backward. This is our backward pass functions and stuff now. Okay. Now we need to build function which can create variables. you know we have to find this is some abstraction for us this variable.
Now we need to define variables um like create variables. So a variable will a function that returns variable create bar arena arena model state and uh 32 rows 32 columns right and flags. Now in arena you will find like this push truck function which is here push truck function which basically can be used to allocate the memory for the specific bar first right so we will use this this autograd you know I hope I'm recording yeah all good so this auto grad this create basic Basically first it will do is like v out equals to this is how you define it right this and and this out index = to 1 + = to 1 + out and then it will have now we need to def create create a matrix so matrix m= to create matrix a function that and create metrics. Now before creating that we should have created matrix create matrix that contains like you know arena arena rows columns and the matrix is this right rows column data I guess like this is uh let me confirm it will be a pointer to be honest um yeah it will be a pointer so yeah I think this This is uh this is how you define create metrics. Create matrix and uh you basically going to push struck the metrics. This push truck will contain arena and that say out columns return out data will be okay. So yeah data we probably we can like define data as as push truck this data is push array because we already have this function push array.
So we can use this arena tn. So arena t is basically the tricks size of tricks and then rows and columns rows. We can just like push array like this. Yeah, exactly. That's why we use this. We can directly use this. Uh it's getting confusing.
So this working we can try finding create metrics.
We don't have arena. So create arena create arena. A arena will take capacity. So let's define 1,00 and uh arena 25 25.
We can also print out the matrix which is the matrix A is it has 0 0 0. So this printed the first. Okay. And we just print the rows you verify. That's great.
So this is uh we are this is here we are right now. Uh instead of like defining this we can also say like push and then uh it contains Um and then wow then freedom metrics arena rows columns here and uh we haven't defined the flags right we haven't defined the flags so flags is something you know we can define here as enum flags which has flag none which is zero where have this value binary flag. So these are the flags you know u like what exactly this variable have is it a variable that contains gradient or is it this variable is a parameter like it's none or something like that just useful computations.
So that's what it is required right and um yeah I hope it is recording my voice is clear and uh yeah so this is what so we are here right now define the matrix and then matt will have you know this will have a name value and then out val equals to val out and grad equals to if it has uh you know if I would just say type def flags it's just defining as u32 let's say if flags if flags flags if it is a bad require cat then probably we also need to add gradient into it create metrics columns that's it I think inside this bar there's nothing more else uh uh there's inputs. So for that like I guess we also need to define the inputs. Um 0 = to a to b and it will only happen that if this if out dot number of inputs contain greater than one then you have to basically store it there. And this number of inputs comes you know comes from type number of inputs contains them at this you know we also need to add type here which is like which we will get from type type and uh we will also I guess like model state is not required here.
just remove this and we can also remove uh I guess like when we are creating bad like first time we will create so we don't we will not have like inputs and we cannot define this so I guess uh earlier version was better which is like model state and uh this thing and uh type we can say null say this yeah uh that's it now we can just return the we will define like the inputs and stuff and we we will create a node you know so out we we have defined and uh type because we haven't defined here should be like this is working I guess and uh then we have to define node this creates the bar and we can actually go and then try to create it uh we can just go and v is aware arena then model state is state I don't know like why did we needed a model state here you know state model flags will be like zero I would say so yeah it's working uh does it have a grab it won't have a grab like we have to find this well uh you know then this model state that I just remembered like it have a number of vals so when we create this thing that's why we need model state because in index we can say it number of this will have a index right this will have a index so number of bar model number of bars plus+ Means like as we create this var that will also reflect a model which is like a you know model will have it will actually automatically goes inside the model you know and then it increments the number of bars although like this limits the possibilities like if you want to create number of wires outside the model then but yeah I think for now it's good so create V this This is done.
This is working. And then we have to create a node. You know, uh what does node will do? Uh node basically the gra nodes of graph you know because we have graph from bottom to like operations like soft max. This is it itself is a node this itself is a node. This is a node. This is a node. So as the computational graph grows it will create a node based on that. So that's why like we need to create a node and for that like we have to define graph uh which is like again a strruct strruct graph and this graph will contain you know um this graph contain the variables. So because graph is contains like a notes we can say like a notes but like instead we say like where and that where is a pointer the area of pointers. So you can say like also this double aster is the uh the pointer to the location of pointers you know. So this will have bars and uh will have a size. What is the size of the graph? That's it. Uh this is the graph and uh matrix in the graph.
What exactly is happening here? Why is did I did something? Okay, I just did something here crazy which is okay because the model wise is is is is null it's just definition is wrong but yeah I think number of hours will be going to plus+ in this Um, I'll take a little break, I guess. You know, I should take a little break or like I should go create something really. Okay. So, so graph has no problem with this and okay now this is done. This the graph is like we have defined a graph. Now node that node basically will give us the v itself you know create node and that contains v a and v b you know and uh because a node itself is a softmax let's suppose so it can contain v a and bar b and uh can have like a model state can have flags nobody max that's it. Um so if inputs and it will have like a type v type type and this type will be basically a pointer.
So this uh we'll create a node sorry create a bar this bar basically arena and stuff will have like the bar a and v bottle state for and then okay got it. Um now this bar type you know if type is type equal equals to bar will have like type equals to if now this type contains you know this type contains shape. So and that shape is basically can have a b can have a and b. We have seen it like that function contains a and b a b columns and rows a b f. This is basically returns true because it's a um this shape this v fun type is a boolean itself. So if this thing contains if it is true then that means that the shape is right of a and b compatible enough to the operation whatever the operation uh based on the type and uh if it is true then I guess like we can just move forward with that.
Um so I have just do it like bar type shape and before that I think I have to define row= to zero column to zero and then we have to provide the reference of it and that will have the things but what we will do inside of this is like create a node and then model state I guess like that's why like models was required. This is the right way is creating a new variable based this is based on this the the the goal of this create node is that you know to create a new variable and uh but that variable is already coming from something so variable A and variable B you know so that's why like it's not like independent variable like this so but I guess like we don't have to do that Because this is already doing inside this create v that requires model that's all like model rows column and flags we are making complicated a lot with this because uh you know so a lot of things we are actually taking here which is not required. So as if the goal of this node is to create a new variable based on the past A and B. So we don't need like a lot of information. We already have some information from A and B. So I guess like we just remove this thing arena we have model state v and this this type only. Now this flag we can just define you know the two flags as a bar flag none first right and then if flag if a isn't if a isn't equals to null and and a flags and required grab r isn't equals to null and and B flag and where flag requires graph. So if this condition is true like means like A has this either A has like the gradient on or B has this gradient on then we can say like flag is requires you know I it requires only one of it because we are using R here.
So this is how we can get like black and uh for the rows and columns here we have defined it here rows and columns and then inside this if this is not true you know if if this type shape is not true then just return null and uh after that we can like create this bar because we now we have rows and columns. So this shape will you know I haven't defined the shape function but this shape will do some kind of like manipulation with this rows and column that's why like we are passing the reference so it will change the whole this rows and columns and then we will provide here so we don't need to pass from the arguments we already getting it by when you like because a and b you know is a this new variable is coming out from A and B. So either it is exactly their shape or like some kind of uh matrix multiplication they're doing. So we can already compute like the out shape and compute the rows and columns of the out thing out shape. So this and then flags right arena model rows columns and flag and then where type is type we can define it like this. We already have defined it doesn't make okay now if so we can also define this because it is coming out of A and B so A and B's are basically and we are the inputs of this U so if bar v bar v bar v bar v bar v bar v bar v bar v bar v bar v bar inputs zero is basically a and if var dot type number of inputs greater than one inputs to That's it. Uh this is this will create the note. We will return the awesome.
This is this is great. Four errors which is like a flag. It should be flags.
Okay. It should be flags in. Yeah.
Should be flags, columns and rows. It should rows and columns. I guess this wasn't the error. Incompatible pointer to integer conversion.
do stuff. What does it mean here? Which is this should have this is working. I guess I should take a little break and then we can just come back and start it.
Okay. So this is great. Uh next we have to implement uh the graph.
So the basically the how we going to build the graph is we can build the graph using the output. So let's suppose the first is input and the last is output. So output could be like after the soft max and in between there's a matrix multiplication relu and all these layers but soft max is basically going to be the last layers and that soft max we can put at outwire and then outwire that will back propagate and discover the nodes discover the nodes and then eventually going to figure out. So I can just show you like I basically like this is slides I made.
Um yeah like this is the graph like this will have inputs matmo I I think it's visible right this input matt activation stop. So this is like the forward path but you know during the gradient computation stuff it will going to be a little different because uh we need to provide we need to go backward. So we will provide the output w and then it will discover the inputs because we are storing the inputs also.
So it will go to the all inputs and then discover everything and then finds out like okay which uh finds out the dependency of the inputs and then compute the gradients. Um and uh yeah I think that's we have to do now for that I guess uh there's this thing called create graph return graph and uh it will have out. So there's this algorithm called uh depth for depthwise traversal algorithm which will take uh the one node and then finds out the inputs node and then traverse back you know so it it will have like a stack so it will have a stack and in that stack we can put a lot of different things. So it's like this uh you know stack will have some you know we'll put lot of things u more like a b c d t e all these computations and but it is going in the forward direction sorry it's going in the from backward direction so because we want to create this graph we will probably what we will do we will go out and Um using output graph we will okay let me explain again like so in stack we have this a lot of computation a b c and d and all stuffs right so these computation basically will have uh from backward so we are going from outward which is like out to the inputs so this is going to the from out bar which like the out bar will be here and then it will discover the inputs which dependent on the out. So that why like it's it is it is not in the right order.
So we want in the right order for the graph to execute the forward pass. Uh and so there will be two kind of of these things and uh this will be one is stack and other will be out which will have all the in the right order whatever the right order will be. So the the first it will the f it will contain leaves first and then contains the all the output and stuff. uh I can explain you better if I have like I have chat GP asked about like to create the computational graph so you can look at those graph let me just pull this thing so yeah so let's suppose like you know you have this graph zed zed is the output and then you will traverse back to discover some of the inputs so zed is dependent on x and y okay and then x is dependent on a and b and y is dependent on b and c. So you will see like you know um you probably find that okay when you go in the in this direction you will see you will check the inputs Y okay and then uh you will add in the stack you will add Z in the stack Y in the stack and then add C and B in the stack and then check another input X then B and C.
So you will actually create stack like this um from backward.
But the problem here is like there's a lot of problems with this uh which is like here you can see like this y is also dependent on uh b and x is also dependent on b. So if you just like go in one order and then uh discover the inputs then you will basically going to have basically add this b twice. So you will have two bs. So you have to make sure that okay you are not adding the the same parameter same variable and you are also adding these variable in the right order. So then we can actually create the graph create a graph. So yeah, I think this is what we want to do uh here right now to create this graph like first we need to define some of the things which is like graph and this graph is empty here and uh some of the variables like stack that stack is honestly the pointers of point locate it. It is a pointer to loc uh points the location of pointers. This stack and this maloc size of bar model number of bars and same as out and then visited because we have to check like we are not adding adding the same variable twice. So visited will be uh B32 booian visited um will this will have this will also be a pointer and this and then basically You know stack will have this uh initial stack will contain stack size.
Stack size equals zero and out size equals to zero. Stack stack size plus equals to stack size stack size ++ equals to out. Okay. So this will contains the out bar and then we will traverse back you know wire. So I hope you understood this like which is this is a post increment thing. So it is exactly like this stack size. So initially it will get the value of like zero index stack and zero index we are storing the output and then stack size increasing after that. So stack size will become one. While stack size is greater than zero, what we have to do is like we have to go and then pop out the stack.
So back current and uh stack stack size this will pop the index. So this is a pre-increment.
So it will turn to zero and then it will index.
So this current is stack stack minus minus stack size. Then what will happen that we have popped out the stack size is zero. It has zero length.
But um we can actually see if this current dot u current if this index if this um visited current dot index if it is visited then just add in the output you know add the out add in the output out don't go further just continue because like we already so as we have seen mean like okay if it already visited this kind of a thing then just uh add into the outsides and continue u otherwise like just go and do more computation which is more computation which will be like just discover more inputs. So let's suppose what we are trying to say is like you know if you have not visited this thing you discover more inputs and if you already visit visited this variable okay don't discover this okay we will add add this thing into the uh output I hope you're getting I I know this is not kind of makes sense but like when we actually run this create graph you can actually understand we will see like what exactly it's doing through the output So if visited current index out out size++ current continue and uh and then then then then we have this for use 32 I = to0 less than input size. So we have to like find out like the inputs which is the if type is not null type is not null type will have number of inputs otherwise zero. So if it is inputs I less than input I ++ input is okay. So if this input if this input is um if this input is less than the input is we have to like iterate uh for U32 K =0 J less than S it will make sense like I'll explain what I'm trying to do here I J= if it is like equals to that means like it is like a duplicated so because like as we have seen like X also had this B variable able Y also had this uh V B variable dependent on so both shared the same variable. So we need to make sure that okay one is only added. So first when we discovered X we will add the B. Now when we discovered Y's variable the variable of Y we can actually uh update B you know we can replace the stacks let's suppose like this it will add X Y B let's suppose like this B and A right so again when it finds that you know Z now in next iteration like it if it finds like y is also dependent on b you know b so this condition matches okay this is actually already in the stack right this is already in the stack so it will going what we need to do is like remove the old position thing just have this in the here so it will re basically remove the old thing and then have this in the right order.
So only one B is going to be there.
That's what we will going to do. Um it's already there. How we can remove this is basically we can shift the order. So let's suppose when we discovered this you know this is here. So we can just remove this and uh shift the order. So how we can like do this is basically we will just go iterate k equals to j. So we have discovered this at the j position. So we started with J and J will go to the stack size minus one and uh K++ and [snorts] this will what will what will happen that stack of K equals to stack K + one right so the K + one position will become the K position so Yeah. And then here we can just do the stack size minus minus. So one position like because one position is already shifted to the left.
So reducing the stack size also and that's it. Um what else is needed like we haven't stored the inputs. So this was the input right?
and uh stack stack size plus nice that's it like this is the graph uh so it will run the this traversal until the stack size is not less than zero and uh pop up the st pop the last um last thing in the stack and um finds out if it's already visited. If it's not then uh continue here if it's v visited already just adds into the out um stack and this input is basically will discover the number of inputs and then I trade over the inputs and then I after I trading the inputs let's check like whether we have already uh added this in the graph in stack if it's not all if it's already added then reorder the position and and then remove the last one. So, so kind of a graph it will have you know and we can see like uh we can print it that will give a better idea. So first let's let me find the model state you know I don't know like why it is not getting initialized in the right way I guess uh okay this is done to be honest like I wanted to check like this thing but we haven't like defined the model yet like so we need to first define the model and then we can use that model to define a model state let's define the model uh let's see I'm going basically copy some of the things here just the imports because imports is like difficult to manage stuff um in C and uh yeah to base arena m we don't have anything in that I guess can define the m okay nice this should be let's define somewhere else also okay so Because of this it should point the input is coming from the current.
not from here to. So here we can define like the actor model just this will give us model actor model because we are going to use re uh reinforce only not ppu and stuff right now which I thought to actually also implement this but probably this were going to a lot long so yeah let's go with like act actor model only it's like simple not do much in this. We don't have a value function in here which is great. Um but it will be very um high variance. So make sure like when we were going to train we'll see like a very high variance and it will train in a very long time. So yeah I think create actctor model model state model it will have you know the weights and stuff. So where input arena I guess model uh the input will be so we are going to create some uh kind of we can create a graph later on we can create a graph later on uh so when we have like a model state defined what we can do right now like and we can uh finish this function which is these things so you have actually just Notice that it requires a var type you know. So for that var type I'm going to define the var type here which is bar soft max which will have number of inputs equals to 1 dot name equals to soft max and uh I hope like it's the right syntax number of inputs shape will be okay we haven't defined the shape you know so there will be two functions actually which is bar B32 which is like boolean and bar shape this will contain the bar a b and we will check if in this implementation I'm not checking too much thing making it simpler so we'll only check like if a isn't equals to a dot rows isn't equals to b rows or our a columns isn't equals to the columns return false because this is a same shape thing. The function the goal we are trying to achieve with this function is that it should have a shame same shape and uh this same shape will going to have also update the rows and columns because these rows and columns here you can see like when we are creating the node we are calling this shape thing and we are passing the reference so this will change the rows and columns here which is which is defined as 0 0 it will change to the right colum and So this we will dreference this and we will update this as a and similar to like where a b. Now in u the matrix multiplication it's kind of different.
So if a rows isn't equals to a columns isn't equals to b columns. Oh my alarm is when did I have alarm? I did have alarm. I I I didn't know like I had alarm. I always I guess I always had skipped. This is the first time I'm discovering. Anyways, it's 5:30 a.m. right now happening though. So if a columns isn't equals to uh b columns and uh so it should have like this kind of because in matrix multiplication you basically compare with like the first dimension of the matrix last dimension the first matrix and the second dimension the first matrix. If it compatible with this then it can only do the matrix multiplication otherwise it can know. So return false and uh dreference this rows and change the parameter will be a rows and columns will be b column. This is it now. Yeah we can pass these things into here dot I I hope like I'm defining kind of the same. I have to check and then forward function which is going to be you know soft max forward function and uh dot backward function will be soft max backward function. Okay. And uh we can define the same thing with further also v= to dot name although let's check like if it's compiles. Wow. Because because because because it is giving errors because the thing is that it is a bar you know so it will have val is defined here. So we can also this should be wide. Wait this should not be wide. Okay. So I guess like the the error is basically because of the signature is not matching with this error is because the bar soft okay so I found it like what exactly the error is uh you know so this is just our definition right now we can define it not coming right now this is good um now what we can do um we can here we can here define these functions you know um we can put information inside this function where softmax this will basically do the softmax operation.
So you know soft max operation we can just define it here 32 which is what is soft max operation if if you just think of it or q32 inputs uh right have inputs and uh so we can actually define size so u4 size equals to you know and iterate and compute the softmax so I've already like did some define the actual thing like this is soft max and this is the formula really this is this and policy gradient like reinforce this is it so you probably have familiar with this I guess um and then what we're going to just do is like we will iterate over this Z to collect the Z I and then compute each of the uh with each of this element and similar to all those goes you This size will come from bar columns and then go to size I ++ you know if now what we have to do is like okay we have gone into this size now because if you check this formula I hope I'm recording this if you check this formula you are basically doing the max of set so first data looks like a max say 32 F32. Okay. And then we can do uh this should be the output of it. So I guess like we have defined it the right way. Input is the data which is the I= the formula I check equals the I guess the max value like this. we have to just do the uh sum and then we can divide this uh with the thing right this is the this is this this is the thing we have to do um var val and uh this is only called max value then i then we have to do the sum because you know uh we also have the sum and then we'll divide it with the um we'll scale this later but like let's comput the sum sum first and then we will okay let me explain this again I guess you haven't uh really got what exactly I'm saying to to compute this uh softmax forward uh what we need to do is like you know we have to compute these formula and it's it's simple it's not really difficult so because we are we have this V we can get actually the inputs so like this Q3 we can calculate the size which which is the which is the which is the bar val columns and multiply by v val although like it this is not a matable operation so I'm just like doing val I'm just resting on the val otherwise like I have to I can take the take this from input as well so for 32 I zero I go size I ++ now because if you see this this basically like we have to compute this in element wise where the uler has the power of like ZI minus max of Z and then it is normalized with some summation although the this is exactly the same like this equation and this equation is same. It's just like it is summed over it. So the easy way to do is like you basically I less than zero you do max value max f32 max value is let's initialize with this v inputs zero is also a val right input is also a val I forgot that like input is also a bar so will have a data. Now this max value going to be changed because like we need to find the max value. It can have max value on this thing. Now again go for height size plus and the sum you know. So sum will be and which is the this function which is we are trying to do is and is the function name uh it's x f you know um the xf and it will basically just and z i is basically you know uh is the input.
So this will going to store in where this will going to store in var val data i. And so the upper part is done like this will also compute the upper part and then with the sum we can compute this um and then we can actually divide it you know. So how we can divide it divide with the sum. So go with the U32 I = 0 size ++ then that's it. This is our soft max. This is our soft max and exactly this is going to be for the value also that um you have the size then it trade it over it and uh in this iteration I hope like I'm audible to yeah in this iteration in this iteration we can just like go and then see what exactly is happening like because really is basically turning all the negative thing to zero. Z and then all the positive thing to whatever the the value they have. Right? So if the data if the V dot inputs of zero it will also have the only one input which is zero. V data I is less than zero. then v input v val data equals to zero. Else if it's not like this v val= to inputs whatever the input is just store there and it should be input.
Now for the R operation it's R operation is good it's not very complicated.
So again the size will be this and in this size we can just say you know if val data I = to bar inputs zero val data I plus v inputs zero v data I should be one. So we have element wise addition right the same goes with the sub also we just going to subtract it here. Now math m is kind of very very different because it will have you know a lot of things which could be a transpose non-transpose and stuff. So for this one like we will define another set of function. Okay. So, so the batmo is kind of different here because it will have a a transpose and different kind of stuff. So, what I can do here is like no here we can define another set of things which is matrix. This matrix multiplication v4 will have uh you know we are not going to take anything out from the function just actually means no transpose in either side of it.
So it will basically going to yeah I guess like we don't need this kind of a thing void because we are passing the out metrics matrix and uh a B right and for do I = 0 I less than size I ++ basically this uh this this this this a you know uh how we can compute the size which is we don't need to like to go in the same way like how we went through in the last add and uh soft packs thing but rather we can call this And then u32 J = J less than matrix A to columns and then A++ for Q. You can say I 64 because you know maybe matrix is bigger very very key = to we need three iteration because you know the first is the out rows and then we are going indexing on the actual matrix A which will help us the second uh second iteration will help us to indexing on A and B right and um the out which is again going to less than these three indexing which help us to which can help us to do the whole thing. Now I'm going to explain again. So white mat nnn this nn is basically the uh not transpose. So it will have this out matrix matrix out matrix A matrix B you know and if basically we can just 64 I = to0 we are just using U64 here maybe like matrix is very big although like it doesn't make sense why we use only U32 in earlier version but it's Okay, audio version means in soft max radio we could also use the U64 it will go to the A rows columns sorry a out plus I ++ you know so out to the rows plus again J = to zero J less than columns J ++ for K =0 K less than it is K K less than out columns K + so basically these three iteration we are going to do to actually index the matrix. So if you don't know like the matrix is basically just the you know how we index things because we are not using like any kind of array we are just like indexing through the the actual memory. So if you think of this is as a matrix then 1 2 3 4 5 6 7 8 9 you know this is a matrix room if you want to access the memory the location of this matrix the row major you basically mostly use major basically and how you store the how you also store in the memory of the metrics which is the row major majority of the time which is like your flattened memory will be 1 2 3 5 6 7 like this. It will be like this. And the column major will will be if you want to store with the column major it will be like 1 5 93 like like this 1 93 26 10 like this.
So here we are using like a row major thing. So it has a formula to access the do the indexing basically which is like column plus rows number of columns multiply by number of columns.
So row multiply by number of columns will happen first and then you just add with the column. This is exactly like you know this is exactly like telling you have this matrix M and then row and column and like probably you have used this in Python which is like slicing this is row and column right um so this is how you actually store these things otherwise right now we're going to use this formula which is column row so here's a twist Let's suppose if we want to transpose because you know if you know more about transpose if you don't know transpose is basically transpose is like changing this row to column. So this will become column and row and then so this thing will also change. So this will become row and column right? So this will change it. Now when this will change we can able to define functions like with if you want to multiply with transpose or not transpose. So we can also change the formula and use the right formula indexing thing right. So transpose and basically you it is exactly like a transpose if you do a transpose on i on k it is exactly like a of k and i. So here the out of data will be I + K I + K. So sorry I + J J plus I sorry if I just say J + I because J is the column and row should be here because if I just using this column plus rows multiply by number of columns. So J I 10 columns equals to you have this out a data and uh which is K plus I which is so here you can see like K is zero and I is zero the first index and then you just and if it's zero then basically this whole thing cancels out and you're left with like a K. So you K is become like a column for you here and row is zero. So here's like K is something that I able going to index over it and uh you're going to do the dot product with B then B dot product will happen with the J and K. That's it. Like this is the whole formula. This is this is the thing. Now the cool part is that you can just change the indexing you know let me check like if it is correct. Yeah it is actually correct.
Now this map n can be like nt which is like a transpose the p metric. So you just have to do you know what you have to just do is like you have to just change this order just I + you know so if you just change the order sorry uh I guess like I change the A of it which should be the here K + J then B is transpose.
Now if you want to transpose this t and n then basically a should be transposed.
So if a is transpose so the iteration we are doing over the k over the a sorry will become rows here.
So and this will also change i + k. Why it will become why it will become rows?
Because you know it will also it is transposing things you know the A is transposing because right now we were actually moving over the A. We're using A to it but like A wasn't changing b was changing. So our iteration loop was exactly like this. But like right now our A is affected which was using which we were using in the um in the iteration. We can also use the B.
So that will create a non effect thing.
But like right now it's good that we are using only A for the iteration.
But as A changes, A is getting transposed. So we have to also change the iteration loops. Okay. So and then we can have like a TT which is like both are changed. So now it will become I + K K + J then also become like rows. Now we can just like define the final ML which is like case of it you know matrix A and matrix B. Now this will basically you know A. Okay. So this is like just a zero if if all are zero then we will use this n1 and if n1 is true we can use this thing. So we can define like um case thing like you know there's one more thing like we can also say which is zero out. If you want to do the zero out although like it's not very important.
And then we can just get rid of both because like if both are true then it's already going to be tt. So zero out will help us to basically initialize the matrix with all zero. Um so f0 out. So I initialize with for u 64 and uh transpose transpose a transpose b if it is kind of a situation and switch and then I can just copy this case that going to be. Okay. This is this now we can utilize the same manual here. We can see let me just check if we haven't made any mistake. Yeah, that's fine. Let me just fix this quickly. These errors.
Let's implement this reinforce forward which will be the the computation of the log and the returns right uh it will be the logs and the returns. So the log of probabilities basically will be I can just copy it from here size right and in that basically what we can do is uh to make it more safer because we will get like a probability from the inputs. So max of I2 inputs zero this is the uh this is the max or basically do the one will be the returns this is rainforce forward okay now we have to compute the backward passes and stuff so backward passes the gradient we have to compute it's very simple uh also like it's not difficult I have tree and what will be the loss? So you can see like in the for the reinforce loss the green will be like GI times the RA I which is the reward here divided by PI and negative. So and really for it is x if it is greater than zero then ti it's like the the previous gradient or the evaluation gradient of value or if it will be going to zero and then soft max gradient will be this. So yeah, I think let's let's compute this first. We have we can compute the reinforce gradient can do like this and if I guess this should be 64 and uh and then over it and uh grad will be TI is basically the ground. We can also uh do similar kind of thing here as well.
What we have done there is I think you know if if this is like this you know in the input uh this is getting very confused you know but the thing is that I want the prob should be the uh gradient you know so input will have this input will have this this zero index is probability you know so val I can say I guess this is something like this I think you know it's basically a transpose so if when you do when you do uh matrix multiplication it will going to transpose so for the gradients you know I will do much better if I just if I can just do it like this a and b and we have to compute the gradients of a and B. So here you have seen like we haven't computed the we haven't computed the gradients of inputs two because with input two is basically RT which is the minus R it's like reward here. So we don't need like uh to compute the gradient of return. It'll just it is just a weightage. So we can just leave it and then we can only compute the gradient of the probability which is the probability of given action and state probability taking action given state you know. So if variable this flag a we can just say a flags then we can do match we can pass the inputs 0 b transpose will be because it's a so transpose b will be transpose here to calculate the gradient uh for the a having the gradient of a will be basically going to be the transpose of AB. So AB transpose B transpose and this here A is transpose and B. I hope you're getting it. So yeah for the addition it is basically the addition.
Yeah. Now we have to do the same thing which is like addition and uh put the entire thing in here. It will be not just like plus equals to but rather okay here we have just did some kind of a mistake which is it will be no it is fine okay so some errors which I will solve here 394 this is for the A and same goes for the B so B it will be this B is b going to be 1 one and that's it right like for the subtraction I think for the subtraction it is also kind of a similar I guess it's it's it's very similar because but here it going to be negative we can change this to now again like this input equals to only one input okay and size will be this size is you know this size this uh and in this input what we're going to do is like if equal = to 0 is greater than zero this other that's it yeah I can also add one more thing which is like a floating point right and it's going to be plus equals to okay now we only have left this soft max so the easiest part about this is that in the vector form. This is uh our equation which is like a dotproduct of G and S. S is basically the soft max data and G minus G is the gradient of the bar and uh G minus G multiply G dot product with this and then soft M data again dot product.
So let's compute this dot product going to take this and uh also going to take this I will say dot because you know we want to use the softmax data this is the dot product it will do now again we have to do but this time data dot product dot dot dot s minus s okay so I hope The computation is right. Okay, we have implemented everything now like all the things which we have defined as a function. So the next what we have to do is like we can start building up a model you know because we have built all the things required to build the model. So we can start building model and for that like model C. Okay. So we have implemented this autograd thing. Uh now let's create environment u or we can create model but I think uh I want to create environment first. So environment let's see is this the here we probably going to do is uh create some strct and enums. So let's first understand what kind of environment we are trying to create. We will create a snake game environment. I think this is going to be a really simple one and this uh which can actually have some level of difficulty also which have reward structure where the snake will get reward and if it hits the wall it will get penalty and stuff.
So this will going to be simple snake uh simple environment. So let's go just going to have like actions action left uh left will be zero right will be one up two down three right uh this is going to be action now snake env this will contain a lot of information like you know state uh which which has a state uh variable type so this state variable type will be Drop right and this state is not state it's snake state food what will be the state of food and uh score score will be f32 score and uh action will be POV like what exactly the point of view because snake can point because in snake game you probably have seen like snake has a point of view it goes in one direction then we if In keyboard like in keyboard phone when you press the low button down button then it POV changes and then it moves and downward. So something like this it's basically the point of view of the snake. Now this is the action POV and then what we'll have POV. That's it. I think what else we can have this current state will be snake state food then score this. Yeah. And uh then we have like this uh type defctr. I hope it's recording. I guess yeah, it's recording.
Type def.
Then we have a replay buffer, you know, because in reinforcement learning, you probably have like heard of this kind of replay buffer where you do the gradient accumulation later after doing the rollouts. So you do the rollouts. So I can explain you like this. You do the roll outs. Roll outs. You do the rollouts is basically collection of data. Collection of data. You do the roll outs and then after doing the roll outs you do the step gradient step using that roll out. So you collect the experience. So that's why like AR is uh is basically doesn't require training data because it generates the training data uh the pure AR thing. So this will generate the data the experience from the snake and then we will use this experience to correct the agents action.
So we need to store these experience uh that's what we call like replay buffer strct. This replay buffer you know will have um a lot of things also this snake game will have like a rows and columns and the grid size and action POV and number of steps it had took. So use 64 steps this replay will have you know the first is like trajectories and this trajectory is itself a shock. So let's suppose the proper size is 512.
Okay. And uh episode length because episode is basically snake is collecting the uh states and actions in a single game. So it goes to x0 y 0 like the coordinates x1 y1. So it basically navigating the environment. So how many steps it is taking in a game is called episode length because it cannot like take forever. We have to like it can take forever. It can just like take uh until it's win or lose. But like let's define a definite uh steps. So it could be like 200 right or maybe like we can define more let's go with like 100 and uh this can be in,024.
Now this trajectory you know this trajectory contains a lot of information which is like states state episode length state it states which is like all the states the snake went through and this food states episode length action episode and all the action it took and uh action what is the POV episode length rewards and returns. Um F2 returns and uh state next state because we have to store the what next state it went through. What will be what was the next state after taking the action and uh next states and done like is the game done? Do we have a result of it?
So not then we can actually again and again. So this is some kind of abstraction layer in trajectory we can say 32 count right and here we can also say length. Now this is the abstraction define what exactly is exactly is this error just pass the chance of why we getting this error now the error will go away.
Now create the env. So this basically create env because we are not like defining this in arena now like it's very array kind of thing. So here like in trajectory and all stuff we haven't used any kind of a pointer thing but that's okay I guess like let's go with grid size rows will be two side because it's the grid size grid will be a square. So it's like QRTF and uh credit size right and this Q32 calls equals to side same thing then this will have like a columns and rows and then we can define like snake takes the push truck which is the arena itself and the size and all stuff. So I'm going to go like there's other ways which is like maloc you know we can just say and say size of this thing size of CNV but like let's go with like arena arena arena and now 25 or 36 let's go with like 36 size okay just created the env here you uh snake dox = to zero that's how we can define 0 and actual POV right so because it's starting from 0 0 so 0 0 will be like top left bottom left and then in the x and yaxis x and y graph so 0 0 will be like a top b sorry bottom from left and then it can UV towards the right. So this will define the ENV if we can actually enter.
Okay. Why we were doing this? Damn. What exactly is this? It should be zero, right? I think it should be zero. Um I guess like you know we have defined in a wrong way we can define like this uh which is much better just to m like this why it is getting like this long with this damn it when we are returning bro that's it small small mistakes okay so now this is done we have created the env U this is our env and uh we need to find the you know we need to reset this state also. So after episode we need to reset it reset again like this going to be like this but we can make more interesting here which is like jet phantom lock. So this will give you the random food location based on the number provided number provided which could be so I'm not doing any checks here so just to make it simple but you can just think like okay there will be a check which checks the range within the range of like grid size and rows and columns. So if there's function random in the PRNG this file which is which is which is which is this function define it like this this will be I32 because this can be negative yeah so that's why define this side here why this cut it got cut it cut it and same here now okay so this reset env is done again you know you can see by passing it we are getting random now taking action we we need to make sure that we taking the right action in the environment the valid action so we have to define that you know action this will basically ask game action action there will be one more action here Okay, that's it. I guess there can be one more action like which could be none like no action. Okay. And uh take action. If a np action if action equal to write no action equal to none action if action is none then action will become you know POV env is POV like because we are not taking the model decided to not take any action. So the POV the whatever the POV of the snake will become its action you know. So just take action make push snake in the POV direction because snake will going to take action at every time step with its POV. What the model has to do is to take action to prevent it from hitting the wall or like to get in the direction of food. So if action is none there's this action is right PNV X will be snake will be snake X will be + one I guess we are okay so this is the and then the end can change the POV to action whatever the action was okay that's it in the else we can return action is The snake position would become should be one. Nice one. And if it's left then it should be minus one. And if it's up should be the y should be okay. Y it's been a lot of time. So yeah. So this is working right. You know what we need? uh we need to build uh some sort of vector which is the some sort of vector in the sense because you know you have this inputs and math and stuff like that. So this input will be like some kind of embeddings and some some one hot coded vector which has all the information about the snake world or snake env you know because we don't have any kind of like input like text or something like that but this snake env world which has a lot of information like where the snake is going what's POV what's date and all stuff it's actually a lot of information so we will provide this as input to the model and model will actually find patterns and observe it and produce an output based on that. But we need to convert this input in a right way so the model can actually understand it. So uh the best way to do so is to create a some kind of vector right. So build state vector which contains some informations like which has like metrics and state and then state of this will be state of snake and food and uh action what will be the POV what is the POV of the action and uh columns and right now to compute the snake I we know the formula of this this column row in row major thing. So in a grid size like we are storing suppose this is the state it will have like for 0 0 0 0 0 and then for like grid size till the 36th 36th of 36th of this thing length or I can say 35th because if we have like 36 grid size have like this kind of a state and snake eye will be you know one here and one like if the x and y is like this. So this will have some kind of a information in our flat vector and we can do this same with the snake with the food also you know snake position it's like a one hot encoded thing and then for the POV we can for the POV it can have again some kind of a similar similar similar vector which can actually have some information. Well, we are going to create a very simple simple vector not very like kind of a embedding type dimensional thing. So just a one hot code which has some information which can actually help the model to absorb it. Q snake I which is uh this columns plus rows rows is basically Q32 snake columns will be snake doy right and uh multiply by and then plus like it's little bit weird I would say but I can define it like this x state doy that's kind of different here but like it's Okay, columns is x. Sorry, I'm just got confused in this state.x and then plus state snake snake doy then multiply with columns. This is the item exactly with the food and then we can store in the inmetrix which we are which we have here which is the input matrix data snake I just snake 1 0. So whatever the index of the snake I will have like a one zero floating point and in data I size plus 0. This is a flat vector which have like a you know in Python or like in other matrices we in numpy we actually see shape like this you know where we have this three three inputs and then the grid size but here we just we have flatten thing right so three into grid size so this is exactly our input vector there will be like a another function which will help us to find the reports Because you know we have defined the take action but we haven't defined the reward function which help us to get the reward. So when the agent will take action in this let's suppose if it finds the food and uh finds the food of the state uh then we have to give them a reward and if it hits the wall then we have to give them a negative reward. So right this will give reward and it will it will have snake.
Now if env.x X dot snake dox greater than less than zero or R env dot env snake dot by less than zero rv if it is like this you know then give them a negative reward or something like that right r env snake dox is greater than the env rose R or we can say to I32 and if it's like greater than the ENV then it's like we should give it reward F32 give them uh basically minus min - 5 okay 0 F now if it lands on the food we have snake dox equal equals to env you know and then envake doy equal = to food doy then reward give them reward like + 10.
These are the two things basically. Um now it is done. Um like it will get the positive reward here but like now we have to when it eats the food when it actually lands the food we have to give them a new location to this uh food. So that's why like uh new food will be we'll do do while do while new food equals to state get location close. Okay. And if new food new food dox= equals to inv snake.x X and and means like if we don't want like where where the position of the snake is we don't want the food should be again come at this position so we'll just like okay if this is the condition is true then you know run again iteration and find new food a new location that's it uh that's it I guess and uh yeah have reward then turn reward are all done. We can actually train this uh simulation this environment you know we can simulate this environment and train the model. Uh we haven't defined the model yet but it's okay like we will define later.
So here we can actually create a training pipeline. training pipeline will be simple also it won't have much uh there like you know in training pipeline we we have this training of RL and in training of RL you basically do rollouts collect experience you will compute the returns of this compute the returns of the roll out and the buffer and then sample experience from the buffer and then train it. This is the whole pipeline of the RL training you know. So void train arena I guess like there's no arena model state model and uh snake env memory there's this arena I don't know why we need arena we need to create here but okay uh the first thing we have to do is like you know we'll do the forward pass and we haven't defined the forward pass you know because uh roll out will do the forward pass and Well, I will just you know uh for the ease of this what I will just going to do is like I will paste this epox roll out size episode length and gamma which will help us to basically these are the hyperparameters and uh let's see if it's compiling okay uh we'll we'll see this later here you need to do is like power pass Right. And backward pass. Right.
This forward pass is basically will have arena. I guess arena. No, this forward pass will have you know graph.
Now U32 is the size of the graph dot. This will be a pointer and then U32 I =0 I less than size I ++ F. Now iterate the Irate the size. Uh what else we need from here is access the V because you know we the graph will have that build graph we had here create graph. This will give us the graph from the forward pass and we can use this to write where it went. So we have to iterate over this I compute the bars location right dot type current type which is like a itself is a bar and if it doesn't have a type type isn't is equal to null return otherwise just to because it will have a forward function type and this forward function you know will need a bar which this again. See that's it. This is the forward pass.
But the backward pass here uh it will be exactly kind of similar to this.
But if uh if this is not true, if this is not true then what you have to do is like continue and if you have to continue again and if it is like a v require grad if it's parameter which a v require grad then we need to clear this thing uh it's grab. So this will clear the gradients the gradients.
So clear the gradients and uh then you can use fill uh I guess fill is then fill the graph r² minus one the last variable should be 1.0 right uh so fill the last variable last gradient last variable's gradient to 1.0 Z and then traverse back basically I 32 I 64 = 0 less than I I should be sorry I should be I zero I - reverse back and then you know where where check if it has gradient gradient then you know current equals to zero Then continue. If current dot type type isn't equals to null and current type backward isn't equals to null. We can change to current because like we already find C. Uh okay this is good. We have implemented the backward pass is not defined.
Why fill is not defined because copy this there's some function we can actually import we can define in which basically fills and fill the random we don't need this kind of thing okay is compiling then in the in the arena now we can go and train it proper so this will we will do like a roll out experience collection so 2 I epoch = to 0 to epox++ right and then in this epoch B probability equal to forward pass how how we can do forward pass from the model because we have model right so but I guess you know in here model state when we have defined it has only these things. So it will have forward graph via graph output. These are the things model state will have. Again go for J =0 I less than roll out size I ++ this will do roll out. Now we need to take a step in the environment. So let's generate an action first. Let's see like if it is working or not. So we can call train model state. So this training function is working right. Uh then what we have to do is uh what we have to do is roll out sizes and then then then then in here and say model dot forward graph is equals to build create graph and uh you know the create graph what the create graph will take arena model state and out the output will be you know And the password graph will be other output.
This will be a cost. That's why like we need to define a cost here. It is null.
So it will probably give this auto graph return a pointer of graph. And this is not done yet. We have defined this graph as this. I guess you know we don't we can change the definition of the model state that we don't need this and uh this will graph dot because you know we are actually adding in the out of it here out so the graph dot varss will be out now the graph varss is basically not defined yet for struck pull struck or basically we can do there's There's another thing which is push array to int. Uh it will saves some memory of it but basically just not initialize this array like others like you can see it is true. True means like it initializes everything with zero but it doesn't. So yeah, graph bar isn't equals to null means it has space allocation after class equals to graph dot equals to dot number of bars equals to out size and then we can just do me cpy to move the variables locations pointers to here from out and size of will And then we can just free the stack because it was a temporary location which visited this not number of hours.
So the issue is uh issue was coming from here this this kind of a issue because defined after this I'm using here. So that will resolve issue and uh okay so this is done. Uh now what else we need here? we haven't defined these functions you know so bar softmax which basically creates the the variable of softmax so we already have this create node so there's not really needed for any kind of logic just need to call these functions right so we need to call this return it and this uh takes up some input which is arena model state variables Okay, that's it. And then v type you know a and b is basically input null v type. We have defined it, right?
We haven't defined that thing. We haven't defined the v type really. Oh wow. We have defined the bar this thing and rel for the rel dot number of inputs equals to 1 dot shape equals to where same dot forward type it will be exactly like this. So this will be M inputs where shape M forward audible. Yes. What is this error type of max? I can find this. Let me search like why exactly I'm getting this error. Okay, it's again like the same issue. I have to put this in here. This is a big issue. So what I will just going to do you know I'll just define definition here.
So it will be better. No it will not going to work I guess because it's static. So okay we can complete this now type this one is rel this is nice u so we were completing the our training pipeline and this we probably you know take the action so let's suppose the action for right now let's take the action of this thing you know print of action you this is the action on this action is okay this is working I would say we have like a five action 1 2 3 4 five right so it should be five why sometimes it fails I just don't know but it's okay like we have this action and uh take action and this is the env we can provide after taking the action x will be taking this thing. It does not take action. Um Oh god, it got it. Got it. It can happen because you know is game over. What we need to do is like v 32 env snake.x less than zero or r. It's exactly the same. It is working right now. Much better way. But okay. So if this is true then okay. Yeah it's working. And when it gets minus one it is it. Yeah. When it gets minus one the game is over. Right.
Uh see I'm going because it is like it's been so long and we haven't finished yet. So I'm going to take this so fast and uh let's go let's go finish this. Uh okay, this is uh how the game will over you know and uh every time we have to reset the state for every roll out size we have to take this thing and uh for the trajectory trajectory define a trajectory trajectory which is a buffer and this is this buffer you know it's coming from here and uh this has I okay now we I can store it inside this trajectory the information 32 I I go to episode length okay length is I already defined define but it's okay I ++ now in this um inside this we will take the action now we have state equals to ent state food state okay action P is np. Now we need to build the state vector and inside this input model val. I hope it will work I guess because val is not as null state and pv columns nv rows like this it is going to fail because we don't have model right.
So what I will going to do like I will you know quickly I will just going to copy and paste model because I have a model model is simple it's nothing here so these are the things it has create after critic model which is basically arena and will take model and uh this is input and we are creating a bar of size like 76 and one because you know 36 is crit size and again 36 is for the our uh storing the food position and four is for storing the POV you know. So this is 76 one bar magnum actually I we have five actions but like eventually the actions becomes four because you know if it is non action then we are actually defining it. As you can see here action equals to. So yeah even if it's a action none we are actually changing the action. So it becomes four action that's why like 76 we have to add one more. So uh this weight weight zero is we are creating bar create for this 128 76 it's 178 um size and the bias this is the bias and we are providing the flag of it which is a it's a parameter and also a require grant for the input it's required none basically doesn't need to have gradients and it's not a parameter also it's just an input So and then we have this uh weights one and bias one weight two and bias two with like 128 128 hidden dimensions and the bias will have 128 one dimension.
Okay. And then here we are initializing this with like rand and all stuff uh which we had already defined I guess or if not then I think we have to define it here and we can import stuff here. Okay, just remove this. This is that's why like it is. So let me just like think like what exactly is the error. Okay, so the error is gone but I have to change a lot of things because the code I pasted was different what we have created.
So like different kind of function name and like parameters required for that.
So yeah I think now we don't have any error. So yeah, I think this is uh weights bias weights bias. We created like a three weights and three bias corresponding to that. And then this basically randomly initialize this things and um you know it will initialize the layers and uh z a is basically the the map of weight and input and then we are adding v adds using v add we are adding the the thing z a and b kn. So that's why like we created this v mat where add v which basically creates the nodes and then do it will not going to compute right now it will create a nodes that what are the operations required okay so it will create each node like this and then similar to that we have another matrix multiplication with weight one and the input is a not which is the previous uh output it exactly like PyTorch we do in uh in Python but like we we are basically doing more scratch way and can see layers in front of it. So Z 1B is basically the bias again is added with Z 1 A B1 and then this is the radio layer after to normalize this thing. And then for the again doing the matrix multiplication with the last weights and then doing the addition bar add and then we are basically doing the computing the probabilities using v softmax and defining a model output equals to output. Okay. And advantage is basically the it's actually none right now like it doesn't have anything just create a bar and it does not have any information but we will add the information inside this advantage during the training and this uh cost is basically the reinforce loss we have so reinforce and then basically it will have like a bar type reinforce and output and advantage it will have then the cost function is this right uh now Here we can now say no we can now say create actor model arena model. Now this will work. Okay.
Okay. So there's something which is basically doing some sort of you know we can use a coding agent I guess to other it will take a lot of time.
So we can just say like okay what exactly is okay so these are basically the errors uh which is like null but we have created the model isn't it okay number of bars equals to zero because number of bars are basically kind of this condition we can actually have to secure the model okay we have a I get more memory. Now let's see. Okay, this is exact. Okay, I got it. That was mistake.
Okay, so the major issue was basically here as I was like very confused on like you know maybe like we needed 77 dimension because like we had five actions.
So I thought like no okay it's only four I guess because like action is taking but it is actually going to be 77 uh that was making the issue here and now it is actually working fine and uh yeah this uh will be cost graph. Now if you just run the training okay build the state and uh now let's finish this quickly. Um we have this uh arena.
C there's this you know we can sample the action uh using our probability distribution right now but let's not focus on that. Let's build the pipeline first. Take the action. We can take the action. Now we need to store uh the store the all these things. So if uh for this we have to like f2 reward at reward or if this thing is reward uh now this give function we are getting here reward then we can actually score equals to reward right and then done will be capted like here and then next state is basically END snake after the action the previous state was the like this you know then trajectory we can actually copy this and it will actually store all the information states food states POV actions reward all stuff this will store the informations there's no there's no rewards okay there should be a rewards yeah you can see this st because it's a trajectory we are iterating over uh done done actually there's still more errors [snorts] trajectory now this roll out is done you know after this roll out this uh roll out is done what we can do is like we can compute the returns compute the returns like because what exactly the returns of each thing returns that it's going trajectory of 0 to episode length and we will compute the returns which returns the sum of all the reports at each step in a trajectory. So I guess like this is buffer count it's not very efficient way to do so but like it's okay. So this trajectory gain the buffer trees F32 GG is basically return 0 F or 32 32. So you know advantage is you basically go from you go from trajectory length to zero and then computes the trajec uh the returns at each step. So t equals to trajectory rewards rewards of t plus gamma returns returns of t. I can uh basically paste this algorithm. This will go to trajectory and computes the using gamma. Gamma is basically used for is a discount factor and then trajectory returns is g and then sum return sum. We haven't defined any kind of return sum here but like so we don't need this return sum return square sum yeah it is useful like u standard deviation calculation sample count uh that is also not needed average return well we can actually define this thing to average which will be outside of this Let's print this.
Uh we are going to we also have to add this in our buffer. We are getting returned which is like in negative. So our snake game basically snake is getting killed every time. I don't think I can find any positive here. Can you find any positive? I don't think. Okay, this is good. What else we need?
uh we have computed the returns now it's time for to compute the gradients and all stuff which is like actual training thing isn't it so we will going to go again and backward pass in that backward pass you know we have computed this thing which is clearing the gradient clear the gradient uh so again like we will do this before computing the gradients we have to clear it. So 32 I = to zero. I will go to graph dot size I ++ and then there's no model.graph. Okay, this is clearing the gradient. Now what we need to do is like again this is a training phase. In this training phase, you know, uh what we need to do is like just go through all the buffer. buffer count is basically the rollout size. So U32 I = to U32 and then go to the trajectory the trajectory iteration.
But here instead of absolute length we will go trajectory dotlength. Okay.
[snorts] And uh again build the vector build this vector. And this time the state is coming from trajectory trajectory dot state no states of I and trajectory state tjectory.
Now this is done. Uh what we need to do is like again clear the advantages.
So for U32 I K = to0 K less than the model dot advantages this will go to the I we just want like because we will compute the advantage so just get rid of like the old advantages thing. So advantage now to compute the advantage it's simple advantage is basically trajectory returns of t minus the baseline. The baseline is basically the return mean and some okay this is basically the return mean. How can we compute this? We can compute this uh average return. We can not average return we can just say 52 return mean K and standardization is kind of so for the return mean I can uh like there was this sample 32 samples number of samples zero and return some average return is basically something else you can just like remove the average return and put this uh inside this number of samples you know in a trajectory and uh return sum was equals to g that's it and then after that you number of return sum f32 return mean is sum is sample number of samples So this is our advantage and uh yeah this is our advantage and then we can just model advantage equals to advantage right um this advantage is basically val trajectory actions. Okay, that makes more sense because you know so there will be five accents so five index for it but like it will only store whatever the action it took. So it will only store the advantage on the whatever the action it was. So action actions now just do the forward pass with given the model dot forward graph and then do the backward pass given the model backward pass cost graph actually. Awesome. This is actually computing the gradients. Now after doing this uh after completing this optimizer step you have to do the optimizer step. Gradient scale is learning rate divided by 32 samples count right. Um now what we have just done here I have to do the same thing and uh inside this the gradient current data will be into gradient scale. This will take the gradient. Well this will just the gradient scale but we need to also subtract this. So it will also going to subscribe subtract this and subtract current bar width. Inside this the out matrix will be current val current width current gradient sub will be not defined I guess because sub is where it is in the autograph can say like this. This is actually working now and this is the whole implementation of the reinforcement learning from scratch and see and I know probably I have made some of the mistakes here like implementing in life some of the few things probably although like the library is pretty big uh what I actually built but it's a very minimal implementation of it where I just try to get rid of lot of functions and try to attend everything in a single single file like it's going 400 500 lines codes in some files. So I try to attend in everything not try to manage everything not try to check the uh safety checks and all the stuff. So it was like very minimal implementation try to simplify this and I hope this made sense. So I guess like probably I don't like this will learn but it will going to learn probably like if you just like leave it for like 20 30 minutes it can find.
So you can see right now it has started getting positive rewards after uh I adjusted some of the things and improved logging system and stuff. So you can see like the returns are like positive earlier it was negative um it's negative negative negative negative then started learning and I trading it I trading it I trading and then become better so reinforce is a very like unstable algorithm you can say because it has a very high standard deviation but still it can learn uh it can perform really good uh when you actually train for longer time and you can figure it out.
So yeah, I think this was about the reinforce algorithm. In this video, I thought to actually implement also more like um more algorithms like PPO and um the the versions of PO and stuff.
But I guess like this video is going very long. So I'll cover this in next few videos or like in any other project.
But see you next time. Bye.
Related Videos
12:08
Artificial Intelligence
Expanding Stikbot thumbnails
leopoldshorts
2K views•2023-09-24
32:31
Artificial Intelligence
Digital Discrimination: Cognitive Bias in Machine Learning
redmonktechevents2974
4K views•2019-12-18
1:22:05
Artificial Intelligence
Evolutionary Approach to Clustering by Ujjwal Maulik
ICTStalks
279 views•2019-06-26
55:30
Artificial Intelligence
Rose Yu "Learning from Large-Scale Spatiotemporal Data"
networkscienceinstitute
2K views•2019-03-04
28:24
Artificial Intelligence
Stanford Seminar - Generalization through Task Representations with Foundation Models
stanfordonline
4K views•2025-07-14
8:29
Artificial Intelligence
Satellite-Based Wheat Yield Forecasting using GEE & Transformer Neural Network
gisrsinstitute
634 views•2025-06-15
32:06
Artificial Intelligence
Paradigm Shifts in Data Processing for the Generative AI Era: Robert Nishihara of Anyscale & Ray.io
GradientFlow
2K views•2025-01-02
39:48
Artificial Intelligence
How to Build Your Own GenAI-Based Knowledge Management System
2150GmbH
360 views•2025-06-03
Trending
13:38
Biology
Gremlin Arrives… While Dorothy May Takes Another Step Forward
The-moons
10K views•2026-07-23
13:25
Business
Playstation NO DISC/NO BUY Fight Is Over...
DavidJaffeGames
4K views•2026-07-23
16:52
Anthropology
Americans Confused in Australia for 17 Minutes Straight
IWrocker
17K views•2026-07-23
15:34
Political Science
FURIOUS Raskin CORNERS DOJ over Trump DARK PAST!!!!
MeidasTouch
237K views•2026-07-23