This approach marks a necessary shift from the vanity of massive parameter counts toward the pragmatism of task-specific efficiency. It is a compelling demonstration of how to transform expensive general-purpose intelligence into lean, production-ready assets for real-world engineering.
Install our extension to search inside any video instantly.
Distilling an LLM into a 150M-parameter model | AGI Tamer
Added:Hey everyone, I hope you all are doing well. My name is Monal and today we'll be solving a technology problem that comes from something that we all have personally felt or noticed while using social media platforms. Whenever we use social media platforms like LinkedIn, X, Reddit as well as Instagram nowadays a large number of post talks about AI or layoff. People are trying to showcase their products that they have built using W coding. They're trying to showcase the agent that they have created. They're trying to showcase a rack pipeline. They're trying to showcase a new GPU model that they have been using which is giving them better inference time and lower latency. So everything is about AI. On the other hand, people are talking about layoffs.
Companies are laying off their people.
People are losing their job. Maybe may not be because of AI. It can be a skills issue. It can be that companies trying to reduce it. But anyways, it is about layoff. So there are two different kinds of news that is AI or layoff. And for someone who is already working in AI, whenever I see this post that is too frequent for me, that is too overwhelming for me that new model has came, new agent is coming, new framework is coming, it makes me sometimes little depressed or a little overwhelming. So I was thinking that maybe can we how can we solve this? Is this a problem? Well, this is a problem for me and this might be a problem for a lot of users. This is more about user experience as compared to the actual technology problem. So can we solve this? Yes. So today we'll be doing that. We'll be creating a backend system, a core backend system that is nothing but we are going to train a transformer model. It's not fully a transformer but it is only the encoder based model. What we are going to do our task is can we categorize this post whenever we see into whether this post talks about AI, whether this post talks about layoff. If we can do that, if a system exist or if a system we can create that can make a post and classify those into multiple categories, then we can save that into a database. Then companies like this, companies that are social platforms, they can use this data to kind of enable user a filter or a toggle which can filter this post. So it will not be overwhelming anymore. It's like whenever we have toggle we can turn it off and maybe on weekend I need to talk about I need to see what people are doing not of AI not of layoff people are literally talking about that they have got this promotion people are talking about the skill set that they are gaining people are talking about their hobbies this is what social media platforms was originally for not of a particular category and the irony is yes I do teach in AI but that's the problem the person who is teaching in AI person who is actively working in AI. Not every time we have to uh kind of look into the same topic whenever we open social media platform. Sometimes it is just personal.
We just want to see people growth. We just want to see how people are doing.
How can we learn? What can we learn from them? What can we learn from their skill set? What can we learn from the path that they are following? So this is the overall problem statement that not every time we want to see the post regarding AI and layoff and can we create a system that can filter those post and how to create that backend system that can understand what post is talking about and categorize them into multiple categories. So that is the problem statement we are going to solve today.
All right. So let's get started. I'm calling this project as topic guard because this is nothing but it is going to guard a particular topic that we are creating a backend system and before talking about anything like what is the problem statement what is the data set that we are going to use what is the flow that we're going to follow what is the model we going to select the evolution before talking about any of that we need to talk about the prerequisite because if someone does not know about this prerequisite they can watch those and can follow this project later the First prerequisite is they should know Python. So if you're familiar with Python, you can watch this particular lecture. Second is you should be a little bit familiar uh with the framework scikitlearn or the library.
Okay. Why? Because I will be using train test split from scikitlearn because that is easy to follow. So I'll be using scikit learn.
All right.
And next is they should be a little bit familiar with the library transformers transformers and hugging face. Well, we are not going to go in huggings website but I will be showing you that how I got this particular model which we are going to pre-train. Okay. So, Python, scikitlearn, transformers, hugging face and little bit if you know numpy. Okay.
Yes, one important thing is we should know torch. So I'll be using torch as a framework and we'll be using this transformer that we have here. We are going to load that model. We are going to create our optimizers. We are going to train it manually. Okay. In our notebook. So we are required to know Python scikitlearn transformers hugging face numpy as well as torch. And if I talk about the interface we'll be using VS code. I will not be using any fancy library management. I will be using only V and V. Okay. that is Python native. So if you have Python in your system, you have Venv installed and to train the model, we are required a GPU. If you don't have GPU, don't worry. I will be sharing the Google drive link or I will try to update the same in the GitHub. So our model is already present. If you have the model, you can run via CPU. But if you don't have the model, then you cannot train this on CPU. We are required to have a GPU. I'm using GPU Nvidia RTX. So if you don't have that, you can use you can try Google Collab as well to train your model. So I'm using VS Code. I will be using Python virtual environment to create my virtual environment. And then I require a GPU to train the model. If you don't have GPU, as I mentioned, you can use CPU to just inference the model that is provided.
And what else? Okay, next thing is we are required to know how to work with notebooks. Okay, Jupyter notebooks but I will be using notebooks inside the VS code that's why I'm calling it as notebooks. So if you're not familiar with how to work with notebooks then I will suggest to watch any tutorial on notebooks any tutorial how to install VS code how to create a virtual environment etc in order to learn or in order to use all of this. Okay so these are the prerequisite. So let me write this is nothing but prerequisite two. All right, that is about the interface and the one is about the libraries and the concepts that we should be familiar with. All right.
Okay. So, what are we going to build today? Very very important. So, what we will build? So, we are going to use first notebooks.
The main agenda of this particular video or this particular lecture is we are not going to create any end toend project.
We are not going to do any deployment.
So let me write what not to expect here.
So I have not written this part here.
But let me copy this and write what not to expect. Okay?
Because sometimes solving a problem is more fun than creating the full end to end that you guys can do it later. So what not to expect.
Okay.
That there will be no deployment. Okay.
So let me write no deployment.
What else? We are not going to create any APIs and for the simplicity I'm not doing modular coding because training a transformer itself is a hectic job and if you end up creating the full modular coding creating API then this will not be this short of a video. Okay. So we are not going to do we are not going to follow any object oriented programming that's why we are going to have notebooks and we are going to create two different notebooks. One is we are going to create a notebook where we are going to train the model, save the model, perform the data set, split all the operations that is required for the training as well as we are going to have a model for evaluation. Once the model is trained, we're going to load the data set and then do the evaluation in this testing set. So we are going to have two different notebooks that we're going to work and then what is the artifact that we are going to get from this? We are going to have a model.pt PT file that is nothing but our actual model along with the configurations of it. Okay, plus the configs. But trust me, this is going to be a very fun video because we are going to train our own transformers, a encoder based model along with the data set that is very unique to this particular problem statement and then we are going to follow the full end to end of training as well as testing. This is what we are going to build. We are not expected to have any deployment in this video. no API creation and no object-oriented programming. Although I already have a stream application ready but that is not a part of this video.
Okay. So this is what we are going to build. Next thing is now let's talk about the problem statement.
Okay. So if you are able to see the problem statement here that readers are feeling increasingly overwhelmed by the constant flood of minuteby-minut AI advancements and distress news about tech layoff. I guess all these statements sounds little harsh here because of the because I rewrote all of this using AI. When people open a blog or news site, they often just want to discover interesting projects or learn what others are building without being forced to or doom scroll through stressful topics. Right? As I mentioned that even if I teach AI, I'm I'm the mentor who teaches ultimate data science here. But sometimes it is also stressful for me that lot of news regarding AI, right? And not every time we want to have that. However, current platforms does not have any flexibility to filter out all the specific intents. We call that intent. Intent means nothing but the classify the classification of those the topic of those leaving them stuck with feeds that cause fatigue rather than inspiration. Right? So previously before all of this, LinkedIn was inspirational. Right? We are able to see how people are growing, people are working and what decisions they are taking. But now all of them are kind of hidden behind this AI doom right now.
Okay, that is the problem statement. So what are we going to do? This is the proposed solution. To solve this, we can shift the filtering process to the moment content is published. Okay, so what we are kind of proposing is that yes, we are creating a system and because I don't work in any of these social media platforms, right? I don't work in LinkedIn, I don't work in Reddit. So what we just proposing we are proposing a system where whenever if they are using our system or they are creating a system like this what can they do and how they can build a model like this. So whenever user will click on publish right at the same time that data should go to an API API will be handling our model doing the inference and then predict whether this particular post talks about AI or it talks about layoff and the details of what different classes I have here. I will talk about that. That is the proposed solution. So for that what we require the model we are going to create and test an intelligent lightweight model.
Lightweight model is very important. I will talk about that why. So we are going to create a intelligent as well as a lightweight model that understands the context of an article and can accurately predict whether the post is related to AI layoff both or neither. Yes. So these are all the four classes that I will be talking about that. So let me pick a highlighter and then create it this way.
So we are going to have AI. Okay. That seems too dark. Okay. So we are going to have AI layoff these are the two classes plus we are going to have others that is both and neither and to have all these four classes we have to do some architectural changes in the training itself that we are going to see. Okay step two once we have defined the model and what are the different classes we that is the engine what you can do later once you have the score model as the engine you can deploy this as a realtime engine during the content creation phase. So step one is what we are going to cover. Create a model create and test the model whether it works or not. But once this part is done you guys can create a API from that you guys can create a stream application of your own and then try to simulate the part. Once we have this engine coming from step one we can deploy this prototype as a realtime engine during the content creation phase. So whenever a new blog, new space or article is created, it passes through the model exactly once only once and then we have those classification. Step three is this is again all the step two. So let me write all the steps that we are having all the step two, step three as well as step four is a part of uh future enhancement that you guys can take on and then create or build this project further.
The step three is once the model created that intent or the classification we can save that into the database. So the model predictions are permanently saved as simple flags right model predictions are permanently saved as simple flag and we are going to have only two flag is AI is layoff and now here I'm talking about two layoff some of you might be asking that but before you talked about four different classes why do we have two flags here I will talk about those little in a little bit on that post row in the database so in the database on that post row maybe from the index we can save this flag and once the flag is there on the UI part of LinkedIn or on thread. They can give us a toggle or can a filter that will give us a better user experience. So the platform then uses this database flag to offer reader simpler on or off. So if I don't want to see AI then I can just turn on is AI. If I don't want to see is layoff then I can just turn off this layoff. And if I don't want both I can turn off both is AI and is layoff. Then I will not see those post. So yes, that is uh uh what we are going to build. Okay. So this approach empowers users to customize their feeds instantly while keeping the platform fast and avoiding the high cost of running AI checks every time a page is loaded and okay so this is more about what is the proposed solution and we are only going to build step one which will take us little bit of time. Okay, this problem statement is interesting and fun because this is not something that we solve and we see daily, right? This is something that a user is facing that particular problem. User means all of us. So this is more like a UX problem.
But that problem cannot be solved by a simple programming. Developers, front-end developers cannot just pick JavaScript and then just try to solve this. No, this cannot. This requires AI knowledge and that we have and we are going to use that knowledge to create this model. Okay. So this is the proposed solution and I hope we all are clear with what is the problem statement. a fun problem statement and what we are going to solve how we are going to solve I will talk about that okay the first thing is let's talk about the direction okay that we are going to have this post this post is I have literally copied this from my LinkedIn post my recent LinkedIn post there I was talking about that neural network is trying to u run the world right neural network is running the world but my smart watch that I have when I kept it on my PC tower it was showing it that it had a fantastic night, fantastic sleep, right? So that was kind of a little funny. So I took that and then I will be passing that post to this new model to this modern bird based. This is the model that we're going to select. I will talk about that later. That is nothing but a encoder based model and that is going to classify that post whether this post is AI or that post is layoff. Okay.
So this is the direction that we're going to do and we are going to spend majority of our time and not here. Okay, this looks too thick here. So okay, so majority of the time we are going to spend here. We are going to load the data set which I have already created and here we are going to provide the evaluation. So if I talk about the different sections here, this is nothing but data set. This is nothing but training and once the training is done we need to understand whether it performs good or not. So here we are going to spend our time that is evaluation. It will not take time but evaluation is what which will give us the confidence that whether this model is even working or not. Okay. So the three different parts and we are going to cover data set and training in one notebook and evaluation in another notebook. So this is the direction. Now the next thing is data set. The first thing is data set. Training will only happen if we have the data set. But do we have this data set available over internet? Do we have a data set that talks about whether the post is AI or layoff? Well, we don't have any such data set. So that's why I had to create the data set which is already available to you guys. You don't have to waste your token. You don't have to spend anything. The data set will be provided in the GitHub URL. Okay. So data set is here. So what we are going to do? we are going to create a data set which I will be talking about how did I create the data set but let's clarify one thing that what classes we are expected because if that is we have we are not able to figure that part out then we cannot create a data set right so from the problem statement we need to define exactly what kind of data we have to create and to create those data set we need to define what classes we are creating data set for okay so here you are able to see that yes at the end I want my model to predict whether something is AI or something is layoff but can we do more if I talk about a post a natural scenario a real scenario a post can be of AI post can be of layoff a post can be of both a person talking about layoff because of AI but if I only have two classes I cannot flag them if I try to use softmax in softmax I can only get one particular class right I can I cannot probability of multiple things are higher. So what can I do? I will talk about those structural changes. But what are the different classes we require? Well, for is AI, what we require? We require that we will be predicting one in is AI. We'll be predicting is layoff zero. And when I talk about this, I will be naming this class as AI. Next is when I talk about layoff, when I talk about class of layoff, it means my layoff should be one, my AI should be zero. This is standard. This is pretty straightforward. Now, what about both?
When they talk about both, it means AI as well as layoff. It means I require one in both of them. But when the post is neither neither AI nor layoff, it means the post belongs to a general category. Okay, post belongs to general category which means both of them are zero. So this is what we are going to do. Okay. So we have decided now that these are the four classes and we have to create data set for four classes. But if we have not decided this part then we cannot move to the data set. What we are going to train the model? If we train the model only on AI. If we train the model only on layoff then our model will not be perfect. Our model will only classify AI or layoff. But what about both? So this is the decision part. This is something that we have to decide even before creating the data set. Okay. So now that this part is decided we have four different classes. Now we have to generate data set for AI for layoff for both and for general. How we are going to how we are going to generate the data set is what I will talk about in the next section. All right. Now let's talk about the data set. You are able to see here that the distillation right distillation is used to produce label data. So we are using a technique called as distillation which is nothing but to create a synthetic data set. This is the technique that we use to produce labeled data. What actually is distillation?
This is nothing but a very simple thing is distillation is a pattern. So let me have this highlighter so you all can see my mouse a bit. So distillation is the pattern of using a big slow expensive model. This is very important. We cannot use a simple model because simple model will not have that much of reasoning capability and that understanding of the real world data. So we require a big expensive model and when the model is big and expensive it by defaults kind of goes into the slow category. So distlation is a pattern of using that model usually an LLM to produce label data and we use that label data whatever we are creating from the distillation to train a small fast and cheap model. It might not be a cheap model but we usually use distillation because we don't want to create a model with 50 billion parameters. Okay, that is not our goal. We are not here to create a 50 billion uh parameter model because we cannot train it in our GPU. But we want to use that knowledge and create a very specialized model that is only good in one or two task. We are not here to create a model that is good in all task.
We are not here to generate or create a generalized model but a very specialized model which is nothing but a fast and cheap model. Okay. And why are we using a cheap model? I told you before as well when when we were talking about the proposed solution that we are creating a intelligent as well as lightweight model. Well, if I talk about the problem statement uh we can solve this with the help of API. We can solve this with the help of prompt engineering as well. But why we are not doing it? So let me talk about that here. Okay. So let's go here.
So if I talk about any post, if I talk about any post, we guess we can pass that post to a prompt.
We can pass that post to a prompt plus a very good model. Okay. plus a very good model that can classify a particular post whether that belongs to AI or whether that belongs to layoff or that belongs to both and the last one is neither right none of them. Okay. Yes, we can create we can create a solution exist from the prompt engineering plus a great model or good model that can classify it. But what is the problem? Well uh if I talk about the social platforms users daily they might be getting millions of post daily right and people are interacting on that. So it's a huge amount of post that they have to pass through this particular this particular flow and what is wrong with this flow? There is nothing wrong with this flow but this flow is expensive post. If the token size of the post is higher then the model will be uh using a high cost to give all of this prediction and the we have to write the prompt we have to pass that to group good model the API charge is also there.
So it is just lot of charge that we are getting in this particular path. So what we can do we can create a lightweight model. We can create a lightmade model that can run even on CPU with lightning speed. And if you can create such model then the API cost then the API cost is kind of negligible. Why? because it is running on CPU it can give you answers within milliseconds 10 millconds 200 millisecond not more than that and you can get all this information and even if the even if the post is changing the model can still predict all these four classes we don't have to change the model we don't have to change the prompt what happens in this first path is that sometimes the model gets deprecated we cannot use that and once we go with the new model then the prompt needs to be changed because the structure might not be following the same thing and we might not get the result. So there is lot of inconsistency here plus a constant cost we'll be getting in this first part but this is a one-time investment. We can generate the data using a good model train a lightweight model which can predict all of those and we don't have to go in that path. So that's why we are considering training the model rather than using a prompt engineering to classify it because over time the lightweight model will have more advantage. We'll be saving lot of cost uh with if I compare that with the first one that is prompt engineering plus using the APIs of large models like highQ set Gemini GPT etc. Okay we have chosen this path. So the main thing that I wanted to talk here is why this one is better. Why I was talking about a lightweight model. Okay. Lightweight model. Where else?
Yes. So lightweight model is also here.
A cheap model.
Okay. So two different highlights I have created for this one. Okay. So once the model is trained the small model can replace the big one for that specific task when we do the distillation. Okay we were talking about distillation. So once the model is trained the small model can replace the big one but for a specific task as I mentioned that the uh the large model is specialized for all the cases that is a general model but we want to only solve our problem statement for post whether is AI or is layoff.
What is the economic advantage? You pay the LLM cost fund. This is what I was talking about just now. You pay the LLM cost once that is to generate the data during the data generation. From then on, you pay only the small models inferenced cost which for a fine-tune encoder on a CPU is effectively zero. It is going to be negligible. Yes, there will be a cost but it will not be zero but effectively uh zero. Okay, so the data set that I have generated I have distilled. Okay. And we have around 1,352 examples uh from the distillation okay in training set and we have around 200 examples in test data okay all this is present in the GitHub link provided and you will be able to see in the same location that is in the root location inside data inside train you will find train.json JSON and the next one is you will be able to find in test. This is the test set which we are going to use for the evaluation. So this is test C.json. If you are not familiar with JSON L if you're only familiar with JSON not familiar with JSON L. JSONL means each line contains a JSON a individual JSON that is JSON line. Okay. And if you're familiar with only JSON, we only have one big of a JSON and inside that we have all the information but that is little harder to keep. JSON L is uh simpler. Uh we have only one JSON per line. Okay, that's what JSON L looks like. That's all. Nothing fancy just a simple thing. Okay, so how I am creating the data set. So this is the overall levelwise information about how I created this information. how I created this overall 1500 examples from the distillation. Okay. So let's talk about that.
Okay. So we will start from the bottom section and then slowly we will move up to understand the depth of the synthetic data. So here we are talking about synthetic data generation. All right.
And uh I'm creating data for four different categories as mentioned earlier. AI layoff both as well as general. So for each of those we are going to apply all the things. For each we are going to apply all the things.
For each we are going to apply all. And for the last one also we are going to apply all. So this is how the flow looks like. So now let's suppose for AI what we are creating we are creating data set for AI and for each category I'm creating 30% of edge case. What do we mean by edge case? Well edge case is nothing but a special scenario or a complex scenario. As I mentioned 30% edge case means a deliberate boundary case complex as well as trick examples I'm calling it as edge case others are the rest 70% is simpler examples the 30% is the edge case think of it as scenario like uh in the example we are mentioning chat GPT while writing about a hike it means I'm not talking about AI I'm not talking about layoff but in my big statement in my large statement I have just mentioned chat GPT once and that does not make it as a AI post. Okay, whenever I talk about AI, whenever I think about a AI post, it means I'm talking about that I've used AI, I've used AI here, etc. But if I use that I have utilized AI uh to for my hike, okay, or to write my leave application.
It means I'm talking about leave application. The intent is different.
The intent is not that I'm using AI. The intent is I'm writing this for my leave application. Okay, the same thing. So 30% edge case on all the scenarios I have written. Okay, now let's talk about the scenario one. For AI, 30% edge case is uh clear. Now for AI, we will be following four different classes. Okay, inside it register, tone, format and length. These are the four things for each row I am selecting. So when I talk about register, register is nothing but what is the writing style or the dialect of that particular data generation. What is the tone we should generate? Okay, the emotional register. What is the format for that synthetic data surface form of the post? What is the length of that particular post? So if I talk about register, I will talk about whether that person or that data that we're generating the post that we are generating belongs to a native, belongs to academic, belongs to journalistic, genz casual. So this defines kind of a style who is writing it right. So that is the register. Tone is what is a tone?
Someone is writing a bitter post a neutral analytical optimistic motivational and there are others. So I've mentioned this dot dot dot which means others. So I cannot write all the things here but I have utilized the tone which is beta neutral analytic motivational etc which defines the post tone. What is the tone of the post? Then the format whether it looks like a X thread or news or LinkedIn abstract because in LinkedIn we usually have some smileys we usually have some emojis in Reddit we usually have some uh larger information right the post are large in size abstract essay others this is the format and then the length length is micro short medium long I only have decided four length because sometimes the post is only two three lines sometimes it is a paragraph sometimes it is like a chapter a book chapter right so that that will come under long. So for each post this is how it will flow like I want to create a synthetic data for AI I will create 30% edge case and then for each example I will randomly select register I will randomly select tone randomly select format randomly select the length and then using the prompt engineering I will be passing all these parameters to generate a single example. So every time it will be random every time because we have a large selection it will generate lots of data.
So that's how we will generate for AI.
Similar thing we will do for layoff.
Same exact thing but the topic will be layoff. Same exact thing for both. It means it should have AI as well as layoff. General means that port should talk about anything other than AI or layoff. Okay. So this is the flow I am choosing. This is how I am creating the synthetic data. Uh this is just a overview of that. Okay. Now if I talk about the generational uh the generation pipeline, okay, how I created this train data pipeline, it has to be different for train and test. I will talk about that why. But now let's look at this uh particular flow.
All right. If I talk about how I generated the training data, this is not a training um model. This is about how I created the train data set. So first what I decided before passing or before selecting any of these parameters I first decide what is my label. Okay. So target the label first. First we will select what label we want to generate.
Then ask the generator to generate the example. So we will decide whether I want to create the post for isi is layoff is both or uh general. Once I have decided the label then I will pass that label to the generator. Then I will ask that you should generate for this particular category. And once this is decided then I will input that into the prompt. Okay. And for the prompt I'm using Claude HQ model HQ 4.5 plus I have written some base prompt where in that base prompt I will be ingesting I will be adding these parameters class register tone format length and I have few more things like persona geography subtopic which I have not written here but these are the four major ones that decide the data generation that decides the nature of the data but I have four I have two more class register few more okay here I have not mentioned the all the things. So what I do I have a base prompt I write let me choose again I have a base prompt I have written some prompt here and I have written some placeholders that we are generating data for such label that is a step number one in this train pipeline that is here I will first decide the label that is a placeholder then I will decide that I'm generating for this register this stone this class this format etc and then I pass this prompt to a cloud IQ model using the API and then that generates the data in the format I require. Okay.
So here I have also mentioned the structured response so that I can every time I will get the correct structure.
This is how I am creating it. I will not be teaching you the part where how I created the data set because it took 2 three days to generate this whole data set and we don't have that much of time.
Okay. And this particular process I repeated multiple times actually I think I don't remember the exact name but I created 240 different parallel um task for that to create the model and it took like 3 4 hours but the other part took days okay because creating this base prompt uh making it perfect took time this particular process I created 1300 examples but I am not stopping here okay because there is still uncertaintity because this is I'm using AI I'm using AI to generate synthetic data. So what I did once I generated all the 1300 examples approximate 1300 examples then I created another prompt that I'm calling it as validator prompt. The task of validator prompt is when I have this new row of information and I actually have the ground truth because this is what I'm passing to my prompt first. So I have the actual label I have what the model generated the claude hiq generated I am passing to another prompt called a valid data prompt okay and this one is a different better strong model so what I'm doing I'm passing that this particular data belongs to this label and this is what I have generated can you validate if the generated text is matching with the actual input if it is matching if match then fine if it does not match if it does not match I'm already doing the reabeling okay so let me write it more clearly so what I'm doing I have the actual ground truth and then I have generated data okay what I'm doing I'm passing it to a prompt that is nothing but generated data a ground truth plus generated data and then I'm asking whether they are same or not if not same what is the label if same what is the label. So I am reabeling every generated row in a new row. So previously I had for this text what is the actual data what is the generated label and what is this new validated label are. So I have three different labels here which I'm using it to check.
Okay. So once uh the validator prompt generated new label then I'm checking if there is a mismatch. It means if what claude is saying this new validator prompt is saying is it matching with what the model is saying. If both are not matching both are means what we generated versus what the validator prompt is saying then I am passing it to a human review. Okay. So this took lot of time. This is a different processing but just to give an idea that how the data set I have generated which we are going to use in this particular project.
Okay. So if there is a mismatch a human will review that part. If there is a not if there is no mismatch then it is already correct. We don't have to do anything. This is how I generated training data. So target the label first. Let's again review that. Target the label first. Then ask the generator to generate the example. First we will select the label. Use a model a base prompt and then pass all these parameter diversity parameters which will generate the example. We will do it in iteration to generate lot of lot of examples. Once that is done, now it's time to validate.
For validation, we are using a different prompt, not the brace prompt. Okay, we are using a different prompt this time and asking the prompt that I have this input and this is what the model generated whether it is correct or not.
If correct, what is the label? If not correct, what should be the label? If there is a mismatch, then a human will review it or delete it. We can review or delete it. Okay. If you don't want human to review it then if there is any mismatch then you can drop it. If you want to review it you want to save your data then please review that particular part. Okay. If you follow this you don't have to worry about all this. I already have the data set. This is about training. In testing what I did the same pipeline is there. But if I use the same base prompt as the training then the model generation will be biased. My testing will be biased towards my generated training data. I don't want that. I want my testing data to be different than what model has been trained on. So that's why what I did I use the same exact thing but only one thing changed here is I have used a different model to generate my test data and use a new base prompt. As you're able to see here I have written new base prompt and in the training I have written base prompt. So testing uses a different prompt plus a different model to generate the data. A different model will give you a different data. A new base prompt will also give you new data.
So the data that we are generating 200 examples we are generating in the test data is a new set that is very different from the training. Okay, it will be different distribution and we have generated 200 because I don't want to spend more on the APIs. Once that is done, I'm using the same thing that is validator prompt with the same cloud sonet model because the task of this one is to validate not to generate anything and it will reabel and if you don't like and if there is a mismatch the human will review it or drop it and that's how I have created the test data. All right.
So so far what we have seen we have seen the problem statement what is the actual problem statement what is the proposed solution and what is the direction what we are trying to do here and I've also mentioned what is the future step that is step two step three step four which you guys can follow later once this whole uh video is completed and considering the direction we have decided that what things we are solving in terms of uh we now we need to write out the exact classes because if we don't have the exact classes we cannot generate the exact or precise data set.
Once we have defined what we are trying to solve and how what things we require technically. Okay. Then we are generating we are generating the data set based on that and we are using distillation that is nothing but a big slow expensive model to create label data. Once that is done the next step which we are going to do is nothing but train that fast small and cheap model.
Once trained we can use it to replace the big model. That's why we are not going towards the API subscription model that is pass the prompt to the API okay the cloud Gemini and GPT models API and then turn our and then do our task we are going to use our own model okay that is nothing but will give us the economic advantage so we are going to generate we have generated already okay I have generated already 1352 examples and 200 examples in the test data set and I've also given you the examples of how I have generated this whole In the next part, what we're going to do, we are going to take this data set, load the data set, create a notebook, and then start the model training there.
I have already prepared a few things beforehand. Things like installation of virtual environment and writing all the requirements..txt. So you guys can follow it. All the instructions are present inside the instructions..txt, which will be available under the mentioned GitHub. Okay. I've also mentioned the GitHub URLs present in important urls.txt. Here I've mentioned where you will be able to find all the code and where I'm using uh how did I got the code to install the pytorch with the CUDA version and where is the model present what is the actual source of the model that we are going to pre-train okay so this is instruction.txt txt. The step one is run this particular command to create the virtual environment which is present here. Step two is you can activate this particular environment.
This is the code which you have to follow in Windows when you're using git bash. If you're using a normal cmd, you have to use this particular command. But make sure that you are already present inside the current working directory that is inside your project folder topic guard. So I'm inside topic guard. I can activate this particular environment. As you can see this is already activated.
I'm using cmd and you can also use git bash but if you use git bash you need to follow the source and then the vnv scripts activate command. This is on windows. If you are in Mac OS or Linux you can follow source.v/binactivate.
In Windows you don't have this bin folder. This is present in Mac OS and Linux. In Windows you have scripts folder. Okay, that is the only difference and these are already the prerequisite that you should know how to create virtual environments, how to activate etc. And the specific version of Python that is installed in my system is Python 3.14.3.
You can try with 3.12, you can try with 3.8 etc. But make sure that you are installing the correct dependencies.
Okay. And one thing that uh before installing this version of PyTorch, I will suggest you guys to try Nvidia Smi.
So let me close this. I don't want this bash. And if you write Nvidia Smi, let me clear the screen.
Nvidia Smi, you will see what CUDA version your GPU supports. My current CUDA version is 13.1. It means even if I install 12.6, it is supported in my CUDA version. But make sure that your GPU accepts or GPU runs on the CUDA version that you are installing via PyTorch. And if you don't know which CUDA version, you guys can check using Nvidia - SMI and then try to Google it out and see which version of PyTorch CUDA that your GPU accepts. This is a different GPU version that you're able to see. Lot of people confuses this particular part.
This is nothing but your GPU CUDA version which is different from your PyTorch binary CUDA version but they should be supported. Okay, if you want to know more about it, you guys can follow the important URLs here present here. go to this URL and then you will be able to see let me open it and then you will be able to see what are the instructions present here okay you can see the install PyTorch you can select if you're working with cond is now not supported but if you check the previous version of PyTorch you will also be able to find the instructions for right now it is for pip lip torch source this is the package manager for Linux Mac windows you can select it select the stable version this is what I've installed but I'm installing using the Python and you're able to see CUDA 12.6 is the latest one which we're able to see here. Okay, 13.0 13.2 is also there but I am using 12.6 because it is already pretty stable. Okay, so I copied this command and ran in my VS code. The same thing is present inside my instruction.txt. Okay, so when you run this command only after activating this you will see that the installation will take some time because the file size is 2.6 GB. That's why I have prepared all of this beforehand. Step four is only once the PyTorch GPU version is installed then try to install the requirements txt. Okay, because if in requirement txt if you try to mention PyTorch directly without the GPU then it will try to install the PyTorch the CPU version that's why install the PyTorch first and then write other things present in requirements. TXT that's what I've written. So once the PyTorch is installed then I'm using pip install-r requirements.txt.
Once that is done to check if the CUDA that you have installed via PyTorch is able to communicate with your actual GPU or not. I have written some code here that is present inside check CUDA. py which is present here checka.py. Here I'm doing a very basic thing. I'm trying to import torch. Then I'm checking if torch.coda is available only when your cod is compatible. the version that you installed is compatible with your GPU, it will be true. If this is true, then we are going to print this particular print statement and then I will get my device name. If it is able to retrieve my device name, it means it is working correctly. It also works if this is true then it will work. But this is just one additional step. I am already inside my virtual environment. I'm already inside my correct working directory. What I can do? I can follow the instructions. Just run this command python checker. py and then it should return true and it should return that my device name is this okay I'm not writing true but cuda device is available below is the device information this is my GPU that I have Nvidia GeForce RTX 4070 super okay so it means my CUDA installation is working correctly what is next in my instructions nothing it means all the requirements of this particular project is satisfied so first thing is create a virtual environment, activate it, install torch with CUDA. Else if you don't have CUDA, try to do this particular series. Try to do this particular video. If you don't have PyTorch installed, try to do this particular lecture instructions and the Google Collab. Once this is done, install all the requirements except the PyTorch because that is already installed. Then run this file to check if CUDA is available or not. And once everything is done, it means our system is ready for the notebook creation.
Okay. All right. This is the virtual environment. And what I will be doing, I will be creating all the notebooks inside my experiments folder. Okay.
Experiments folder. Why I'm following this experiments? Because uh whenever we create object-oriented programming, whenever we create modular coding, we create in the root folder. We don't create a notebook inside the root folder because it will look messy. All of this is a part of experiment. So I'm keeping the notebook here. So once this is done, you guys can take this repository all the code and can create the modular coding in the root folder. So it is already clean there. Okay. So first thing is let me also draw this things in here because we have the functionality.
So this is nothing but excal draw. You don't have to worry about it. So what we have done we have install system dependencies or system requirements.
Okay, this part is completed for us.
Next thing what we will do, let me save this. Next thing what we will do, create a notebook. I'm going to name this notebook as 01 train.
Okay. And here we are going to start it.
First thing that we are going to do is import. So let me write imports. We are going to do few imports. I already have all the imports ready. So let me do copy and paste for this part because we cannot wait. We don't have all the time to write all the things. We want to be faster in this particular part. So let me select Python. Okay. So these are all the imports. I'm using JSON because we'll be loading the JSON L random. I will talk about why we are using random.
We have time to check the inference time to check the training time. We have the counter to count things. Path li to get to the path where we can load the data set where the data set is present. Numpy torch torch.n nn dot to nn as nnn. This is the base class of neural network which enables us to write the custom architecture as well. Then we are going to import and we are using the optimizer Adamw. I have experimented with this and this is giving me good results. Then I'm using data set and data loader. That's why I'm saying that you the prerequisite is pyarch. If you don't know what is data set and data loader, you will have little hard time. But I will try to explain what data set is and how we use the data set inside the data sub data set loader and what data set loader or data loader enables us to do. Then there are some metrics like precision, recall, F1 score and from there from the F1 we are going to create a micro F1 which I will talk about later the micro F1.
Okay. From the sk model selection I am importing train test split that is one of the requirement. skarn. We should be familiar with how to import things from the skarn. We should be familiar with how to do train test split and what is the format it returns for the split data. And then finally, I'm using transformers. I'm using automodel. I'm using auto tokenizer. This will load the model. This will load the tokenizer and this is nothing but a linear schedule with warm up. Okay, this is going to warm up our GPU, do the training and then linearly decrease the learning rate. Okay, so it is going to handle all of that. And then this is the same exact code that I have written inside the check coda py. This is the same thing and we don't require this particular part. So let me close this. We've already installed all the things. Now we are left with the notebook. Let me run it and see whether it works or not. Yes, this will also show you this particular part that it requires IPY kernel package because we are using notebook and our notebook needs to communicate with the Python kernel from this notebook. So it will take some time.
All right. So it is installed. It took around 2 to 3 minutes. And finally let's see if it is working or not. It is trying to import it. If everything is successful sometimes in VS code when we select a kernel it does not run for the first time. You have to close the VS code. Open it again. Then only you can run the cell. I know that is a weird bug but yeah this is working for me. I don't have to restart my VS code. All right. I guess it is able to see we are able to see that the CUDA is available that is true and the same information that we were able to see in Py check CUDA. py okay second step very very important step a lot of time we don't really focus on this step but it is important for me because we are watching a tutorial and for tutorial uh important thing is we all should be able to set or see the same let me first write it set all the seeds we all should be able to reproduce the same output and for that it is very important that we don't just focus on setting the simple seed. So what I'm doing I will be setting multi-level seed. Okay, I will not be only setting the seed for pytor or the python. So let me write random dot seed is seed.
This is python seed. I also need to set the seed for numpy. So np.random random dot seed seed I also need to set the seed for torch because all of these are different library they all select they all have different randomness if I only set the seed for python and not others and expect that others should give us the same behavior then we are wrong we should set all the seeds whatever seeds that we know of so we can get a better or we can get a reproducible output okay that is torch domanuel seed is it and if torch.ca dot is available because the GPU seed is also different if torch.ca CUDA dot is available then we should also set the seed present inside that that is torch dot cuda dot manual seed all okay that is also seed so this is random seed set to seed nprandom seed uh seed 7 is set to domanual seed this is also to set and if CUDA is available then we will also set this seed. Okay. And we are done. So I will write print seeded with seeded with and then after comma we can write the seed. Okay. So all are set.
Okay. This is very important. Once this is done then we need to write the configuration that we are going to use for the model training. Okay. So let me write those all the configurations in a single place. So I'll be writing third step is configuration configuration.
So here we are going to set all the hyperparameters that we require. So what I'm going to use I'm going to use the model name model name and if you want to know more about the model you can check the important URLs. This is where I got this model bird base and I think it is here. Yes, you're able to see this is modern but base. This is a small model 0.1 billion parameters. It has a tensor type of float 32 and you can see all information about the model. Okay, you can install the transformers which we have already done and then you can use this model. So this is the model ID. So I'm going to copy and paste the same exact thing because that is going to be my model name. Okay, model name. So this is answer modern bird base. So we are using it.
You can use any fallback mechanism. You can use robot base. But there is a reason why we are using modern base. So let me talk about why we are selecting modern base and not Roberta or any other model. If I talk about Roberta or any other older bird models, the issue with them is if I talk about this one, this one was introduced at do we have a date.
Let me check for the date or we can just check the files and we will able to see when this was uploaded over one year ago. If I talk about the terms, let me enable my screen. If I talk about the terms like chat GPT, if I talk about terms like agenti, if I talk about terms like rag etc. All this latest model claude claw code etc were not available like 3 4 years ago they are recently introduced and they got the traction but if I chose a model that was pre-trained that was trained on the older corpus older corpus data they don't have that information even if you fine-tune them we require huge amount of data in order to tell them that cloud code gemini GPT all these models means AI and our particular problem statement is dependent on all those terms A model should know these terms beforehand. If I use a old model which does not know all of this then the classification will also fail. So we require a model that is trained on all different new things.
Okay. That's why we we are using modern but okay and modern but is trained on code files as well as all this new latest information on from the internet.
That's why this model is capable for uh all this new task and is suitable for our need. Okay, that's what we are doing. So let's get back to our notebook here. So this is the model name. I don't have to write the reason because I already told you the reason. Let me trim this a bit. This is model size. And this model has a max length uh which can fit a huge amount of data but I'm choosing a max length of only 512. Okay, this particular model length accepts around 8,8192. I'm not really uh I forgot what was the actual thing. But if you see the model length config.json let's see if it is present here. This model accepts somewhere around 8,000 uh max tokens.
Okay. But we are only utilizing a few part. We only utilizing 51.2. You can use the whole max length but I'm using this one so that it is easier for me to train. Okay. You can change this parameter. So let me write change this param if you want to process larger post. Okay 512 is enough for us.
Okay each length each token will be converted to 512 here. Okay that is a max length. I'm using batch size of 16.
If you have a bigger GPU, you can fit more data. But I'm using this and for the evaluation because evaluation does not have all the data and I don't have to do that much of processing. So I will be using a batch size of 32 in the evaluation. Okay, the learning rate if I say I will be starting with 2 E minus 5.
Okay, it's a small learning rate and number of epochs. You can change this particular part. But I'm going to train this for let's suppose for eight epochs.
Okay, each epoch in my system is taking around 3 3 and 1/2 minutes. So number of epoch equals to 8 will take like 20 minutes, 25 minutes etc. Okay, or it can be done sooner depending on how occupied my GPU is. warm-up ratio warm-up ratio I will talk about warm-up ratio later but warm-up ratio is 0.1 it means 10% of the total iteration okay 10% of my total iteration uh should be warm up okay total number of iteration not the epoch that will be my warm up that what I'm taking then weight decay weight decay that is 0.01 01. Okay. And uh next is dropout.
Dropout I'm selecting it as 0.1. This is only in training. We don't do dropout in testing. And this should be obvious for someone who already knows transformers.
And I'm using validation split. I will be using only 0.15% as the validation because I don't have huge amount of data. We still have enough to train the model. But I don't want to spend 30% just for validation.
Okay. And then the threshold okay the threshold I will be using threshold I will be using is 0.5 why what is this threshold this threshold I will talk about more but we are going to use BCE binary cross entropy with logit loss which is nothing but internally has sigmoid okay and the model that we are creating if I talk about our nodes the model that I am creating that we are creating here that is encoderbased model is going to output two different classes because we want both of them. We cannot use softmax. All right? Because softmax will be true for any of them. But it will not be true for both. If you require both of them, it means we require we need to tune both of them to output probabilities. That is for both cases as well. That's why we require threshold. So I'm writing threshold for both of them as 0.1 because we'll be using sigmoid internally and we are using binary cross entropy. So that requires this particular threshold. Okay. threshold is 0.5 and this is very important thing you might not have RTX 40 series or the MP GPUs but if you have those you can enable this as true okay if you don't have RTX 40 series or MP GPU or uh love list GPU don't enable this as true because it will not work it will go to CPU and then make the training more slower b is a data type which kind of enables us the same precision as FP uh 32. Okay. So this enables us faster training if your GPU supports it. My GPU supports it. That's why I'm using BF16.
All right. This is about the configuration configuration of my model training and I will be using few of them in my evaluation as well. So let me write this is model train and evolve event config. Okay.
Next configuration because we are in the configuration tab itself configuration section itself I want to write the path path to my data set as we able to see this is our train and test data set if I try to show you this train data set see let's see this okay as we can see that each of this line okay each of this one is let me move to the top because it has lots of things here and where is the cursor I can't even see the cursor okay each of this line you can see it's its own JSON If you try to see the end of it, you will find that the end is also closing as and is also closing which is making a single JSON complete. This is each line is a JSON that is making this file type as JSON L. And here I have ID, I have class. This data set belongs to AI. This data set belongs to layoff.
This data set belongs to both. This data set belongs to general. This is the same class that we have written in the nodes.
And what about the what about the one hot encoding of this one right? Well, this is I'm saying it as one hot encoding but it is basically what it is basically is AI and is layoff just a flag. So let's try to do this. Is AI is one is layoff is zero it means one means it is AI case. When is layoff is one it means it is layoff. Okay. When both of them is one it means it is both. when either of them is one it means it is a general case and here you can see all the configurations I have used to generate the data I've also noted down this is not used for training but this is only used when I was generating it so I've noted down all the information you can see register register is uh generalistic tone is neutral format you can see all this part one thing that as I mentioned here that I was generating 30% of them as edge cases. What about those? Those are also present here. You can see each of them have a new flag called as edge. So let's look at that.
You can see edge as false. So whenever you see edge as true, I'm also noting down what was the edge condition. So you can see whenever edge is true, whenever edge is set to true, we also have a new key and value that is what is the edge reason. So this was the edge reason of this one. All right, this is edge true.
This is edge reason. The same thing is also like that for test.json L.
So this is our test JSONL. Same exact thing. Okay. It also has let's see it also has edge cases. Okay. And we already know that how this part is created using a different pro using a different model as well. test C.json L.
So we need to pass we need to create a path here so we can connect to our data set. So what I'm going to do I'm going to use the path that we have imported and from the current working directory we are in the current working directory.
We are inside the experiments. So we have to go one folder back and then from the root go inside data go inside test to get the test. Go inside train to get the train. That's what we are going to do. So path dot current working directory I'm going to write it as here.
Okay just underscore. So it means that we are going to use it somewhere else.
This is not complete by its own. So if then I'm going to write here. Okay. So from here what we have to do we have to go to the data. All right. So go to data and then okay what I can do better because from here we don't have data. So we can go we can select here dot parent because this is already a part of object of path. We can go to its parent and parent will give us the parent that is nothing but topic guard. If you want to check I will show you how will it look like. So here and here parent if you print we will see topic guard as our root. Okay this is what we're going to get. So from the here parent we have to go inside data.
All right we have to go inside data and what we can do we can go inside test.
Okay, for the test data, but what we can do here is instead of writing test, we can only focus on train because as I mentioned, we are going to have a this as training notebook and we are going to have a different notebook as test. So we don't have to load test data set here at all. We're just going to train and we we already have a validation set here. We will create it.
The testing part is on the evaluation notebook. So let's focus only on train and from the train where we have to go from the train the train data set is present inside train dot jsonl okay and if this exist only if this exist then we will load the file okay then we will load where do we have our data so I will write here dot par okay so what I'm trying to do here I'm trying to create that only if this file exist we will consider that the repo is present. Okay, repo means the folder structure is present and once the folder structure is present then we will create the real JSONL path. Okay, JSONL path.
So JSONL path will start from repo/ all these things from the data train JSONL only if this is present. Okay, so repo/ data/train.path.
So let me print few things. Let me print things like if statement I will write this is our location for the rebu and here I will find the parent. This is where our parent is and then I will write train data.
This is where our train data is.
So train data is present inside train JSON. And we can also write does it exist. So we can write exist. We can directly check it from here and write train JSON.exist.
These are few of unnecessary things but sometimes it's good to have. Okay, exist. Yes, if it exist and then we can also write the model output directory.
Whenever we are going to save a model, we have to save it. So we can create from this path. Let me copy the whole thing. We want to save the model inside the repo but I want to save it create a new folder called as models and inside models I want to save it inside a folder called as notebook because I will be saving the model from the notebook not from any script. So this is nothing but model output directory model all capital because these are all config files. So this will help us understand that these are configurations. So model output directory I can also print what is our model output directory. So model output directory is nothing but let me write f string output to output set to and then the model output directory. So let's see how it looks like. Okay, something is wrong because I'm trying to check the bracket is missing here. not bracket. I don't even require bracket here. Yeah, now this is correct because this is closing the bracket and then exist needs to have this colon. Okay, Windows path has no exist. Okay, because it requires suh it will work. Okay, so this is the current repository. Does it look correct? Project use case others topic guard. Yes, it looks correct.
Train data is set to topic card inside train inside uh data inside train we have train dojsonl that is also correct exist does it exist yes it exist that is confirmed and then we are going to save the model by creating a new folder here and then called as models and inside models we'll be creating notebook and inside notebook we'll be saving the model that we are going to train all right so this completes our configuration then next thing is because we are going in a systematic way it might feel little flow but we can again write device because we also need to save the variable into device okay device so I'm going to write devices if we have it if we have it otherwise otherwise CPU all right and we can write the code for torch dot device and return cuda then we Right? If toss.cuda, okay, if toss.cuda dot is available.
Okay. Only when this is available then return cuda. So only when this statement is available tors.cuda dot is available then return cuda. Else return cp.
Okay. So this will be set into the variable called as device and we're using this variable across while training while evaluating etc. Okay. So, torch.cuda dot is available as CPU. So, we can print it. We can print what is the device? Print f the device that we are going to get from this one is let's see sorry not decide it is device.
All right. So, we are going to get device from here. Okay. Yes, device is CUDA. It means my doge is working. And if the device is CUDA then we can set few more things. We can check how much memory is free in my system and what is the device name etc. So I can check this. Okay. If a device memory is already running low it means something is using your GPU and if something is using your GPU it means you can kill the process before starting the model training. So we can check torch dot cuda dot check if memory is available or not.
it is present inside memory get info and this will return us two things that is how much memory is free out of how much total memory. Okay. So we can just print this particular part that let me write memory memory used and we can write this as free.
How much memory is free by total memory?
Total memory and we will get it from here that is total. So we will see this particular part and let's check. Okay.
Okay. So we are getting nothing. It means I'm I have written something wrong here. Oh, this is not a f statement. Now it will work. So I'm using this but this one does not look correct because it is in different format. So what we can do?
We can divide 3 by 1 e 9. Okay. 1 e to power of 9 colon.1 f i only require one precision. So this will give you answer in terms of GB. Okay. You can divide this divide this part as well and now you're getting 11.6 by 12.9. If you want to check the same I can show you my task manager. I can show you my GPU. You can see GPU is now that I tested it. It is showing 34 but it has okay 1.6 11.6 is free sorry this should be free memory not used okay free memory how much free memory I have out of total memory 11.6 Six is free because I'm utilizing around 1 GB of it that we have just now seen. Okay, this is about the device. Device part is done. Now let's go to the next part where we'll be talking about load the training data.
This is where we'll be using the data set class and data loader class from the PyTorch and we'll be using it to create the data or the loading part of it that is compatible with PyTorch. Okay. But first we have to load the data set.
Okay. So let me create a function to load the JSON L because it cannot work directly with JSON L. So I will be requiring a path. I'm creating a function where I'll be passing the path and we will get all the JSON data.
Currently this is rows is empty. Then what we will do with open I'll be using path encoding UTF8 standard encoding UTF8 as F creating a file for that and then we'll be using for line NF remove all the empty spaces remove all the spaces from left and right and then give me the line. Once I have the line, let's append the line. But only if line exist. If line exist, then we need to append that into the rows and rows.append, we need to append JSON.load of line because now it is still in JSON.
Okay, JSON do.loads of line. And finally, it will be completed. Okay, converted. Once this is done, then we need to return the full rows. Okay, so this completes our load JSON and we will using this function. So first this function is created and we'll be using this function load JSON and we will pass what? We will pass the path that we have created that is nothing but the training where do we have our configurations here. So training JSON L is the path. So we'll be passing it here and then see in the data set. This will give us all the rows that is in the list form and then we can just print few information about how much information we have inside this. So we can write loaded how much length loaded length of the rows loaded training rows.
Okay. So we are able to see okay it is it is printed nothing. Am I saying it right? Okay, sorry. This should be length of rows. Okay, so loaded 1352 training rows. And if you want to see the first data, we can just print out row of zero.
And because this is just a sample, I just want to create a copy of it. If you want to make any changes and this one, I will say it as sample. And we can just print the sample here.
Sample. And this is the sample. You can see ID. We are able to see this as our first row. Now it is in the format of dictionary. Okay. So ID AI class AI is AI is layoff. It means it belongs to class. Do we have that class? Yes, class is AI. Is this edge case? It is not a edge case. Text MR published a new open weight model in Paris on Tuesday Thursday claiming competitive reasoning etc etc etc. This is a short form. Okay.
Let's see what is the length. Yes, length is already short. Format is kind of news. Yes, it does sound like a news.
Okay. So, this is about the load the training data but we have just loaded not for PyTorch but only for Python.
Okay. Once this is done, now let's see the class distribution. So, our sixth part is class distribution.
Okay. And here we will see we will use the counter that we have just imported okay at the very start. So counter what we need to count we need to count the class from where for R in rows because this is a full data set that's the point of loading it so we can do more things with it. Okay for R in rows we want to count what we want to count the class it has a class this is what we want to count we want to count how much data belongs to each class and this will give us class counts.
class counts.
All right. And then what we can do, we can print this particular class counts.
Uh let's print it. So I want to print by class how much classes that we have here. And uh for this let me print by class.
Okay. By class for K in sorted. I just want in sorted uh smallest then to highest but I want to sort by the alphabet. So it will automatically do the alphabet based on the key. Okay, that is the class itself.
So sorted I want to sort by class counts and then I want to print an F string that how much value do we have. Okay, so I want to print K. K means the class itself. And then I want to print after a space I want to print what is a class count of those K. Okay. So we will get the class count. So class count of the K itself. Let's print it. So we are able to see by class AI 332 both 339 general 343. And if you want to say it in a indented fashion with equally spaced you can just write K smaller than equals to 10 uh colon smaller. It will give us a 10 padding in total as a space. Then it will show this count. Okay, this will look better. So this total means 10. So AI 332 both 339. So it is looking pretty balanced to me. Okay, what else you can do? You can also count what uh label belongs to it. So instead of AI, you can also see the label. So if you want to see the label instead of doing the class count, you can you can also do label count. Okay, label count is also straightforward. We have all the information here that is in the label is AI is layoff. So what I will be counting here in the counter let's count let's count let me remove all this you want to count both of them okay so I will be writing R is AI let me copy this R is AI R is layoff okay so now it will be counting these as well and just like we printed this part we can also also print we can also print the label count. Okay, label count will be same instead of this one but I will be just writing label count of K. And now it will show the same exact thing but something is wrong.
What have I done here? Count looks good for R in rows. This part also looks good. That is R is of AI R is of layoff.
Where do we have this error? Okay, we are having error in the print statement.
So we are trying to do the label counts.
We are trying to print it. But yes, this K is not a string. So we cannot apply this. We just need to write the thing as it is. Now it will work. Okay, let me write the last statement.
Let me write another print for now.
Okay, that is nothing but the separation and I will multiply it by 10. So we can see the separation. Okay, the same thing uh this one is 00 because we are not sorting it out. So that's why it looks little weird but 00 means general 01 means this is the layoff part. It means it is layoff. This is AI. This is both cases. Okay, you can match these numbers and this is how it looks like. Okay, now that we have our training data, it's time to split it. Before trying to do any processing, we need to have train and validation class. So let's try to have train and validation class. Let me write seven stratified split stratified train and well split. Okay, if you're someone who has worked with PyTorch and created deep learning models, then you know that creating a stratified split is very important. Okay, if you don't create a stratified split, what will happen? If you don't create a stratified split, then it means your class will be imbalanced. some because it is random in nature the separation part will be a little random then what will happen it might be possible that 70% of something will go into the training then only 30% is here for uh AI class but then for general class 50% can go to train 50% can go to test for layoff 70% can go to test 30% can go there so we don't have control and to have that control and to have that balanced data set we use stratified train/well wellsplit. All right. So let's go there. This is train/wellsplit.
And what we are going to do, we are going to have we already have this classes class counts here. But instead of counting this class, I just want to extract all the classes available. Okay?
Because this is nothing but my y column my target. So I'm first creating the target because stratified split work with target. We need to provide the whole length of target the whole data.
Okay? The whole series of target. This is where I have it. I will enclose this part in a list. So I will get the whole thing. You can see AI AI AI etc. So this is giving me that. Okay. And this one I will call it as classes. This will go to the stratify y is equals to. Okay. So stratify will take this. This is our target column. And what we will do? We will now use that train test split.
Train test split. And here we'll be passing rows. Our rows contains the full data. We need to split this rows based on stratify. So I will write the test size later but based on stratify.
Okay. We need to split this rows based on the classes based on my target column. All right. And then I will be using the random seed. Okay. Random state that we have already created at the very start. So this is the seed and the test side test size is something that we have also decided earlier that is well split okay that is 0.15 this is the validation split that we have and this completes our all the things and once this is done this is going to return us train then val rows okay once this is done let me print it so we can see it clearly so if I show you train rows you can see that train rows has one data okay including the classes so it is having everything okay this is the whole train rows now what I will do I will just print how many data do we have after split so let's write train what is the length of train length if this is fing then I can use this inside the bracket and I will use the length of train roots same thing I will do for validation.
This is nothing but validation and I have validation rows as well as uh this is not a machine learning problem statement. It does not have train X train Y because our class is already a part of the JSON. So we are splitting the whole JSON L with the help of stratify that is our classes. This is serving us as the Y even though we don't have a separate Y inside the data set.
Okay. So we are not able to see any Y because it is a part of JSONL but we are able to split it only because we have extracted the Y as the column. Okay. Now let's see how much data do we have.
Okay. So now we have 1149 in our train.
And we will use this validation script to check if a model is improving or not.
Okay. And if you want to check whether the class is balanced or not, what you can do? You can check from the train.
You can check from the validation whether it is balanced or not. You can you can print it. Okay. So it will be pretty balanced. But if you only check the validation, it will tell us the story that yes, all the classes are balanced. And we can check it with the help of print. Let's check validation class balance.
All right. And inside it, I'm going to write first I require the f string and then I will write using the dictionary.
Okay. Then I will use the counter because at the end I want output as my dictionary. But counter will be doing the actual counting. So counter and then what I want to count I want to count our class.
But R class of where? R class that is present in my validation row. So for R in well rows.
Okay something is wrong. We are missing this. So this is this looks correct. But what is wrong? This is f string for r in well rows. Okay, got it. Because we are missing the counter bracket.
Okay, you can see 51 50 51 51. So this is pretty balanced. It is able if it is able to do the balancing and validation, it means train is also balanced. So this is validation class balance. Okay. Now once this is done, it's time to finally go towards the training. So we we need to set up few things before training. So we are doing tokenizer. Okay, we require a tokenizer. And where we can how how to load a tokenizer. To load a tokenizer, we have to use auto tokenizer. Okay. And to load a model, we need to use auto model because we are doing auto we need to load the tokenizer. Sorry about that.
We need to do this from the pre-trained. And where do we have the model? Model is present inside the model name. So it will go to the hugging phase check this model.
Okay, you need to have access to the internet and then it will load the tokenizer.
Okay, we are not we cannot expect to load a tokenizer of Roberta and use a model of modern bird. We need to use the same tokenizer as they have used to train that model. So the tokenizer itself belongs to that model and we'll be using the same model. So tokenizer and the model is compatible. Okay. So let me print few things. If you want can check what is the tokenizer vocab size.
What is the vocabulary used for the tokenizer how much it can understand.
Okay. So I will use this tokenizer that I have fstring tokenizer dot vocab size dot vocab size and this will print what is the webcap. Okay. And we can also check the model max length as I mentioned. Okay. We are using 512 but model can give us more things. This is vocap size and model max length which we have set to 512 but model has better ability which we are not going to utilize. Okay, length. All right, let's load it. It will take some time. You can see you are sending un authenticated request to the hugging face hub and uh we are using that vocap size is 50,280.
is good for our use case and model max length and we we don't require multilingual support right we only require for English that's why we don't require a huge vocap size model max length is 8192 as I mentioned at the very start but we use we are using only 512 okay so this is about the model and what to do now we can create the encoder this is our tokenizer we also need to create the encoder okay so we can test it out we can test this tokenizer by converting a text to its encoder ing and then convert that encoding to the token to see how it works. So this is loading the tokenizer. Now let me create an example. So I'm creating an example. You can see this is the example. I'm saying that DeepS released V.3 yesterday. This looks like a news article claiming GPT 4.2 parity on reasoning benchmarks. This is the example. Now to use this tokenizer what we have to do? We have to pass this example to the tokenizer which will give us encoding. So in this tokenizer we are passing example.
Okay the tokenizer is object is here. We need to pass the example and we are saying that return us the tensors that is compatible with pytorch. Okay. So pt return the tensor and this will give us encoding. If you print this encoding right now it will look little weird cuz encodings is nothing but you will see the encoding of those. Let me show you that. Okay let me show you encoding.
Okay let's do one thing. I will show you the encoding later. I will show you how the token representation looks like. So now this is encoder using this tokenization using this tokenizer object we can convert um ids to tokens. Okay, this will give you all the ids we can convert back to tokens. So let me show you that. So convert ids to tokens.
We are not doing this for training. We are doing this just to understand how this tokenizer looks like. So in this encoding we have multiple things. We have mask of it. We have to uh input ids etc. We are extracting only the ids because we want to convert the ids back to tokens and see if it is possible or not.
Okay sorry about that. This should be input ids.
All right. And we want to extract because it will be inside an array or a torch. We want to extract that index. So we will see the actual value. So this is the encoding. This is the id that is from input ids we convert it back to token. So id to token and we are calling it as tokens. So from an example from the text we can get the encoding we can get the token. Now let's print all this. Okay. So what I want to print I have all this print statement here. So I want to print the input that is actual example. I want to print the tokens.
What is the length of token? Each token length. Okay. What is the actual token looks like? That is nothing but we converted from ID to token. What is a token ID? Whatever ID we are getting, how does it look like and what is the attention mark looks like? Inside the encoding, we also have attention marks.
Okay, so I will explain you that attention mask in a bit. Token is not defined. Why token is not defined?
Because it is token. So it should be token token and then done. Okay, this was the input. You can see input is only 1 2 3 4 5 6 7 8 9 10 11 around 15 words here is present in the input. Token is 23. We are able to see 23 tokens. What are those tokens? First the start of this sentence in this modern but base is mentioned by cls and the end is mentioned by SE. So this is how our transformer or encoder is going to understand that that start is CLS and whenever it sees SCP it means this is the end. Okay, different models are trained differently but modern bird uses this.
And if I talk about the token ids, it means the classification as the token is given a ID of 50,281.
And if you see the same for SCP, it is 50,282. It means these are the ending tokens. These have been introduced at the end of it. You can see this is 281 and this is 282. And if you check the vocab size here, vocab size is also 50281.
Okay. So it means total vocab is 280. At 280 it means 50,000 plus 280. Okay. But they have added two tokens that is meant for transformer. It is 81 and 82 that is classification CLS and SCP. And we are able to see that deepse has been splitted into deep se EK released GV etc. These are all the tokens. That's how it is made 23. So token this is nothing but the token itself. This is how the token looks like and each token has a ID and if you see what is the length of the token. Length of token is 23. That's why it is all 1111. So if you count this this will all be 23. So let me show you that length of this is 23.
Okay. And because the by default maximum length size is 8192.
Now wherever we don't have this token will be zero. So this one plus whatever is left. Okay this one plus whatever is left means if the total length is 8192.
If the total count is 8192 minus we are utilizing the 23 tokens only in this sentence. It means all the 81 69 tokens that are left will be zero. Okay will be zero. And that's how each sentence representation will be created. Okay. So 23 belongs to the actual sentence.
Whatever is not present, whatever is not present in the word or not coming from the word will be zero. It means these are the mask. So this attention mask means one means this is where you will find the actual sentence. Zero means that is nothing but padded tokens or empty sentences. All right, that is what attention mask means. This part is completed and now let's talk about the data set class. Finally, let's create the data set class.
The data set class. For the data set class, what we need to do? Why we require a data set class? Well, PyTorch requires a class where when we call with a ID, we can get the data from it. Okay?
So, if I want to extract some data, what will I do? If I give you a list of 10 elements, if I ask you give me a data, I cannot ask you give me a data because I'm not providing enough information. I need to provide give me a data at index number five. Then what what we will what Python will do? Python will go to that index and retrieve me the data. But here in our case our data is not like a single value. What we need to provide if PyTorch will ask give me a data at index number 100. Then we need to return whatever data is required for the training. So we want to create a class where PyTorch can use that class pass the index and get the data. That is nothing but a data set class. Okay. a way for the data set because data set is currently in our format. We want to return the data set in which what PyTorch requires. That is the first thing. Once the data set class is created then we need to create data loader because the data set class when we pass ID it gives you data. But to train a model we require more things. We require that if I want to pass 36 it means 36 data in a single batch.
Currently data set class cannot do it.
So do we need to implement those functionalities um by scratch? No. We can use a data loader class. In there we can pass that we want a batch size of 16. We want a batch size of 32. We can write the instructions where it can it can help the model training by providing all of those wrapper on top of it. So we don't have to worry about passing batch size of 16 etc. The data loader class will handle those including the shuffle part of it. Okay. The data set class is for PyTorch that PyTorch can ask the index. It can return it. data loader class is top of that data set class that we're creating just now. It is going to create shuffle. It is going to get the data in batches. That is what we are going to do next. All right. So let's create the data set class. For the data set class I'm using class and then call classifier data set. Okay. Here it is mandatory to import the data set that we are getting from the PyTorch. If I scroll up, you will see that this is what we are importing from torch.youut data set. Okay. So if we are creating our custom data loader, custom data set class, we need to inherit the data set.
Okay, that's what we are doing. And this name can be anything that is classifier.
Okay, this should be sorry about that classifier. Okay, classifier data set then it is pretty straightforward. It needs to have few things. It needs to have the init method and init method we need to have self. We need to have few things that how many rows we have. What is a tokenizer we are going to use because when we train the model what we require? We need to pass the model the tokens. We cannot pass the uh data as it is the raw data. So we need to pass the tokens and also when we are creating token we cannot just pass all the 8192 because that will be little overburden on the training part. So we want to restrict it. So whatever we want to do beforehand we will be passing it here.
Okay. None of this is mandatory. We are just passing the things which we require for the processing. So for now I'm writing pass. Okay. It needs to implement two method. These are mandatory that is length. What is the length of the total data set? Okay, so we need to implement length and we also need to implement the final thing that is the part where PyTorch will be using it and passing it via the index. PyTorch will say give me the data at index 100.
So we require a way to pass so that PyTorch can pass that 100 and we will get the data. This is self comma I. So PyTorch will be using this data set class. Pass a index and we need to get the index. Where do we have that data present in rows? Do we require to process that data? Yes. How to process?
Using token. What is the length? We are passing the maximum length of this one.
So we will get PyTorch will get data from this. But at what format, where etc. we are going to implement via other functions. Okay. So length is pretty straightforward. Length is nothing but it just need to know what is the length.
That's all. So length is return length self dot rows because self dot rows because we are in a class. Okay, this part is done. Then we need to create self dot rows is equals to rows.
That's how it is able to load self dot rows. And then we need to load the tokenizer. So we can use it in get item.
So we are creating self dot tokenizer.
Okay. And where we will get the tokenizer from here. Let me write it as tokenizer. This looks better. And self domax length which we'll also pass it from here. self domax length is equals to max length. Okay. Which is already here. Okay. Now these are all the things uh two things are implemented. Now the get item. So whenever pytor will ask for a index first we need to get the data.
Where we will get the data? We will get the data from self dot rows of I. This will give us the data. Sorry, it's not a tupole, it's a list. So it will get the data from here and we are going to call it as R. But can we return this R? No, we cannot use this R as it is. So what we have to do whatever tokenizer that we have created self dot tokenizer, we are going to use this self dot tokenizer.
Okay. And then pass this R here. But not R. R does not contain the whole thing.
Our R has lot of information. Where is our R? Each row contains lot of information. Do we need to pass all this? No, we just require the text. So we are going to utilize R of text. So let me scroll the bottom. This will be R of text. And what we have to do with that?
The tokenizer. This is the first argument. Yes, we want to tokenize the R of text. What else? What else we have to do? We have to do the truncation.
Okay. Yes, we need to allow the truncation. Okay. It means cut off if the data is having zero. And the max length, the max length of our any data can only be self dot max length that we are setting that is 512 that is the maximum. Okay. So if a word is or if a sentence let me write if a sentence is having thousand words if a sentence is having 1,000 words it will cut off at 512. If a sentence is having 80 it will not become 512 it will not become 512 but it will become only 80 because we are making the truncation and true. Okay. So if because max is 512 it will be cut off at 512. If sentence is 80, it will be 80. It will not grow to 51.2. Okay, that is what we are doing. Okay, that is the max length. And do we need padding? Okay, truncation we have allowed that yes, you are free to truncate. But uh if I talk about deep learning, okay, if I talk about deep learning and the architecture itself, we cannot have data that some data have 80 and some data have 512 as the length.
All of them needs to have of the similar length. Okay. The truncation is yes, we are allowing the truncation. Okay. So, truncation was not for 80 to 80.
Truncation was if the data is 1,000.
Okay. If the data is 1,000, truncate it to a maximum length. Okay. Truncate it to a maximum length and it will truncate to 512. But if it is 80, okay, it will keep 80. But does it support? No, it does not support 80. Internally, all of them needs to be of a single length. And this is where we will decide it. We want padding. So if words are 80, what we should pad with? We should pad till the max length. It should have the same length.
Okay. So padding is max length. And what we need to return? We need to return the tensors in the pyro format. So return tensors of the PT. Okay. So this is how we are going to return it. And this completes our self.kenizer. This is pretty straightforward. Previously I have just used tokenizer with the word and written tensor. Now we have three more arguments that is for the final model training and this will give us encoded data. So for this text we have encoded and we know what is present inside the encoded. In encoded we have input ids in enencoded we have attention mask. That is what we are going to extract now. So what we will return to pytorch okay when the model is training we will return few things. We will return input ids. Okay. We will return attention mask.
We will return attention mask also. We will return the actual answer. The actual answer is also present inside the R because we are loading R here. We have the full information. In encoding we are only passing the text but this R also has the actual labels. So this we will get it from here. So we need to convert first. Let's convert the label to the torch tensor because in the calculation it will only support the torch. So we will write R then is AI that is the first thing. What else? The next thing is R is layoff is layoff.
Okay. And then this completes the tensor part. So this is the tensor dimension completed. But we need in the dimension of float. Okay, the data type should be float. So d type is equals to torch dot float. Okay, now we need to implement few more things that is how we will get the attention mask. Attention mask and other things are already present here.
So I'm going to copy this whole thing.
So we have to do less writing. I will remove this part. So where do we have attention mask? Attention mask is present inside.
Okay, it is inc attention mask but it will have a dimension which we want to remove. So we are doing the squeeze of zero. We adding dimension. So if you check what is the attention encoding looks like. Let me show you that. Okay you are able to see this is having with torch. But if I try to add a squeeze here. Okay. If I try to add a squeeze here, it will remove that dimension. Okay, extra dimension. So now it is compatible. So we are removing that dimension. And this is uh attention mask. What else? We also need this part that is for the input. Okay, we also need to squeeze this as well because this also have that extra dimension. Now this part is done. Let me remove it. And this is how the final data set class looks like. Okay, input ids, attention mask, labels, etc. All of them have a squeeze. This one does not require because we are creating our own.
And this completes our data set class.
Let's run it. Where is the issue here?
Because we have a weird comma here.
Let's run it again. Some invalid syntax.
What is the invalid syntax? Just wait.
Do we have that comma? Oh, comma is at the end. Done. Okay, so we have created the data set class. Now it's time to use this to create the data set train and well that PyTorch utilizes. Right now our data set is in the format of scikitlearn object. This is train rows.
Okay, this is where our training data is. If you check the type of it, the train rows d type of train rows. Sorry uh only the type not the type. Okay, it is in the list because inside this we have the data. Now we need to convert it to the PyTorch compatible format which is nothing but tensors. So train rows we are converting it and how do we convert it using the class just now that we have created. So this is train rows and we are passing what we need to pass whenever we create it we need to pass the actual row tokenizer and max length.
What are those? tokenizer is already present inside the tokenizer object that we created before.
Okay. And then max length. This creates train ds.
Okay. Same thing we will do for val. Okay.
Wros. This one will be ds. Everything else is same. This is simple. And I can show you that that how data set looks.
Okay, classify data set takes no arguments. And why is that? Because this is not a special method.
All right. Now it is done. Now if I show you, if I ask you that what is the index? Let's suppose P spy is asking us give us the data at index number zero.
And now you will see okay something is wrong. I think we are writing weird thing here. D type D t Yp. Okay.
Now you will see that whenever I call with certain index, I will get all these three values. What is the label? Label means this belongs to AI. This is attention mask of it. If you see the attention mask, it has a length of 512.
Input ID. This is also a length of 512.
Okay. And you can see all of these are paddings. It means these are all the padded values. All the input ids. If you want to print this, I can print this input ids.
Okay. Okay. I cannot print more than that. But uh all of this basically means that these are the empty spaces. You can see the mask is also zero for those.
Okay. Now once we have the data set, now it's time to make it compatible for the final train that is nothing but in batches. So for the batches, what we require? For the batches, we require data loaders. Okay, how do we create data loader? Data loader is pretty straightforward. We just use the data loader class and there we pass that where do we have our data? Our data is present inside train DS. What we want to do with the data? We want to have that in batches of train uh batch size. Where do we have our batch size? This is our batch size. Do we want to shuffle this?
Yes, we want to shuffle this. Shuffling is very important. And number of workers, it means to load the data. Do we want to do parallel job? Currently, I'm using zero. It means no parallel job. We don't have lot of data. When we have millions of data, thousands of data, then use this to two or four. But keep this in mind that whenever we use number of workers in Windows with this notebook configuration, sometimes it keeps on loading. It keeps on working till infinity. So, we don't want to do that. So when we work with zero at least it is in my system zero works but 0 1 2 etc does not works in notebook but even if you set five it will work in the script py file but not the notebook okay number of workers then I want to pin the memory okay pin the memory where device type cuda okay device dot type is equals to cuda okay so I'm pinning the memory Sorry, this is for faster as sync GPU transfers. Okay, this is the data loader. This is data loader for train.
So train loader.
All right, same thing I will do for well. Everything else is same but well will be well loader.
Well loader and this one will be well ds.
Okay, well DS everything is same. Now what we can do here itself we can just print or let me make it better.
Okay. Batch is not defined. Where do we have batch? Actually it is batch size not just batch. We have defined batch size not batch.
What else? Number workers. It is always s that is giving us this error. Now it is done. All right. Now what we will do next is print. Let's see if it is working in batches or not. Okay. So, first we will print how many epochs we have. So, let me write train batch/ epoch.
Okay. And then let's print directly with comma that what is the length of train loader.
I will tell you the calculation how did we get to this particular path. Same thing I will do for validation. Okay.
validation batches per epoch. This is train batch per epoch. Okay, sorry batches per epoch. So how many batch we will have in each epoch? Because in each epoch it needs to learn the whole data.
So one epoch all data, second epoch all data, third epoch all data. Similarly, so one epoch how many batches we are going to have for the data considering the batch size is 16. We selected 16. So we will see that right now. Okay. So we are going to have validation. Sorry, this needs to be val loader. So we are going to have 72 batch per epoch and we are going to have 13 batch per epoch.
Why? I will tell you how many how much data do we have in total? How much data do we have in total? Let's go to top where we have the full size of our data.
This is train and this is well. Okay, let's copy this.
Okay, this is our full data. And if I talk about if I have this number, let me comment this. If I have 1149 and in uh okay let me divide this particular part by total number of batches that we have.
Okay. So in each batch we can have only 16 data. How many data do we have? 1169.
It means how much iteration we need to do in each epoch?
71.81. And if you round it off it will be 72. So it means inside one epoch inside one epoch we are going to have 72 iterations. Why do we require 72 iterations? Because we have 16 as our batch size. It means each iteration can only learn from 16 data. So if we have 72 iteration then it will cover the whole 1149 rows of data in each epoch.
That completes our training. Okay.
Similar calculation will be happening for 203. So if you apply 203 by 16 you will get uh we will not do 16 why let me just show you that okay so here we are getting okay this is for validation is fine validation is 16 only it means it will be 13 epoch but if I talk about testing when we're going for testing in our testing keep this in mind that this one is set to 32 so if I talk about the testing testing will be 200 by 32 okay which will be 6.25 25 something which will be around seven batches in evaluation. Okay.
But for validation we only have 16 for both the cases. That's why we require 72 and 13. Okay. That is a calculation if we care about. If you don't care about then let's move to the next cell. Okay.
What is the next cell? Our data is ready. Now it's time for the model.
Okay. So now I will write the model the actual two head classifier. Two head classifier.
Okay. This is our model. So what we require? We need to import the nn.m dot module to create our own model. Why we use nn dot module? We use nn.module to track the model. When we import if I talk about nn dot n dot module. Okay.
Sorry, this is capital M nnn dot module which we are going to inherit to create our uh classifier. What is the advantage of NN dot module? Well, it tracks model weight. Okay, it tracks model weight.
What was the previous weight after updation? What is the new weight? It does forward propagation or forward pass. Okay, and using this NNO module, we can convert we can transfer our model to GPU or CPU very easily. If you don't do this uh we cannot create model basically. Okay. So this is n dot module. So let's create that model. So let me write class I'm going to write our model as two head classifier.
Okay. Two head classifier. And I will write nn dot module. First part is done.
Then we need to import few things. We need to implement few things. First is init. Okay. I will talk about init. What things we will do here. And we need to implement forward. Okay. So two methods we have to implement in order to compute this. Okay. So forward pass will require two things. Forward pass will require what is the input ids. Okay. What is the input ids? Which means the tokens itself the tokenization and then the token numbers. This will be the token numbers itself. And this will be the attention mask.
And this will be the attention mask.
Okay, this is what we have to implement.
So now let's type in init. Let's write what the things that we require. First thing is whenever we use n.module, we need to initialize those as well. So we can utilize its functionality. So we will do super init. So its constructor is ready. Now once its constructor is ready, then we can use the other components. Now previously for tokenization we use auto tokenizer. This time because we require model we will use automodel and we we are using a pre-trained model. So I'm using automodel dot from pre-trained.
What is the model name? We already have the model name set in our parameters.
And this will give us what? This will give us the encoder. Okay. The actual model itself. Our model is encoder based. It does not have a decoder because we don't require it. Okay. We are not generating anything. We are just doing the compression of the information text and doing the classification. Okay.
Self.enccoder. What else? This encoder model outputs uh let me give you those numbers that what it outputs. Okay. So this encoder model outputs a dimension of 768. So let me write those. Think of it as like all this example. Okay. Let me do the actual calculation. So we know what we are doing here. So the model that we are having outputs the hidden dimension. I'm not talking about the actual encoder what the model this one does. But the hidden size the final model encoder part outputs length of length of token size max token length comma 768. Okay that is what it outputs.
So if we have batch added here. So we are also going to have whatever the batch information is. So in our case model is going to output 16 that is our batch size. Token length is 512, 768. So we need to first what we have to do this is the token representation the compressed representation or the attention token mask everything present here. What we need to do we need to take this output this output that we have that is 768 and then create a linear layer from it so that we can perform the learning. All right. So what we have to do a linear layer we need to map and that linear layer is going to be the same length as 768 and then we are going to map it to the location of two. Okay, this is is AI and this is going to be is layoff.
Okay, this is what we will do. So we have to first extract this 768 that is uh the automated process and then we will map it to the linear layer of this two number of labels. So how much label do we have? First let me write it write those things down because to create that information we need to have this information what is the length as well as this information what is the length let me do that so how many number of classes we have let me write number labels I'm writing inside the model itself number of labels is equals to two that is is AI and is layoff okay and how to get that 768 out that 768 is present inside self dot encoder dotconfig.
Okay, every model has that configuration the object inside it the dictionary and what is the hidden size of this model that is nothing but 768 we are saving it into a hidden all of this is in in it because it will run only one time one time encoder one time calculation of hidden once that is done I also want to add dropout okay I also want to add dropout and we can add the dropout dropout we have already mentioned at the very start of the configuration okay so this will be self dot dropout.
So we are writing all the components that we require for the forward propagation. Okay, dropout we require this format this sequence does not matter. Sequence of these two matters but sequence of dropout I could have written drop out here and it will be the same thing. Okay, because I just want dropout. Okay, this is our model and the classifier self dot classifier is equals to nn dot linear linear of 768 comma 768 comma self dot number of labels that is nothing but here okay if you want I can also write uh this will Okay. So this is all the requirements.
Now in the forward propagation what we need to do any input that we are getting. Okay. Any input already that we are getting along with the attention mask we need to pass it to the encoder.
Encoder will give us the output and that output we need to process it. Why do we need to process that output? Let me talk about that. All right. Let's move to our scribble again and then draw a few things.
Why do we need to do the processing? As I mentioned, our model outputs batch and this one is nothing but the token size and this one is from the hidden state. Okay, how do we get this value?
So this one is nothing but 16, 512, 768. Now we need to solve the problem.
This is the model output that we are getting. But because we need to do the classification, we have a linear layer ready. There is 1 2 3 4 5 etc etc. The final one that is 768 which we are mapping to two that is is AI and is layoff. Okay, that is what we are mapping to. So this is all the connection looks like. This is NN layer.
This is what it looks like.
Okay. All of them will be connected. But if you see this is not compatible. It requires a input. This particular part requires a input that is of dimension that is of dimension 768 cross 1. But are we getting 768 + 1 from here? This one ac accepts 768 but this one is totally different. So we need to do something with the model output so it is compatible with this one. So learning can happen. You cannot just take this input and then pass it.
It is not compatible. So to make it compatible now we will do the processing. Let me write those processing here. If I talk about what is the dimension of attention mask. So let me write attention mask.
Attention mask dimension is 16. That is the same as batch. Total of batches 512 of each but it is only one because attention mask does not have any more information. It has whatever the number of tokens are it will have the values of those as 1 1 1 0 0 etc. So attention mask this dimension is not there. This is what attention mask looks like. Okay.
Now what what we can do we have our input for each information this is let's not talk about batch here right now okay this batch we can process whatever we can process if we are able to process this information to this we can also add the batch information to compress it okay because this uh is the answer for 768 + 1 we require for one row if you have multiple row it will automatically be 16 + 768 that can happen parallelly so what we need to process we need to process the part that I am writing here.
So let me highlight those. This is what we need to process. This is a single text input dimension. Single text attention mask. All right. Now let's process those. So if I multiply hidden state, what is hidden state?
Everything I made it white so it looks weird. So just give me a second. Okay.
So what we can do if I multiply hidden state which is nothing but 512 + 768 hidden state of one text that is 512 + 768 multiplied by the attention mask that is 512 + 1 okay multiplied by attention mask we will get what if you multiply it 512 + 1 the final answer this one is think of it as like a token token is X Y Z. If you don't have token that is already padded okay that is already padded. If I multiply it with the mask that is 0 0 0 all you will get at the end is this XY Z this will be XY Z and then others will be zero. This padded part will be empty.
Whatever the token of those padded ones right the token empty tokens also have some ids that id will be zero. This is what the output that we will get. So what is the dimension of this one? The dimension of this one is nothing but 512 + 1. Now this is the dimension that we are able to generate from here. Now what we can do if we do the sum of this one if we do the sum of this one it will generate a single value. Okay. So 512 cross multiplied by 768 multiplied by so hidden state multiplied by mask will give us a single value. Okay. So this is a single value per row. So we have if I have 16 data we will have 16 value at the end. All right. This part is over.
So this is the first calculation that we just did. So what we just did now imagine like this.
This is the size of each row.
This is row size 512. How many such output this we get from hidden state? We get from 768. That is the dimension that we get. When we multiply this by 512 + 1, we will get what?
We will get what? we will get we will get the same thing as the actual input but the values are different 512 cross 768 but this one we actually have zero so if this was xyz a b c and if this was 1 1 1 0 0 1 * x will be 1 1 * y 1 * z * a will be 0 0 * b will be 0 so here we will get xyz Z 0 0. Okay. If I do the sum of one, it means for every row, if I do the sum of it, what will happen? What will happen?
I will still be left with each row will be of sum. I will still this is value one, value two, value three, but 768 will remain. Now, what is the dimension?
Dimension is 768 + 1. Is it compatible with the linear layer that we have for the output? Yes. So what we have to do?
We have to multiply the hidden state with the attention mask to first get the information that is relevant for the prediction. If we do sum of each row, it will be collapsed to a single row. But the actual dimension of the hidden state 768 will remain. And that is what we have to do. Okay. So this is what we'll be doing. All right. And uh let me do one more thing is that okay first let's write this part okay I hope we understood what we are trying to do there is a compatibility issue that what is coming from the input and what we want to learn okay so how many learnable parameters we are introducing if I talk about the learnable parameters was uh total number of hidden is 768 multip multiplied by two. So these are the learnable parameters that we are introducing. And this one already has, if I talk about the encoder, if I talk about the encoder, encoder already has, let me check. Yeah, around 150 million parameters.
Okay. So if I talk about the parameter count, it is around 150 parameters and this one is around 1538.
Okay. So now that we have the hidden state now let's try to use it. We have all the components. Now let's try to use it. So first let's try to process the input ids. I will write self do.enccoder. In encoder what is the input ID. Where do we have the input ids? Encoder literally accepts input ids and we have input ID present here. Next is it accepts attention mask. Attention mask is present here. What is the output? Output is just out. Can we use this output directly?
No, we cannot use this output directly.
This is having a high dimension. So we need to process it. So what we can do?
We can check output dot last hidden state.
Okay. So this will give us the actual answer of hidden state. Okay. Hidden state is the dimension which I mentioned. Hidden state is 768 + 512.
Okay. And what else we require? We also need to check the attention mask. Where do we have the attention mask? How can we get the attention mask? We have the attention mask here as the actual input.
So we can use the attention mask. We can unsqueeze it.
Okay. We can unsqueeze it to add the extra dimension. Squeeze was to remove the extra dimension. Unsqueeze is to add another dimension. Okay, I will tell you why. and then we will convert that value to float. So it is operation is performed successfully. If I talk about the dimension of this one, what was the dimension? Dimension was total batch.
Dimension was total token size.
Dimension was total hidden 3768.
What is the dimension now? The dimension of attention mask was originally batch and then token size is 512.
But it does not have any information. Is this compatible? This is not compatible.
Okay. So, let me write batch of 16 512 + 768. Is this compatible for the calculation? No. So, we require additional dimension which we added using the unsqueeze. Now, it is compatible. So, what we will do? We'll to do the multiplication of HS multiplied by attention mask. Okay, that is mask itself. So, let me write this as mask. So, HS and mask, let's multiply.
Once we multiply what we have to do we need to do the sum row wise. How to do sum row wise? Sum of one. So now sum will be performed row wise. Each row will be giving one row. This is actually column wise. It means all the columns you want to be single value. So if I talk about row this is how we that's why the pytor information is required. If I talk about what we are summing we are summing column 1 column 2 column 3 column 4 to a single value. So we are actually summing the column and we call it as we are summing by row. Okay. And once all the value is squeezed to a dimension of 768. We need to do one more processing. We can scale it better. So the magnitude will also remain. So what we're going to do we are going to scale this value by using mask dot sum of 1.
So what we are doing whatever we have here okay whatever values we are getting here column by column we are going to get 768 by 1. But this particular value, this particular value is a large number because we have summed it. So we want to minimize it and we also want the scale to be maintained. So what can we do? If I talk about the mask itself, mask contains the information about the information how much sentence we have.
So we also want to have that a longer sentences should not explode the value.
A shorter sentences should not make the value big. So what we are trying to do?
We are trying to do the scaling by total number of ones that we have. Okay. So if we get a value of 100 from here, we are going to divide by total number of tokens that it has. Okay. Total number of one it has that is three. Total number of sum. Okay. If another column has longer text, so we are going to divide by a longer number. Let's suppose 100. So we are going to make the value shorter. So for longer text, we are going to make it one. For shorter text we are going to make it little just for that magnitude difference. So transformer has more thing to learn.
Okay. Now let's do that. So this is the actual 768. The sum is there. We are scaling it. But there one thing we need to worry about is there can be sentences there can be sentences where mask is all zero. What will happen in those cases?
If I do the sum of those sum of those whole thing is zero. Anything divided by zero is it will go to undefined it will go to infinity etc. We don't want it. So there is a special method present here that is called as dot clamp. When we use dot clamp what it checks if the sum of the whole thing or if you're trying to divide the value by zero. It will try to replace it with a very small number which we will mention that is one of the parameter. So here what I will be doing I'm using the smallest parameter minimum 1.1 E minus 9 a very small number. So what I'm saying if there exist no encoding or the mask is all zero divide by a very small number okay which will make the thing very big very big transformer can relate it to something like if it is very big then it might mean irrelevant okay so we are going to clamp it this is like solving the division by zero error so we are doing clamp clamp of minimum is equals to 1 e - 9 Okay, you can choose any number and this one we are going to call it as pulled.
Okay, so the final information that we're going to get for this one that is compatible for the linear rare whatever the input that we are going to get we are calling it as pool. Now what we will do we will pass that pool to the final linear model. Okay. So once the forward pass is done once the forward pass is done then we can directly in the return statement of this one in the return itself we can mention self.classifier.
Okay this is our classifier self dot classifier which is nothing but here it accepts hidden hidden of 768. Do we have that compatible? Yes. So we are going to pass pulled. Okay that will go as a hidden. Okay. And it will process it to the number of this one. Okay. And because we are doing the training, we can pass this pool to the dropout layer itself.
And dropout will handle the 20% dropout that we have mentioned. Okay. And to check the dropout, we can check the values that we have.
Where do we have our dropout? That is NN. Let's see what is the value we have set before. dropout we have set to 10%.
Okay, that is a dropout we are using.
But I think this dropout value needs to be big just wait here we have defined the dropout and we need to use this dropout here.
Okay, but I think when we are going to initialize the model we can create those. So let me write this dropout that we can pass while initializing the model that is we need to pass the model name.
We have not yet created this. The model name is here and by default we are going to select one dropout 0.1 if user pass.
Now everything is completed. We have the dropout uh argument here as well. We have the linear layer number of labels is also present here and we have the forward propagation or forward pass that is attention that now that we're getting the final hidden state output we need to map it to the actual dimension of set to drop out of pool now that we have done it the pool is now compatible with the linear layer and this completes our end to end model architecture okay now that this part is done it's time to initialize the model Okay. So let's instantize the model.
Instantiate the model. Okay. So we are instantiating the model and it is going to be pretty simple. It means just use this class that we created and pass the model name. Okay, that we have mentioned at the very start. And our dropout is also that same parameter that we mentioned at the very start that is dropout. And because we it is inheriting NN dot module as we are able to see here what we can do we can move this to our GPU if you have GPU. Okay move this to device and finally model is created. Okay, model name because there is no nothing like mode name torch.n NN has no attribute drop out. It should be drop out not drop out. Okay.
All right. You can see loading model weights. All of these things are loaded.
Don't worry about this. This is just we are not every model is using this decoder bias expected. This is not our issue. Okay. This is from transformers.
And you can see this unexpected can be ignored when loading from different task architecture. Not okay if you expect identical architecture. No, we don't expect identical architecture. Once the model is loaded, then we can check information about the model. What are the total number of parameters that this model has. So let's write those parameters that this model has is for P in model parameters because it will have lot of parameter, lot of different layers. So parameters we going to iterate over it and for each of them we will do P dot numerical. Okay, this will give us number of parameters in each layer. And then once we have that information, we just need to sum it.
We just need to sum it. This will give us total number of parameters.
Similarly, you can check total number of trainable parameters. So this is n total how many of them we can train it. So for this one, we will use this parameters is there. But we need to check a condition if the P dot requires gradient.
Okay. Next is we can check how much parameters we have in our classifier that is the one that we created that is a classifier. Okay. It is a part of this model. It is a part of this architecture that's why we can use it directly. So model dot classifier not parameter model.cl classifier dotparameters.
Okay that's how we do things for custom.
Okay. And this one will be head. This we are calling it as our head. The final head. So n total n train head. And let's print it. friend total parameters total parameters is and total similarly let's copy and paste quickly so that we can start our training finally so this is total trainable params okay and this is total classifier.
All right. So yes, this is what I mentioned. This is around 150 million parameters. That's what I have written before. And this is 153 and that is 768 multiplied by 2. Okay. Now model things are done. Now the final configurations of model that is we need to mention what is the loss. We need to write our optimizer and we need to create our scheduleuler.
Okay, I know you might be thinking that this is never ending. When we are even going to start the model training. Well, soon. Soon. Answer is soon. NN dot BCE.
Okay. Binary cross entropy BCA with logits loss.
Okay. BCA with logits loss. Okay. This is what we're going to do and this we are call it as loss function. So whenever we use BC with logic use we don't as you can see in the model architecture we have not mentioned sigmoid anywhere. Well we don't have to because when we mention this nn.bc logit loss it automatically uses the sigmoid in the back end. Okay but this is during training but for evaluation we need to create our own sigmoid. Okay. because you'll be only doing the forward propagation but for the loss it is using those internally.
What was the full one binary cross entropy that I have imported? Have I imported this? No. Okay. So let's use it. Okay. So NN dot bce let me check spellch check it with logits loss done.
Let's run it. Okay it is running. Next is optimizer. So optimizer is Adam W. And what we need to optimize? We need to optimize the model parameters.
Okay. What is our learning rate?
Learning rate is learning rate and weight decay.
Weight decay is also weight decay that we have defined.
Okay. This is our weight decay. So loss function is ready. Optimizer is ready.
Then what we can do? We can create how much total steps how many total steps that we need to run this model because that is required for linear schedule.
Okay. When we are going to create the schedule for the warm-up, the total steps is nothing but length length of what? Length of train loader.
How many times my train loader is going to run multiplied by number of epochs because my train loader is going to run that 72 times. But it also depends how many number of epochs we have. These many number of epochs we have. Okay, this many number of epochs we have. Next we need to also find out what is my total number of warm- up steps. Okay, so there is something called as warm-up steps. Let me tell you what do we mean by warm-up. So there are things like let me write the meaning of those. Okay, warm-up steps. Warm-up steps is exact training steps from the ratio. Okay, we will define the ratio but exact training steps from the ratio. We have already defined the ratio. If you check the warm-up ratio as the very top, let's go to our configuration where we have defined warm-up ratio. Here we have. Okay, let me copy this.
So we can write it here.
Okay. Our warm-up ratio is 0.1. Okay.
And warm-up steps is nothing but exact training steps. We need to calculate that. So if our ratio if training has let's me define if training has 1,000 steps. So and warm-up ratio is 10%. This is nothing but 10%. Okay. And warm-up ratio is 10%. Then warm-up steps is nothing but 100 steps. So we are going to do the warm up for 100 steps. Okay. In this meantime, our learning rate will be zero or near to zero because what happens when we start the model? Okay, let me start. What happens when we start the model training? Our data set is very different from what the original model has that is our model. But if we start the learning rate with some number at the very start the loss is going to be very high because that model is not used to our data and when the loss is very high and we still have the learning rate the pre-trained whatever the information of the original model bird but is is going to mess up and we don't want it what we want we want the training loop we want our model to understand that this is what we are doing understand the process so the learning we don't want any learning at first initial 100 steps in the example that I just shown it will not be 100 in our case because the training sample is little bit higher it will be 100 one uh 102 105 these steps for the warm-up.
Okay. So this is the example of what the warm-up step means. Okay. So total steps we need to do. We need to also calculate warm-up steps. And how do we create warm-up steps?
warm-up steps we will create from what is the total steps do we have? So from the total steps multiplied by warm-up ratio that is the 10%. And that's how we get warm up but it can be a float. So let's convert that to integer. Okay. So at the starting 100 or something iteration it will be zero. Then the learning step learning will rise. This is how it will happen. Okay. At the start of the training. So this is learning rate. This is number of epochs.
As the number of epochs increase, the learning rate at the very start will be here. Okay, it will keep on running till the 100 epochs. Once we are at around 100 epochs, once we are at around 100 epoch, then it starts to increase the learning rate bit by bit. Then you will be able to see learning rate is increases. But because it is a scheduleuler, at some time it will start the decay that we don't want. The more epochs we will go, the more it will bring it closer towards the actual decay number. So that's how we are doing. We don't want to keep on increasing the learning rate. Okay, we don't want to keep on increasing the learning rate. At some point when we are closer towards the final epoch or the final training steps, we want to move it slower. So we want to learn things here in this particular part and then slowly stabilize stabilize in this particular direction. Okay, that is what the scheduleuler will do. Now let's create that scheduleuler. Our scheduleuler is present at get linear. This is the model that we have. Get linear schedule with warm up.
Okay. And for this we require the optimizer because optimizer is going to do the actual learning. Warm-up steps will define when we should be learning rate should be very small and total steps will be total steps. Okay. And this will be our scheduleuler.
Okay. And this completes our scheduleuler. So you can print total steps if you want.
The total steps is going to be here. And total warm-up steps if you want to print to get the exact warm-up. The above is just example. So don't consider that we have considering thousand only. No, whatever is the number. Number of epochs is not defined. So where do we have number of epoch? So number ep o c h s.
So let me check where do we have number of epochs? We have to go to the configuration and check the spelling of it. So number of epoch did we mention number of epochs anywhere here? Yeah number of epoch. So it is not epochs, it is epoch. Okay. Now let's run it. Okay. So 576 is the total number of steps including all the epochs and warm-up step is only 57. Okay. Yes.
Because we have less training data. So uh the data is little bit decreased.
Okay. So this is our data total steps and uh now let's move forward.
All right.
What else do we have here? Okay. Mixed precision. Next is we need to set the BF16. So mixed precision. We are training the model in mixed precision as I mentioned.
Let me talk about the BF16. If I talk about the topic BF16 okay you need to be a little familiar with the sign. So each data type has some exponent category and mantisa.
Okay. If I talk about mantisa if I say 1.31 300.131 this is called as mantisa part. Okay. So BF16 has a exponent of 8. It supports eight exponent and it supports a mantisa of seven. And if I talk about the bits, how much bit it accepts, it only has 16 bits. It does not have lot of information. But if I talk about the range, okay, the range is pretty huge.
Range is around 1 e to the power of 38 to 1 e to the^ of 38. It means this is negative. So negative 1838 to positive 1838 is the range of BF16. If I talk about FP32, it means 32 is as the precision. Okay. If you add these two 8 and 7, it will get to 16. 32 means it has 32 bits. Exponent is 8, mantisa is 23. So even though FP2 has higher mantisa, but if you check the exponent, exponent is the one that can store larger values. We are able to see that BF16 has the same exponent as the FP32.
Yes, the Mantisa is much lower. But when we use BF-16, we get almost same exact training accuracy as compared to FP32.
But this is not supported in everything.
Okay, only few GPU supports BF16 and uh okay, the range of this one is same as this is having the same range but the Mantisa is lower. But if I compare this with something with FP16, FP16 is 16.
But if I compare it is 8 and 7. Same exponent, same mantisa. But the range is very poor. Range is - 6 a to the power of 5. Sorry, this is not minus and this is e to the power of 38.
Okay.
This is e to the power of minus 5 and 6 e to the power of 5. Okay. So here you're able to see that BF-16 has the same range as the FP32. That's why we are considering BF16. Okay. And one thing about BF16 is that BF16 is not used everywhere because as it has less mantisa. Okay. As it has less mantis as compared to FP32. We cannot use it for the whole thing. So where do we use it?
we use okay model parameters are always in model parameter are always in FP32 okay model parameters but when we do forward pass when we do few things like forward pass what PyTorch does PyTorch converts the weight to BF16 okay BF16 for operations like for operations like matrix multiplication, okay, and linear layer plus in attention.
Okay, so if you reduce it for the forward propagation, it becomes very fast and it does all of this. It creates a copy. It does not remove the F32 model parameters. It just creates a new copy and then process this for the forward propagation and then move it back to FP32. It does this part FP16 BF16 to FP32 to maintain the numerical stability because when we are going to decrease when we are going to update the actual weights the weight even a small small small precision in the weight updates is very important but if we remove that precision which we removes in BF16 the model will not learn that much. Okay. So this is the optimal method FP32. Few things get converted during the forward propagation to BF16 and then converts back to FP32 for things like for things like softmax when we are doing softmax when we are doing layer normalization. So it converts back to that layer normalization as well as loss calculation.
Okay. So uh this is how it works. So BF16 when we said why do we call it as mixed precision. The first topic that we have written in this particular cell is mixed precision because it uses multiple precisions FP32 and a mix of BF-16 only if your GPU supports it. Okay. So this is for higher training. We get almost no uh we uh almost the same training as we are training with FP32 when we use BF16.
So there is no loss there. Okay. So let's use it. So we will use use AM. It means automatic mixed precision is use AM and we want to use this when we have BF use BF16 and your device type should be CUDA.
Okay. Otherwise don't use it. It will slow down the process because if you don't if your device does not support BF16 then CPU will handle this part. It is converting from F32 to PF16 which will take lot of time.
Okay. So don't run this on CPU. Okay.
This is use AM. So I have set to BF16 and this part is over. Okay. On the single line that we need to use. Finally uh we need to create the last method before the model training because in model training we need to compute how much error we have. And how do we compute how much error? We can compute how much error.
Okay, using an evaluation function.
This evaluation function we will use same as it is in testing. So this is the code that we're going to utilize. Okay, we use evaluation function when we don't want to update. We have to not learn from the model updation. We don't want to do any model updation. So we have to make dot no gradient. It means we don't want to update our gradients. We don't want to track our gradients. So we are creating the decorator and then creating the evaluation function. So evaluate loader.
Okay. It will accept model. It will accept the data loader which we want to test. And then the threshold. As I mentioned for the evolution, we need to create our own threshold. I'm setting it to 0.5. The first thing is set the model to eval mode. And we are doing this with the help of with tosh.nograd. Okay. Uh you can do this with the help of context manager or the decorator. Okay. With tosh.nowgrad is also that is constru context manager and this one is within the decorator. Both of them are the same thing. This one is for function. That one is made for more control. Okay, which I will talk about. So model model is set to eval mode. It means we want to just test the model and we want to write what is the overall probability and we want to save all labels as well. All labels and all probability we are going to save. Okay, that we are going to get from the model. Let's save those. Okay.
So what we are going to do from the loader we are going to load the batch.
Okay. That is for the validation we are going to load each batch. We are going to pass validation loader here. Extract the batch from here and then pass that batch input. Extract first the batch input. So batch input id we will extract input ids we will send it to device.
Okay send it to device and this one is input ids.
Same thing we will do but we need to extract the attention. So batch we will have to extract the attention mask.
we will convert to this one and we will save it into retention mask.
So this is the input ID and attention mask for the whole patch. Okay. And we are going to compute for the whole match in parallel. So we are going to pass input ID.
Okay. We need to pass attention mask.
Okay. And then done. And then whatever the output we will get we will convert to float at the end. Okay. Float will give us logits. Logits means the actual probability because this model has that linear layer. It means the final output that we are getting is probability between 0 to 1. Once we have it we can just use and append all the probability.
So all probability dot append. What we want to append? We want to append. This is logit because between 0 to one but this has not passed. This is logic.
Okay. It is not just 0 and one. The model learning happens in zero number.
But evaluation does not have sigmoid. So whatever the probability is the answer can be 100. Answer can be 200. Can be negative -200. We need to make it probability. How to make it to probability? We will use torch dot sigmoid.
We will use torch dots sigmoigmoid. Pass the logits.
Okay. And then once this is done, we need to convert to CPU and then convert to numpy.
Okay, this part is completed. All probabilities. What about labels?
Now let's compute the labels. All labels dot append.
Where do we have labels? The actual answers. So batch and their actual labels that is part of the actual data loader that we have created. we are converting back to numpy. So now we have the probability as well as the label and now we can do the actual evaluation. So model processing is done. Data saving from the actual model is done. Now it's time for the evaluation. How do we do the evaluation? We can check the probability. How do we check the probability? We can just do np dot concatenate.
Okay, we want to concatenate all the probability because in totis is equals to zero. This is all the probability and then next is concatenate all the labels because all of them are different different right for each for loop there will be 16 uh dimension then next one will be 16 dimension next one will be 16 we need to concatenate all of them into a single thing okay into a single dimension that is why we are concatenating all probabilities as well as all labels. So this one will be labels and what is actual prediction? We need to define prediction based on the threshold that we are creating. Our predictions are present inside a prediction are present inside probability. This is the model prediction. If probability is greater than or equals to the threshold, we will save the model prediction.
Okay. So now we have the actual label.
Now we have the actual prediction from the probability we created the uh prediction this threshold we can tune it later if you want but this one will be a prediction we will convert that because it will be in float we will convert that to s type of integer okay now this is the model evaluation we have all the prediction probability and labels what we can do finally we can use the evaluation metric to check precision recall F1 okay so I will say out F1's that is F1 score I will save out as a dictionary I will talk about what is this out and then all the F1 score I'm saving it as F_sub_1 so for all the validation for the validation I'm saving all the F1s is here okay now let's calculate it for I comma name in enumerate of what the actual data so we want to enumerate Let me write is AI is layoff. Let me copy it because writing too much.
Okay. So we are enumerating over is AI and is layoff. So first we will calculate what is the precision recall for AI. Then we will calculate the precision uh and precision recall for is layoff. So that's why we are doing this thing one by one. What is the precision of each? Okay. Now what we will do? We will calculate the precision that we imported from the skarn.
Precision we need to pass the actual Y and the predicted Y. Actual Y is present inside labels.
Okay, actual is present inside the label. But label of what? Label of is AI will be present at index zero. Label of is layoff will be present at the index of one. Right? So we need to change the index. How can we change the index?
Using the enumerate. If it is about AI we will use I we will use zero. If layoff this will automatically become one. We will get one here. The index will be changing. That means all row and that index same thing will happen for prediction.
Same thing will happen for prediction.
And for prediction we will write this.
Okay. And because there can be a zero division error if there exist we don't want to have any error. We just want answer of zero. Okay.
Okay. This is precision score. So precision will give the answer. You want to convert back to float and then get this as precision.
So this is precision. Same thing we will do for now we just need to change one statement. So this one is not precision score but the recall score and this one is nothing but f_sub_1 score. But f_sub_1 score f_sub_1 score we'll use yeah same thing okay we can calculate the f1 score from here so f_sub_1 score so this is precision recall and f okay what we can do we can just save this because we are iterating over it for ai for a layoff we can save it we can save it in this dictionary so what I am doing we will save the precision recall when it is output We are saving it under the name of as precision. Okay. Then just wait. Let me write these things down.
I'm saving precision the recall and F1 whatever we are getting from P comma R comma sorry about that R comma F. So P will be saved here. R will be saved here. F1 will be saved here. But this one belongs to what? This one belongs to name. Okay. So it will look systematic here. And then whatever F_sub_1 we are getting. Okay. Whatever f we are getting we are I want to append it to f1s. So f_sub_1's dot append I want to append f. So what I want to do at the end for validation I will I want to check what is the overall f_sub_1 not for is ai not for layoff what is my f_sub_1 including both is AI and is layoff. So it means we have to do some kind of aggregation. Aggregation means main. Okay we'll be selecting mean for that. So here what I will do I will create a new in this because this is a dictionary I will use this dictionary provide I will call that as macro f_sub_1 f1 is already a harmonic precision of mean and recall but when we talk about macro f1 it will be a harmonic mean of both is and is layoff precision recall okay that is where we are getting so float np dot mean of f_sub_1 of Yes.
And then once this is done finally completed and then we will just return the full out because out contains my all information. And it was long but this is the same exact thing which we are going to use in the second notebook. So we don't have to worry about it because it just use model as well as the loader to predict everything. So the notebook to work is has become much easier.
Okay. Finally, this one is 16. The training loop.
The training loop.
Now, what we will do for training loop, we need to define few things. We need to select the best model in each loop. And for that, we are going to select the F1 score. Whichever model is going to have the best score, we are going to select their best F1. So at the start I will be selecting best f_sub_1 is equals to minus 1.0.
Okay, we are going to update it later.
This one is markdown. It should be python. And then best state because we need to save the model. I will be saving the model and best epoch.
Which one is the best epoch? Okay. So here to track it for all of them is just some static values. Once this is done, now what we will do for epoch in range of total number of epochs.
Total number of epochs.
Okay. Number epoch. Is it number epoch?
There is lot of code here. Yes, number epoch. Okay. For epoch in range of number epoch, we need to check how much time it is taking for me to complete this whole thing. So I will say time zero.
And within this time we will start we will set the model to model.rain.
Okay. Our running loss what is our current learning loss in this current particular epoch. So running loss at the start will be zero. Now we will iterate over batches for step batch and enumerate of train loader of train loader. We need to extract few things from the batch. Same thing that we did before. So let me copy where we were having this batch information.
Okay.
So batch input ID is two device and also two device also has something called as non-blocking which helps if you're doing the parallel uh jobs.
Okay. So I'm going to write this. You can choose not to write this. It will work. Exactly fine. Okay. So we are extracting input ids. We are extracting batch marks. This is input ID is attention mask and we also need label.
So where do we have label? Okay. Label is not here. It means let me copy this whole thing because we need to extract the label. So we can pass it to this function for the evaluation. Okay. So where is label? Label is labels. Labels.2 device non-blocking is equals to true. So here we will get labels.
So input ids, attention mask as well as labels. Now we will use context manager torch dot automix precision dot autocast.
Okay, as I mentioned where do we use this autocast? We use this BF16 matrix multiplication linear layer as well as attention. Okay, and we'll be using it for the model forward propagation forward pass. we'll use BF16 and for the loss calculation we will pass it inside the autocast autocast will handle in BF32. So whatever we want to handle by autocast we will write inside this context manager. So device type is equals to device dot type whatever the device type is. What is the data type? Data type is equals to torch dot BF16.
Okay. Even we are writing to BF16.
Autocast will handle the parts where it requires BF32. Okay. Okay. Yes. We want to use AM. This one we have created in the step number 14 that is this one.
Okay. So use enables etc. And now inside this context manager we will do the model training. So in the model we will pass input ID and attention mask and it will generate us logits. So let's part input attention mask and it will generate us logits. Now what we should do with the logits? Logits will be passed to the loss function.
So we will write loss function. Loss function will require logits will require logits. It will require the actual labels.
Okay. And binary cross entropy will binary cross entropy with with logit loss which is nothing but this one will handle the sigmoid internally. So it will calculate the error from this.
Okay. Once we have the error, it means we need to do backward propagation. So loss dot backward.
We need to take a step.
We need to update the weights. Optimizer dot step. So update the weights. Then scheduleuler needs to change. Sheduler needs to update. Okay.
We need to clean the gradients. Okay. So optimizer dot zero grad optimizer.0 grad and finally we need to update what is our current loss. So running loss overall loss okay plus equals to whatever the loss we have exactly that is loss dot item.
Once this is done it will run for all the batches. Then what we will do? We will just check what is the average loss in each uh inside epoch in for each batch. What is the average loss? Okay, running loss. So what is the average loss? How do we create average loss?
Because running loss is having all these items different values. We will be creating average loss from that. So running loss is having single number but based on the length we will divide it. Okay. Whatever total length we get, we are going to divide that with length of train loader. Okay, length of train loader. This is nothing but a value. Okay, running loss value, scalar value, we are going to add keep on adding the value. And then finally just for the average. This might not be the best average but uh this is just tell us that this is the average overall. Okay.
Now once we have the average what we can do we can use evaluate loader how it is performing on the validation set. So we need to pass the model we need to pass the val loader here and we need to pass the threshold which we want to select threshold is threshold and this will give us validation. So we have the train average loss in that particular batch or in all the epochs. Okay. So this is epoch one epoch completed. the one epoch loss and the validation on that epoch for the validation loss and then finally it's completed. So time dot time minus t 0 will give us dd.
Okay. So this was t0 I can make it as t1. Okay let's write it as t1 time 1.
And then let's print the information about this loss. Okay, information about this epoch. So what are the informations? I already have all the information written. So let me copy those components so we can be faster. So what we are printing? We are printing what is the epoch? We are printing one because I don't want epoch. We have selected 8. It will start from 1 and end at 7. But we want to see 1 to 8. That's why I'm writing epoch + one. Out of total number of epochs. What is the average loss? What is the validation macro? Here you will get micro F1. Here you will get what is the validation of is AI F_sub_1. What is the validation we will get for is AI layoff 1. Okay. And we already have this value is layoff F1 is something that we created here. Is layoff_p if layoff R if layoff F1. So this particular key is already present there. This will print for each epoch what is the status. Okay. And once this is done, what we can do next is we can just print the updation formula.
So for this epoch, if the validation of macro f_sub_1, okay, validation of macro f_sub_1 is greater than is greater than best val f_sub_1 which is for the first time will be true. If it is greater then what we need to save? We need to save the model.
We need to save the best states best epoch as well. So best epoch will be the current epoch. So best epoch is equals to epoch + 1. And then we need to save the best state. Best state we can save in a dictionary. Okay.
And we can save what is the k value? K is v dot detach. Okay. But how do we get this k and v? Let me write for k, v in model dotstate dict.
Okay, key and value pair of those model dotstate dict dot items.
We will iterate over it. So each key has its own value. So we are going to save all of them. So we will save k, v dot detach. We want to remove it from the GPU, bring it back to the CPU. So detach dot CPU otherwise our GPU our RAM will be filled. Okay. Dot clone. So we are keep on saving this thing in our RAM.
Okay. We will update it that is best best state and then we will print what is the precision recall. I can save it here.
Okay. So best new best macro F1 is saved. Whatever the number is snapshot kept in the memory it will be in our RAM. Okay. And once this is done, the final thing is we will just say training done.
Training done. And I can write in the f string what was the best value where the training is done. Okay. Best well macro f_sub_1 is equals to is equals to best well f1.
But do we want all the precision? No, I only want a precision of four. And at what epoch we got best? At epoch. Best epoch.
Okay, finally done. It will take some time for me to complete this. So, let me run it.
Okay, it will not take time because there is an error.
Enum. Okay, so it should be enumerate. N U M E N U M E R All right. What else? What an expected enables? Where do we have enables here? Okay, this one should be enabled.
Okay, let's see how much time it is taking.
All right. So training is started and there was one very big issue in my training pipeline. My system was getting out of memory. I was trying everything.
I tried this in anti-gravity. I tried this in my original file. But in my original file, it was working. What was the issue? I tried and compared everything. Everything was perfect.
Restarting the system also was not of any help. Then I check what was the issue. Okay. So if you're able to see here, let me search for the term self.kenizer that we have previously used. I have mentioned I have written the spelling of length incorrectly.
Previously if you check the recording it will be max l a n g t h that is a wrong spelling of maximum length. It means if I provide any wrong argument that means the original argument has to use the by default value. I have set max length is equals to 512 but the default one that is the correct spelling of max leth was set to 8192 that is a huge memory update that it has to do right now I'm using 512 my system can handle it but uh that 8192 my system was not able to handle It was filling all the things. I was not even able to do one iteration of training. But now it is length. Now it is set to 512. And we can see the training is now ongoing. Okay. So it is taking around 22 or 23 second in my system to complete one training. As we able to see epoch 8x8 the training is done. Best validation micro F1. We are able to get 0.9778.
That is really great. And we are able to get this at epoch 7. So you are able to see at every epoch this is micro F1 that is harmonic precision mean and recall for both is AI and is layoff and because every time it is updating it is writing a snapshot kept in memory. Snapshot kept in memory that is present here that I've written here. Okay. So this means that the new macro F1 and I'm also writing this particular part. Training done best macro validation F1 we are printing at every epoch. That is what we are able to see here at the end. Okay, that is what we are able to get at epoch 7. And if you want to check more the this loss, this is the average loss per you are able to see that at iteration one we started with 0.6214 and we ended up at 0.0031 which is pretty great. Okay, next thing once the model is done it is still in RAM. We need to save this model in our system.
So let's go ahead and try to write the final part of this notebook that is save the best model. All right, let's try to save the best model. So let me write 17 save the best model.
Okay, so what we going to do? We are going to save the model by creating a new folder location here. Inside models we're going to create a new folder model and inside that we're going to create notebook and we there we are going to save our PT model. So what we will do we already have our model output directory.
If I run it we will see that this is the output directory that we want but it does not exist yet. So what we want we going to select this we're going to write mkdir and we going to say parents is equals to true and if it is exist don't do anything. If it does not exist then create the directory. Right now it does not exist. So if I run this particular cell you will see models is created as well as node is created. And if I run it again it will not create because it automatically it is existing.
So it does not have to create a new directory. Next thing is using the torch we are going to use torch. Let me have some space here. A new line. Torch dots save. We are going to save what? We are going to save the best. And best is present inside the best state. So let's save the best state. Let me move this here. Best state. And then what we're going to do, we are going to save this in model output directory. And inside that directory and inside that directory, we need to write what should be our model name. I'm writing model dot pt. Okay. Once this is done, we also need to save the tokenizer. So we are also going to save the tokenizer but tokenizer is present inside the tokenizer and we want to save the pre-trained because that is what we are going to load. So we are also saving the tokenizer and in the model output directory. Okay, what else? We can also write few things like the configuration.
Whatever configuration we have set for the model, we can also save in the same file that will be helpful to load in the evaluation. What was the threshold? what was the maximum length what is the model name what is the epoch etc we can save it there okay so what I'm doing model output directory in that I want to save inside config dojson and I want to write txt so write as a txt file okay write as a txt content whatever I'm writing but I will be writing things in JSON so JSON dot dumps and let me write this is the JSON All right. And in this JSON what I want for each component I want to have a indentation of two. Okay. So this is about this JSON path and we are going to have indentation of two inside there.
And let's write the components. First thing is what is the model name?
Our model name is model name.
All right. What is max length?
Not this one. But a short version of this one will be max length.
All right. Let me Okay. This time we will make the spelling correct because it was creating lot of issue. And threshold threshold will be threshold and then best well F_sub_1.
Best well F1 is basically best F best well F1 and then epoch. Epoch is going to be same whatever the epoch is but it will be best epoch not any epoch. Okay. So these are the informations that we are trying to save inside our config do.json which we are writing the text. Okay, this is not a txt um I mispronounced it.
It is a write text. We want to write what? Want to write all this information but we are writing with the help of JSON because JSON using JSON we can have the indentation. Okay. Once this part is also done, we can just write print print saved to all right and then we can write what is the model output directory where we are saving all this information and that's all. Okay. So let's run this the final part of today's uh final part of this notebook that is a half part of today's class. Okay. So all of these things is saved. Let's check. So we have our models notebook and inside it we have three things. We have tokenizer, we have tokenizer config.json this is model output directory and uh we have config.json we have model.pt.
Okay. So we have saved all the things and whenever we save model.pin as we can see it is also saving the tokenizer here. Okay. These are all the information about the tokenizer that we don't have to worry about. But class token pad September these are all the tokens not September SCP and these are all the models this is model.pt file config.json that whatever values we have saved is present here as well and this completes our first part of this training notebook that is trainipy nb.
Now that you are done with the training pipeline, now let's create the second notebook for this particular lecture where we'll be doing the inference as well as the evaluation of this model. So what I'm going to do, I am going to create a new notebook called as 02. Let me write 02 inference and eval sorry the name I have written 01 it should be 02. And we are going to do few things. First we're going to again do the import. This one will be simple because we already have the model ready.
Majority of our coding is already done.
So we will start with import.
We're going to select this. Now let me we don't have to minimize this part because this is important and then we will require few of the imports. We using few imports. We are using import JSON same as before. Counter same as before. path lip tors numpy everything same as before but uh this time we have little less things okay to do version torch.ca dot is available we able to see that I'm using torch 2.1.3 2.13.0 plus kuda 12.6 Six good is available.
Yes. Second is we again need to have that path. So what I'm going to do for everything that we have before I'm going to copy so that we have to write less things. Uh seed we we don't have to set seed because we are not doing any random calculations here and all the configurations is already present in our training file. So what we have to do is I think it is at the very start to part is done. Where are we loading the data? Yeah, this information I needed. Okay, so second part is paths and setting the device.
So let me run that. Let me copy all the things and okay. So here we have the current here that is our current directory where and we are going to target the parent. Once the parent is targeted now we don't have to worry about train JSON L. What we are here for is test JSON L. Okay. And it is present inside test and the name is test seed dot jsonl. Okay. Model output directory.
There is no model directory but we have model directory which we are going to utilize and model directory is present inside repo inside models inside notebook. Okay. So this part is clear.
Model directory we don't require anything else. So let me remove it. If you want we can print it but okay let's print test data. Our test data is present inside this one. Does our test data exist? Let's check it. Repo is same as repo. We don't require any output.
But what we can do? We can set the GPU part. How do we set the model GPU? We set the model GPU using this command that is device. So we set our device.
Our device is also set. We can copy this device all the print statement. Now we'll be reusing lot of codes. Okay. So repo is topic guard. Inside topic guard we have test test.json. Does it exist?
Yes, it exist. Device is coda. The next part is the model class. We have to reuse the same model class because for this model class we have to load the state d. We have to load its parameters.
So what we will do? We will again copy all the model class that we have. We only have one model class. So I'm going to copy that. Where is our model class?
This is data set. We have lot we have done lot of coding today. This is two head classifier. This is what I will copy and paste. And let me check what things super init require. We require encoder. We require dropout. Dropout is not really required for the training.
But whenever we use modelevel this particular part will not even run self.ropout. Okay. So it is only it only works when the model is set to model. So even if we write it, it does not matter.
All the things are same as it is.
Self.enccoder model has to pass it. And yes, so we will copy and paste the whole thing here because on this object we need to load the saved model. Let me copy what is going on. So let me copy the whole thing.
Paste the model. We don't require. Okay.
Whatever it is, let's run it. Once this is part this part is done then it's time for load the saved model.
How do we load a saved model? First thing is we need to load the model metadata. Where do we have model metadata? Model data metadata is present inside the JSON that we have. So I'm using JSON do.load load and the path I will provide path is model diir/ this is markdown that's why I'm not getting autocompetition and then config JSON inside the model py model/n notebook py we have config which has the information about the model name okay so we are going to select the model from there so what we will do we want to read the information from the JSON so read text once this is done then what we will do we will just print the JSON okay JSON dod dump so we can read it JSON dot dump we can already read it but just for the sake of writing here I'm writing that is indent is equals to two okay and what we have to do we now have to load the tokenizer where do we have our tokenizer well what I can do for this part let me copy the model. Let me paste this part of the model to head classifier from where we will get the model name. We will get the model name from the matter and from the matter we have inside model name model name.
All right. And uh dropout will proper does not matter. We are doing the model in the evalation mode. But this particular part is important. It is to device. Okay. We also require auto tokenizer. So let's look at the code of auto tokenizer.
Where do we have auto? Not this one. Not this one. Yes. So we want the tokenizer that we have saved before the model. We can do that after the model as well because both are two independent things.
Tokenizer from pre-trained and we want it from the model diir. Okay.
It will automatically look for the file of tokenizer config because when we save this, we did not save with the file name. It automatically did. So when we are going to load from the model directory, it is automatically going to pick the tokenizer_config.json.
So we have the tokenizer, we have the model and now in this model we need to fill the weights. We need to save the weights. So what we will do? Model dot load state dict.
Okay. And then what we have to do using the torch we will load the data torch dot load load from where we need to load from model diir/ the model file name is model.
Okay. Where do we need to map this? We need to map this device. Map location is going to be device that is nothing but our cuda.
All right. And then we want to only loads the weights. weights only is equals to true.
All right. So this is model state lick.
So model uh information will be initialized inside that will be loaded inside that and we need to set the model to eval mode. So drop out and other things will not be applied. It is only used for inference. Okay. So this is model. Something is wrong. JSON object must be a string. So what is going on?
Why it is not a string? model diir/config.json.
So this looks okay.
So we don't have to okay read text comes from here. Okay, this is read text and it will go inside there. Okay, we are able to read the model. Model will be loaded here and we able to see that model is loaded. Okay, this is all the model information that we have. We don't have to read it. But when we set the model to model eval, model.e shows us this output directly to head classifier.
Okay. So our model is also ready. What we can do because whatever configuration we have set like the maximum like the threshold is already present here. So we can load the threshold, we can load the max length also because that is required for the tokenizer. So what I will do max length is equals to meta of max length.
Okay. And we also need threshold. Threshold is present in matter uh get we can do this because if threshold is not present we can load a by default value. But if threshold is present just load the threshold threshold comma 0.5 and threshold is also ready. Max length and threshold is also ready. We can print it. Print max length as well as threshold. So max length is 512. Threshold is 0.5 which is correct which is compatible with our model. Now let's do a single prediction.
Okay. Now let's see what model predicts.
Single prediction.
All right. How do we do signal prediction? First, we need to create a predict function. How do we create a predict function? If I talk about the prediction, we can copy few things here from the inference part that we created before the evaluation part. So, let's go there. In evaluation, we were doing the tokenization etc. But I think this is overkill. So, what we can do? We can create our own for now. def predict.
All right. printed to the prediction and we have done similar kind of prediction at the start when we were working with when we were printing about what the tokens looks like like this particular part. Okay. So what we need to do we need to pass the example to the token and we need to give more examples here.
Okay. So let's use this part here. So def prediction we will be passing text. We will have some threshold by default if user will not pass it.
I'm going to set it as threshold.
All right. And then we are going to have this tokenizer. But this tokenizer this time will be having few more things like max length, truncation etc. which we have defined here.
If I search for truncation, you will see this particular part. And these are the important things that we need to paste it here.
All right. So truncation is true. Max length padding return tensors everything is set. Now this completes this part.
Okay. Tokenizer also we need to send if you want to process it faster we can send it to device. Okay. So tokenizer is also present here. So this is finally completed. Okay. Next thing is we have the model. We will use the model that we have. model is already initialized in above places. So we will use the model and then in the model we need to pass and c. Okay, same thing that we did before we need to pass and c input ids and then input attention mask. So in the model first arguments is input ID.
Second is c attention mask. All right.
And then this will give us output as logits.
Once we have the logits then we need to use the sigmoid. We can again look for sigmoid but sigmoid output sigmoid coding is pretty small so we don't have to go to that notebook the arch dot sigmoid and we need to pass the logits okay and we extract the dimension from it the inner part of it then convert it to float then we will pass to CPU then we will convert to two list so we can see it as a Python data type Python native data type and this will give us scores so we will get a score of both a score of AI a score of layoff and if that particular text belongs to both then it both of them will have higher accuracy and then finally what we can do we can return this particular thing which I'm writing okay to save our time return so what I want to return if it is is AI the score of zero because is AI is present in index zero is layoff is present at index one so if a score of zero is greater than threshold it will say true 1 0 okay It will not say true.
It will say 1 or zero because it is integer. If it is false, it will say zero. If it is say true, it will be one.
So is AI is layoff is AI score. What is the score? Is layoff score? What is the actual score? Without the threshold part and what was the threshold? This is what threshold that we have set. Okay. And we want to do this with that decorator without the context manager. Either we write context manager or we write the decorator dot dot no.
Okay, now let's do this. Something is wrong. Logit's unexpected indent. Where do we have this? Oh, all of this is a part of here.
Okay, now it looks good.
All right. Now, what we will do, we will try some example. What I can do here is let me go to this notebook.
Uh this is this contains some information as well. But we can make it better. Okay, let's try this. I want to call this predict function. And in this one, I want to write some text. Okay.
Uh let's write that learning AI from Kish Nak YouTube Academy. Okay. Krishna YouTube plus Krishna Academy.
It does not care about uh these terms but it only cares about learning AI. So let's see what it shows us. Self is not defined. Where do we require self? Where do we require self? Let me check those predict text. We are predicting the text and this will give us the answer in P.
Okay. So let me write that part is pretty okay. But what else we can get from here? All right. Where is the error? Predict self is not defined.
Tokenizer. Where is the tokenizer? Let's fix the tokenizer if there. Okay.
Tokenizer expects text truncation is true. This is correct. Max length. Oh, this is self dot max length which is not correct. We already have max length.
Okay. Max length is max length. Padding is true. Okay.
I literally copy and pasted that and everything else looks correct here.
Okay, if you want you can make it look like this.
Now let's run it.
What is the prediction? The scores is not defined. Now where do we have scores? This is the score. So it should be scores. Now let's run it. Okay, so we are getting are we talking about AI?
Yes, we are talking about AI is layoff zero. What was the score of is AI? What is the probability? It is 83. It surpasses our threshold of 0.5. And this one is very low 0.03. That's why it is not there. Threshold is 0.5. Now you can use this function to create your own fast API endpoints and can deploy it.
Now let's try a few more functions.
Let's try that when I used AI.
In my daily work, my manager was happy that that I am able to convert my sales faster than before.
Great productivity.
All right. Now again this one is 0.99.
This one is again that but here we're talking about AI. Here we're talking about other things. But if we choose some other context it will be like that.
Now let me choose AI is killing jobs. Now we able to see we talking about AI we're talking about layoff that's why both of them have a higher number. Okay so it means our back end is working. Now what we will do we will just do some examples test out some examples which I have already written.
So I will literally do copy and paste this now on point number six. try a few canned examples. It means those are already created. We will just check if they work or not. So here is the examples AI release news. This is what not I'm going to process. This is I will be printing that and then I will be passing this to the model and then see if we are getting the same data or not of or we are getting the same answer or not. Okay. So for this what I'm going to do I'm going to iterate over this example. Let's iterate over example. So first thing is the label itself. So for label second is the text right. So for label comma text and examples that we have here examples capital. So label comma text and examples that we have here we are going to pass this to predict. We are going to pass the text and we are going to get the answer in P.
Okay. Now what we can do? We can just print what was the actual label. The actual label is F. What is the actual text?
Actual text is the text.
Okay. What is the actual label says?
Actual label is this is the text.
This is the label. All right. And how can we get the label? Just the label and then print the actual prediction.
Okay. Do we need to predict all? And you can predict all if you want. So prediction.
So the prediction is we can just write the whole P or we can write the prediction of is AI prediction of is layoff etc. If you want to see that we can just write this part here. So we can see what is AI what is the probability of is AI what is the probability of is AI score. literally all the things that we get from here. I'm just printing it 1 2 3 4 all the fours 1 2 3 and four. Now if I print it for each example you will see this and I think it will be better if I write a separation line or print a separation line. So print equals multiplied by 20.
Let's see this now. So for this particular example deepse released the AI released news that was a type expectation was 1 0 what we are getting 1 and zero. That is correct. For this one, we are expecting 01. The answer is 0 1. For this one, we are expecting 101.
Answer is 1 and 1. We able to see almost all the things accurately. This one is 0 and 0. This one is 0 0. This one is again 0 and but this one comes one. So this one is little wrong. This one is about house downsizing. But it is predicting that it belongs to is layoff.
Okay. Yes, we are expecting some error.
It is fine.
because we are not having thousands of samples. We are only having 100 132 or something. Right? Now the final thing let's load the frozen test set because we have not not yet evaluated okay on the actual test set. So load the frozen test set. Okay. Now what we going to do just like we have loaded the train data at the very start of the notebook. Okay.
This one we are going to copy and paste this function. Okay, load the JSON and then we will load the JSON just like before.
But this time we will load the train. So here I will be passing test JSON L.
Okay. And this will give us test rows.
So loaded length of test rows. How many test rows we are having? And then write test rows. Okay. So we have loaded 200 test rows here. Now what we will do? we will apply the batching. So we can do the prediction on batches. So 8 dot batch prediction on the test set on the test set. Okay. So for this what we have to do for this we have to create again data set. We already have the data set loader as well as the data set. So let's copy and utilize whatever we can from here. This is the data set class.
Same thing that model will be using for that. So I will be copying and paste and this one is classify data set. It this one I will call it as inference data set. Okay. This one we still require rows tokenizer max length. We still require this particular part. We still require this. But for the tokenizer do we require all that? For tokenizer yes we require all the things here. Max length, max PT, etc. Input ids, attention mask and the labels are also here. Do we require labels? No. we already have the label. So we don't have to print this particular part. Okay. So I'm going to remove the labels.
So only input ID and attention mask because labels is already present in the data set. Okay. So let me run this particular part and then create a object. So what I'm going to do in the inference data set I will be passing my test rows.
What else? Test rows I'm passing. Second I require is tokenizer.
Okay. I'll pass in the tokenizer and I will be passing max length. Okay, this one will give me test ds. Okay, test data set. Then for the data loader, I can copy the exact data loader that we have created. Where is our data loader? This is our data loader. So, let me copy this part data loader and then this is the data loader where I required to pass test data set batch size. I can pass the batch size 32 directly that was present in the training dot notebook that we have created. Number of notebook is this shuffle. Do we require shuffle? We are not training. So we don't require shuffle. Do we need to pin the memory?
No, we don't have to pin the memory. We don't require any stabilization. It is only few data set. This time we created test loader. All right. So now I have cach data set test loader. Something is not working. Now it will work. Okay. Our data set loader is also created. So we have test data set. We have test loader.
Now it's time to check all the probabilities. Okay. What is the probability etc. We have to save it. Let me write all the probabilities. Okay.
All probabilities I will be saving it here. And I will set the model to eval mode. Model is set to eval mode. Then using the context manager with tors.t nograd with tors dot nograd then I have to get rid of our batch for each batch from the test data loader for batch in test loader we will load batch by batch but for the batch we will pass to the model. So again let's again load that part where we were using the model.
Where is that? Okay, this part.
Okay, copy this whole thing. So for the batch, we want input ids. We want to convert to device. Uh change it to device. Then batch attention mask to device. This will give us attention mask. Then we need to do the inference.
How do we do the inference? Model all logistics. Same code. Just copy and paste the same code. We are passing the model input ID attention mass dot float.
We will get the logits. Then we need to append all the probability again append all probabilities loss dos sigmoid logit cpu fitton numpy exactly same thing we have to do here so all probability but this time it is all probabilities so I'm going to write all probabilities and this part is done now what we have to do np concatenate because for each batch all probability will look like this this is batch one this is batch two this is batch three it is still not numbers so we need to do concatenate for the concatenation we already have this concatenate this particular part but once the context manager is done. Okay.
So probabilities we want to concatenate all the probabilities and then for the prediction again copy for the prediction these are the values prediction probability greater than threshold and threshold we already have it. Okay threshold dot as end type. So we have the threshold as well and from this test rows we need to extract the actual labels. Okay, just like we did before here we are selecting the labels but we have to extract the labels from the test. Are we doing that already? I don't think so that we are doing that.
Okay. So let me write those part where are the actual labels? This time we are not getting labels from the model. So what we are doing np dot array inside it we want to get a list of 0 or one. Where is zero or one? 0 or one is present inside r of is ai and it is present inside the next one. R of is layoff.
Okay. I think I have extra dimension here. Okay. Now it looks good. So inside this bracket I have this part. Let me double check this. Okay, I have this.
This is a part of one bracket. And then because it is a list comprehension then I have to write for R and test rows. Now I have the full label. What data type do I like here? I need D type is equals to end. Okay. So now I have labels as well. If you want we can print all the things. What is the shape of prediction label etc. So probability prediction labels if you print it we are able to see all of them are 200 it means everything is going correctly. Now the final thing the final thing of this whole video that is doing the evaluation and see the actual matrix. Let me write global matrix.
Global matrix.
All right. So what we have to do now we need to understand what is a precision recall f_sub_1 macro f1 etc. Again this is something that we have already done before in the compute matrix that is this particular part. So all we have to do is just copy the whole thing and then write it out there but we need to create a function. So I will write df compute matrix. Okay compute matrix we are going to accept wide rule. We going to accept y prediction and then copy and paste the whole thing. We want output uh out f_sub1 out is nothing but a dictionary.
F1 is this part. We are going to enumerate overi. We going to calculate the whole thing for ai. Calculate the whole thing for is layoff. And then we are going to return this part. Okay. But instead of labels we are going to have y true y true y true. We are going to have y prediction. Y prediction y prediction all of them is same. Okay. So if you want we can write like this name etc. So let's write it and then pass the actual data. So compute matrix. Let's pass the compute matrix. Where do we have the answer? Our answer is present inside the label.
All right. And the second one is inside the prediction.
And we will get M. I'm just writing M as matrix. If you print M, this is what we are getting. What you can do? You can iterate over is AI is layoff. But this is the what the precision and recall we are getting. It is already great. We are getting a precision in the test set. We are getting a precision of 0.93.
We are getting a recall of 0.93. We are getting a F1 of 0.93. Is it correct? Let me check. All of this looks too good here. Right. So we are getting Y prediction Y prediction Y prediction F1 score is also there. I think we are getting a good number here and we can also calculate the accuracy because we have accuracy here. So let me write accuracy is equals to let me copy the whole thing and instead of this I will use accuracy but we need to import the accuracy score as well.
All right for accuracy we don't have to do the zero division and this part is done. So where are we importing F1 score is where we will import the accuracy score which is already imported. Okay.
Now let's run it. Is AI is layoff and then see the answer labels as well as the prediction.
All right. So everything looks great.
Next thing is final thing is we just need to add the accuracy here. So let me add the accuracy one last time. This one is ACC. So this will be accuracy. F1 is already appended here. So macro F1. Now let's run it again and see what is the accuracy score that we are getting.
Okay.
All right. So I double checked everything and everything looks great.
It is totally by chance that we are getting this 93 in all of them. But in for is layoff we are getting all the different numbers. This is both again 91. This is 94. This is 88. So macro F1 came to be like 92 percentage that is harmonic mean and precision recall of both is AI and layoff and this completes our both inference training evaluation all the metrics that we require all the suits that we require to complete this particular problem statement. So we started with all the way from talking about what this project is what are the prerequisite what we will build we will build this notebook that is train model.pt PT we will get from the out as a output we will get config as a output then we'll perform the testing and the testing we have done inference as well as model evaluation and that came out to be pretty good for this small sample size and as mentioned this is not what we have decided earlier we're not going to do any deployment we are not going to do any API we are not going to do any object-oriented programming but if you want to take this if you want to take this particular problem statement if you want to take this solution to the next level what you can do I give you a direction for this one. So let me talk about the future steps of this the future steps. So what you can do first you can convert this notebook to modular coding okay to modular coding. Make sure that you're saving the model in some model folder loading it from main. py okay and doing the inference. You can talk about different inference from the main. py using the argument. Okay. You can talk about that if I talk about main. py and do - batch is equals to 32. It should run with batch 32. If I do main py and directly write the text here, it can work as a standalone script as well.
This is one way. What you can do, you can improve this further. You can take this to the next step. You can create a API for it using fast API.
Okay. And serve your model and serve your model as an API.
Okay. So anyone front end engineer stream it also can use this API. So you can create /model predict or something where you can pass the text in the body and then from that text you can do the actual prediction. So once you are able to serve the model I will suggest you to keep the model serving different from the streaml. So start this API server but create a streaml application differently.
So in the streaml whenever someone will click on the predict button okay whenever someone will create click sorry about that whenever someone will click on the predict button that is /model.predict you are going to call okay and you're going to pass the text and it is going to get a reply. So you are creating two different systems here.
One is streamllet. So streamllet is not useful here to load the model. We will load it separately and we are going to create streamlit only for the UI purpose. Once you have it, you can deploy this anywhere or you can just create a single streamlate with all the fast API components within it which will be easier for deploy. So there are a lot of things which you can do and you can take this project to the next level. But the the main agenda of this particular project was never to create a deployment but to understand if I want to create a transformer my custom transformer with my own data set how I'm going to do that. And that's all for this video and I hope you like it. Please uh check the link in the comment section or in the description where you will find the GitHub URL and the other things which I've mentioned. I'm going to push this all the files not the model model I will try to push it in my Google drive only if I have some space but for sure I will move it I will move it to the matter I will move it to the drive provide the link of those as well so you guys can download it and can do the inference and create lot of amazing things from it thanks for watching I will see you in the next
Related Videos
12:08
Artificial Intelligence
Expanding Stikbot thumbnails
leopoldshorts
2K views•2023-09-24
32:31
Artificial Intelligence
Digital Discrimination: Cognitive Bias in Machine Learning
redmonktechevents2974
4K views•2019-12-18
1:22:05
Artificial Intelligence
Evolutionary Approach to Clustering by Ujjwal Maulik
ICTStalks
279 views•2019-06-26
55:30
Artificial Intelligence
Rose Yu "Learning from Large-Scale Spatiotemporal Data"
networkscienceinstitute
2K views•2019-03-04
28:24
Artificial Intelligence
Stanford Seminar - Generalization through Task Representations with Foundation Models
stanfordonline
4K views•2025-07-14
8:29
Artificial Intelligence
Satellite-Based Wheat Yield Forecasting using GEE & Transformer Neural Network
gisrsinstitute
634 views•2025-06-15
32:06
Artificial Intelligence
Paradigm Shifts in Data Processing for the Generative AI Era: Robert Nishihara of Anyscale & Ray.io
GradientFlow
2K views•2025-01-02
39:48
Artificial Intelligence
How to Build Your Own GenAI-Based Knowledge Management System
2150GmbH
360 views•2025-06-03
Trending
13:38
Law
WOW! Judge TURNS THE TABLES on Trump in His OWN $10B LAWSUIT!!!
MeidasTouch
197K views•2026-07-23
13:25
Business
Playstation NO DISC/NO BUY Fight Is Over...
DavidJaffeGames
4K views•2026-07-23
22:40
Business
Steam and Xbox Just Dropped The Hammer On PlayStation
OhNoItsAlexx
9K views•2026-07-23
16:52
Anthropology
Americans Confused in Australia for 17 Minutes Straight
IWrocker
17K views•2026-07-23