Part 2: Linear Model - Data Split Example using train_test_split

[00:00:00]I'm going to demonstrate how to do the train test split approach when you are going to do regression using scikit-learn so this is where the whole machine learning concept comes around whereas the first approach that I ended up discussing doesn't really use any sort of um machine learning techniques so to speak because in this example that I did earlier what I ended up doing is create a model using all of the data so I used all the data related to TV expenditures all the data related to the unit cells split them into X Y variables and just made use of it in the model now when I say split I just put them in their own respective variables like this kind of subsetted all the values the particular values that I want now what I'm going to be doing next in this train test split approach I'm going to make use of scikit-learns object called train test split and I've already used that import statement above in my my notebook and I don't need to rerun that it's still in memory but I do have some documentation links right here that you can review but I'm going to stick with the standard way that the documentation presents how to do how to label your your variables right this is kind of standard when it comes down to using the train test split so what we have here are four variables that are being defined at once you can see the common separation right there so we've got capital x strain capital X test y train y test and then we have the assignment operator and we have this function this function contains a lot of different parameters and we're going to be making use of some of these I'm going to be using I have to indicate the arrays which is going to represent my feature of the model as well as a Target and I'm going to keep it the same type where the features of Television expenses and TV is a target we're trying to predict based off of TV expenditures now there's something called test size and that is basically this is going to accept the flow and it's a proportion of the data in your data frame that's going to go into the test size right so for example if we had a data frame that contains 100 records uh 100 data points then if we say 20 or 0.20 then that means that 20 of the data is going to go to the test right so the 28 observations will go to the test subset and then the remaining would go through the train right so we can play around with that or just leave it as none if we leave it as none that means if we review the documentation further then that means that it's it's set by default 25 so if I don't want to make use of that keyword argument 25 by default will be reserved for the test subset and I remember this approach right here is making use of this tool through scikit-learn where the algorithm will randomly split the data to different variables and we have a train we have x-train x-test by train y test it means that this function is going to return four different things and those things are going to be placed respectively into these variables now what I want to put in first are the arrays that are going to be involved in my model so that means I want to interact with the first one would be the features so I'm going to go ahead and call the feature which is uh I'm going to put double brackets here and put TV and then put values um I'm just going to put that because I don't want any warning signs to happen when it comes down to running some of my models then I'm going to use the other array that is representative of my Target and that would be sales so if you have thought cells if I want to use the test size right it's already by default in the back end set to 0.25 I can play around with it let's say that 50 of the data is going to go into the test subsets and then the remaining one would go into the train we are dealing with 200 observations so we're going to we're dealing with 100 uh test 100 train now if you are following along and want to get the same results as me then you would want to use a keyword argument called random states with the same value so I'm going to put the value of 100. I remember train test split it randomly chooses uh the data points to go into their respective sets trainer test if you don't have this set then you're going to have different values going into these variables and then you might get different results that's not to say that it's incorrect which is not but the whole purpose of random state is for you to be able to reproduce what I am doing here as well if you're following along so I'm going to go ahead now and I'm going to make use of all of these here I'm going to run it and I need the syntax error I'm happy with that and I'm going to look at the shape of the one of the variables so I'll look at this variable now there should have been some values that were unpacked from this function into the respective variables I want to see how this looks right so here are the values that are for the train right now these this value this value this value all these values represent TV expenditures right that are coming from the data frame now if I use X test all right so let me do a print here and a prince here but you know what I want to do a Lin function on this just to kind of get the just to get the counts all right can you clear drain counts and then let's go so there we have we have a hundred for the trade 100 for the test is looking at the training test uh for the X's right you can play around with the Y it's going to be the same values right now once we have it uh split right and these were randomly split into the respective variables I can now use I can instantiate the model so let me go ahead and instantiate a model I'm going to call this reg and I'm going to use the linear model um let me just make sure I spell this correctly and that's going to be based off what I have here so okay linear regression is what I want to put in here you can go ahead and do that here we are so it's not linear model it's linear regression and instead of parentheses here this is how you instantiate the model right so this instantiates the model and now that I've instantiated the model I can now now this variable I'm calling rig has uh these certain tools that this function is capable of handling so I'm not going to refer to linear regression I'm just going to refer to the variable and if I want to use the fit meaning I'm going to go ahead and now create the model using specific data so I'm going to use the train data to create that model you're always going to fit the model with the train right and that's what we have here excuse me and once that is the once that model is created using that data I want to look at these four of that using the extreme data to see how it did right now I'm gonna hold I'm going to go ahead and run this all right and see what it says two positional arguments on the fits oops all right yes of course I did a cut rather than a copy so there we go grab that there we have our this right here represents the R square right so I can right here about there we go now I want to say train so that means what we did when we split it we're creating this model using the trained data and seeing how well it did and it 0.82 8.25 that's it's pretty strong in terms of how well this model is doing so I'm happy with the strength of this model we know that it's closer to 1.82 is closer to one pretty strong now what I'm going to do here is I'm going to use this model that I created right it's it's the model now is that rig variable I'm going to call that rig variable in this cell and I'm going to use a score method but now I'm going to put in the test data right that means I'm feeding the model that I created initially right with the train I'm now feeding this model with unseen data and we're going to look at it's r squared for that okay so now I'm going to run that and I get a 0.77 so on so this is still a even though the r squared is a little less than the r squared with the train we do get a pretty strong r squared for the Unseen data it looks like this model that I created in saying that television expenditures will predict the target which is unit cells right pretty pretty strong model let me go ahead and just make the create a label here we go okay so it looks pretty good uh when I fed it with unseen beta right now this isn't this is an important part in machine learning where after you train it you want to give it new data what I did in the first demonstration is I used all the data and that's it I didn't have a train it didn't have a test and that could be problematic in terms of generalizability generalizability now I could use the attributes to get information related to the model so I can get the y-intercept I can get the coefficient right and these are attributes coming from the reg model that was created so it looks like I'm good to go when it comes down to that I can also do a prediction so if I wanted to predict this I can say rig dot predict and then open up double brackets and put in a value 50 for like 50 000 television expenditure read it and this is what it predicted right now although the values might not be too far from the earlier example I did where I ran all the data rather than have a trained test split um that that's by chance but this would be the approach to take where you first train the model and then feed a new data right now we could visualize this model and I can go ahead and demonstrate how this would look like it it'll be a little lengthy here but I um can go ahead and see how this would look like by creating a plot and now this plot is going to contain the train uh the the X train values right so if I say xtrain.values then here say Red Dot predict and I'm going to just put in the X train values inside right if I run it let's see here oh let's see you got values let's see oh yes I don't want to remove that and I'm gonna have to remove this now too all right there we go all right so um and the reason why I'm removing the dot values is because they're already set in an array format now and what I mean by that is if I do X train let me take a look at it it's already in an array format it's not in a data frame so I can go ahead and exclude that there now I'm going to make this red and then I'll go ahead and layer this with another plot and this is going to be the it's going to be related to the X train again but now it's not the model it's the actual observed values all right this here and then this right there here are the observed values okay and I can take this a step further by placing the predicted values um in this figure so for example if I say X first and then I'll go ahead and put y test and I'm going to run that see how that looks yeah there we go right and it'd probably helped to put some labels in here so for example if I create a label and say that this is the test data and this would be the train beta right oops yeah comma right there and I'll probably need a legend since I'm including labels and there we have it right there I'll do a pl2. show so we can see the train set in the test set and then my predicted model shown right here I won't worry too much about putting you know the title and x and y axis but we know here that just represents TV spending foreign s let's say unit cells right there we go we have our prediction model lined in red and then we have the train and the test subsets shown in blue and orange respectively