Part 4: Linear Model - Data Split Example using Cross-validation

[00:00:01]in this last approach to regression using psychic learn I want to talk about how to use what is called k-fold's cross validation and this is I would say somewhat similar to the test train split in the sense that we have a train set and we have a test set but the way that this works is slightly different and um I want to demonstrate how this works so what ends up happening is we will use when we use the cross valve function and this is something that we have already imported earlier in the in the notebook which is right here this is the function that we are importing we call this an object as well I'm going to go back down there are certain things that we're going to be typing inside here now let me just kind of take a few steps back and you show you that in this code cell I am just instantiating a model I'm calling this model CB for this variable um and now I'm going to just refer to this whenever it asks me to use an estimator and refer to this variable now I have another variable called CV results and we're going to have some results that we'll take a look at look at it in a second but going back to this function right here across about score and you take a look at the arguments we need to indicate an estimator and that would be what I instantiated which is model CV and Then followed by the data that we're going to use for X I can expand this a little bit and if there's going to be some let's give me a Target variable which there will be in our case we'd indicate why and we can indicate other keyword arguments the one particularly we can pay attention to is CV right we don't have to pay attention to every single one of them but CB is something that we will interact with in CV is the number of partitions that you want to uh or the number of folds that you want to have and the way that this is best understood is to this diagram within the scikit-learn library documentation and here we have an example of a five-fold cross-validation and what's going on here in this diagram is that let's say we have a data set and we have a data frame and we're going to eventually split it where we have trading data and then we have tested you can see this stuff is color coded now if we want to do if we want to say C CV equals five then we're basically going to be splitting our data five times um now what we have here is an example of different iterations that the algorithm will go through and if we just follow the first example where it says split one that would be the first iteration of a CB equals five right because we have five splits and we when you pay attention to split one we have one two three four five Folds One of the folds is in blue and then the remaining is in green so the green is the training data and then the blue is the test data that means in the first iteration what ends up happening and you can kind of think about it in the sense of everything from fold one to fold five that's all of the data that you have in your data frame but what's going to happen in this first iteration is it's going to fold a portion of that data and that portion of the data is going to be reserved for the test then the remaining is going to be for the training now the Assumption here is that the data has already been shuffled around it's random because if it's not then you'll generate incorrect results but we're going to assume that the data is already our data from is already in a random order but now this first iteration this fold that's reserved for the test data that represents a certain proportion of the data points in your data frame so this right here fold one would represent uh in this case 20 is reserved for the test data and then the remaining would be used for the train data okay that's the first iteration it'll use the green for training after it trains it will use the remaining which is in blue as a test and it will shoot out a an accuracy score than r squared then after this first iteration happens we'll go to the second iteration and then in the second iteration again just think about everything that you see from fold one through fold five as the entire data frame this is all lined up but then there's a part in that data frame that that is reserved for the test data and that's that where it says fold blue that's the 20 of the data that's reserved for the test the remaining green Parts would be used for the train and then you just keep on going through each iteration now what this is saying is that with this approach right this approach that's called k-fold cross validation what ends up happening is every single data points that you have in your data frame is going to have an opportunity to be part of the train as well as part of the test and that's a little bit different from the train test split approach not every data point is going to have an opportunity to be in a train set in a test set whereas this one will right so this is an example of a CV equals five or a five-fold cross-validation and let me go ahead and go back right here and start working on this part of the problem now what I'm going to include here is the estimator an estimator is what I'm instantiating which is this variable right here I'm going to put that as a first argument followed by the X and the Y now what I'm going to do in this example is I'm just going to do a simple linear regression model where I'll just have one feature and one target our feature is going to be the next one up I'd have to enter in this function and that's going to be the TV variable next is going to be the Target and for the Target I'm going to go ahead and say sales right that and then if I want I can call the CV equals five I don't have to because if I leave it alone and not the CV equals five that's automatic it's automatically going to result to CV equals five in the back end according to the documentation right here it's set to none in the back end meaning that it's going to use a five-fold cross validation if I don't want to do a five-fold I can change it to a three-fold for example I can do that but five-fold is the default but let's go ahead and continue on uh now I think there is also let's see I may actually it may not be there isn't that's fine uh now I'm gonna I'm gonna go ahead and do this now I want to print out what's going to happen when I run these two lines of code and I want to just print out that variable called CB results now once I do that I'm gonna go ahead and actually I'm just going to run it right now I'm going to execute this cell and I get a list of values and these values represent the accuracy scores for each iteration for each split that occurred right you can see that they're all within the wall bark of like the higher 70s to 80s right so it ranges around that much I can say CV results and just give me the mean of them right and it will just give me an overall mean of what we see here and now when it says the like on average the model is Blink accurate we can say is roughly 80 percent accurate so this is another approach that could be done in machine learning when you're doing supervised models particularly regression you don't want to train test split you can do the cross validation the cross validation um is a I would say a slightly more reliable approach because of the different iterations that goes through uh and then every opportunity every data point has an opportunity to be in the train and as as well as in the test set and you have these five different accuracy scores that you can just Aggregate and get the mean and say that my me my my models are accuracies 0.80 right now clearly um the train test split approach also uses randomization splits it into train test so that's also reliable but what's not reliable is the very first approach where you just include all the data in the model and just try to generalize off of that and that's something that you want to avoid because we know that if you were to do that approach and it were to introduce new data later on that you might have something such as an overfit or an underfit type of model right so that is the approach and it comes down to using k-fold cross-validation I can change this value right here and say three that means it's going to go through three splits three iterations you can see right here each of the accuracy scores for those three and then if we just get the average we end up getting this value right here right five seems to be the common one so I'm just going to go ahead and leave it as that and just remember that when you have five that means you can have five different splits or five iterations it's going to produce five accuracy uh scores from the tests for each split and then you can average it out to get a generalized actually score now because we are dealing with a CD equals five then that means one over five twenty percent would be reserved for the um for the test set for each split okay so that is all that I want to cover in related in relation to painful cross-validation thank you for watching