How to handle Regime Changes (by ex HFT quant trader)

Added: 2026-05-19

[00:00:01]Hi everyone, welcome to this video. So, this video is going to be how to adapt to regime changes.

[00:00:07]And so, the goal of this video really is to show like how you can adapt your strategy to when there's huge changes in the distribution of your your data, which we call like regime change.

[00:00:19]So, for example, taking like a a strategy that's not adaptive, that might look like something like this to make it more adaptive, so the it it looks much better equity curve.

[00:00:34]And this is actually more than four ways, but we're going to look at four different methods to to handle regime shifts and to make our trading more adaptive.

[00:00:45]But, before we go into we need to define what actually is a regime change. So, we need some sort of like mathematical definition.

[00:00:54]And so, to summarize in one sentence, we can say that it can be modeled as when the time series exhibits non-stationary dynamics.

[00:01:03]So, we need to understand what is stationarity.

[00:01:08]So, stationarity is when the statistical properties are invariant over time. So, this means that they stay constant over time.

[00:01:16]And the properties more specifically are the mean, which is the a measure of central location.

[00:01:23]So, that's the the highest point essentially in in a distribution. Uh and the the variance, which is a measure of the spread of the distributions. Maybe I should show you like a a distribution here.

[00:01:37]Okay. So, this is the returns distribution for Bitcoin dollar. So, the the central location's like the uh the peak of the distribution, like the central location.

[00:01:49]And then the spread is like the the width of the distribution essentially.

[00:01:55]And then we've got the the third moment is the skewness, which it measures like how symmetric it is. So, if it's not symmetric, then it's skewed on one side.

[00:02:06]Uh so, that's a one important measure. And then, the other one, uh another important measure is the ketosis, which essentially is a measure of uh your fat tails, how these are the tails, right? Which is like extreme events.

[00:02:22]And uh the higher the ketosis, the more chance of like a very extreme event happening.

[00:02:30]So, these are the four moments of a of a distribution, which is and the stationary assumes that they're constant and essentially invariant over time.

[00:02:40]Uh but, we're we're going to show you that in practice, this is never There are There are never stationary time series.

[00:02:47]Uh I think in practice, there's only like, for example, uh there's very few that are stationary.

[00:02:55]For example, you could have like a a very high sharp trading strategy, and you look at the the P&L of it. That would probably be s- stationary because it's like a pretty much like a straight line.

[00:03:08]But, uh let's have a look at Yeah, so anyway, here's here's the For example, just looking at the the moments of the distribution, we can see like concrete example where the mean is positive, so on average, it goes up. Uh standard deviation, uh the measure of the the variance of 3%, and then the skewed, this means it's posit- positively skewed.

[00:03:32]>> [snorts] >> And then, a a ket- ketosis of like three. So, the uh there's tails here, fat tails. So, the the likely the higher the ketosis, the higher the chance of extreme price movements.

[00:03:46]>> [snorts] >> And the key question here is like, are BTC returns stationary?

[00:03:51]Uh there are lots of like test statistical tests that we could do, but I think we can just look empirically and just see like So what I've done is grouped the the returns by the moments by year aggregating the the statistical moments.

[00:04:09]And you can see here like for example in 2012 like they're pretty much positive apart from a negative mean in 2022 and 2026. And you can see for example in 2026 the there's more extreme price movements in 2026 let's say compared to like 2021.

[00:04:31]And we look at like the symmetry we can see like there's a negative symmetry here for when 2022 I think this is dark pertaining to when I think FTX collapsed. Right, so this like just crushed BTC.

[00:04:48]Okay, so So you should see here and even just by looking at the chart you can sort you know you don't have to be a rocket scientist to to see here that there's like trends here and you know huge fluctuations in in prices.

[00:05:07]So for example here in 2022 it like shoots down right, but then in 2024-25 it just like shoots up.

[00:05:17]So there's huge structural changes in the distribution of the returns.

[00:05:22]And so the key question is is like how do you handle this like non-stationarity behavior? And especially like in supervised learning which is the most common form of like machine learning.

[00:05:33]So let's say we we got a model to predict future returns and we're just going to use the the current return. This is what's called like an auto regressive model.

[00:05:43]In econometrics this is called uh, an AR1 model.

[00:05:48]And, so, what we're essentially saying is that the future, uh, our, which is log return, is that the future log return is basic equals a weight, which we will determine by machine learning. Uh, the weight multiplied by the the current log return plus a bias.

[00:06:07]So, before we do that, we need to to calculate the log returns, which we do now. Okay, and I've created, uh, a function here that trains a linear regression model.

[00:06:17]Uh, so, we split the data into train and test. Uh, what's really important here is that we don't shuffle the data. So, we what we call preserve temporal order, which is the order of the time. So, we never like leak any future data points it it does not know about in the in the train. Uh, so, we've done that.

[00:06:37]And then we fit the data with the training data, so we don't fit it with all the data because we need to preserve some data, uh, to for our back test. But, we're going to do it for the back test and test against all the the in-sample and the out-of-sample data.

[00:06:51]Uh, just to show because it's also important that we have good in-sample statistics as well.

[00:06:57]And so, I've just put this model. And so, what I've done is I've, uh, run the model I I just against the previous, uh, return to predict the future, return.

[00:07:08]Okay? And you can see this is the equity equity curve.

[00:07:14]But, there's something wrong with this. Uh, so, this is the what's the weight. So, this is a mean reversion model because that has a negative, uh, weighting and then there's a positive bias as well.

[00:07:26]Uh, so, the but the the problem is, let me go down, is that it looks identical to the actual time series itself. It's just in a different scale.

[00:07:37]So, this automatically like is a red flag. And when we look at the the distribution of the signal, which is basically uh, maybe if I show the signal is basically that the sign of the prediction why I had been on the the models prediction and we convert them to our signal is and then we obviously calculate the what the the trade would be but if we look at the distribution it's just like 98% of the time it's predicting to go up so this is why it looks equivalent to the actual underlying time series because it's just going up all the time. So this is not really adaptive and you can see it's not adaptive because when we we saw it like in 2022 to 23 is that you know it goes down on average and you can see here that it also goes down so it's not adaptive to these regime changes.

[00:08:34]So what can we do about it?

[00:08:36]And I mentioned that one of the one of the things we can do is this like sliding window approach. I'm personally not a big fan of it because essentially you're just looking for a a local pattern localized pattern to predict the next time slice and it's also very sensitive to how big your window size is and so I've just shown like in an example here is like okay we split it into let's say like n windows we predict we train on the first I don't know k windows to predict the next m windows.

[00:09:12]So you can see like for example here like okay we're using the the first four windows to predict the the fifth window and then we just shift it across and predict okay but this but we're not using all the data to to predict and uh you know this could possibly work but I'm just not a personal fan I just feel it's like a team move version of like cuz you just taking a subset of the data, right?

[00:09:42]>> [snorts] >> So another form is that we can encode memory. And so essentially we can do do is like take this like several data points like cuz financial time series data is extremely noisy.

[00:09:57]And we can compress it into a single single file.

[00:10:01]We can take a vector of past returns and compress it into a single scalar.

[00:10:06]This is essentially like a form of like a hidden state.

[00:10:10]And the idea is that we can encode the memory the the current dynamics of the the returns.

[00:10:20]And this is what we do here. It's really important that we're doing this rolling statistics. So this is basically just looking at the previous 40 returns and taking the average.

[00:10:31]>> [snorts] >> What's really important though is that I'm doing it on not on closed log return but the lag because otherwise we'd have data leakage because it would be encoding the a future return it would not know about. And so there'd be data leakage there.

[00:10:48]So that's a very important note.

[00:10:51]And so we're just like summarize we're adding up and then we're we're taking the mean.

[00:10:56]And then so we're going to do exactly the same model auto aggressive model but instead of just passing in the the lag one we're going to take the the moving average of it.

[00:11:09]And you can see here automatically that it looks much better now. So when there was like this huge price drop from 2022 to 20 >> [snorts] >> 22 to 23 we don't see this huge drop in the drawdown, right? And for example here in you know just the beginning of 2026 there's a big change, right?

[00:11:32]So but let's a look at the distribution of the data. You can see here that it was roughly about 100, you know, data point like times it went predict down, right? And you can see here there's actually increase. So it is adaptive, right? So if it if Ah, and we should have a look and see what the model is. So this is a momentum type of model, right? And this is the beauty of like linear regression algorithms because I can just easily interpret it. If this is a neural network, I would have no clue like why did it decide to go up or down. But this is like a because it's just you know, there's just two parameters, I can really interpret like the the dynamics.

[00:12:12]>> [snorts] >> And you can see here that it's increased the number of data points that it's predicted down. So it's more adaptive when there was actually like a a regime change like around 2022.

[00:12:24]Okay, so now we looked at encoding memory, but what about modeling relative memory?

[00:12:30]So for example, we could pass in the the the moving average, but we can also pass in the the current log return.

[00:12:39]And we look at the models here, it's taken the model is essentially taken the difference between a weighted difference of the the current log return against and you know against a weighted value of the the moving average.

[00:12:59]>> [snorts] >> Uh, and you can see here that there's obviously a bigger weight in care for for the the moving average. And you can see here that it's even more, you know, it looks completely different from the the actual underlying. So from original model that we did, the linear model with just the lag one, which is pretty much, you know, going up all the whole time, you can see here it looks completely different, right? So it's more adaptive. So when there's a huge structural change, a regime change, like for example here, you can see that it still makes money. And if we looked at the the winning rate, it would probably only be about like 51%, 50.5%.

[00:13:37]It won't be like a huge winning rate. Uh but you can just see here that it's more adaptive, right? And this is uh essentially just a form of feature engineering for our supervised learning.

[00:13:49]And again, if we look at the the distribution of the signals, you can see that it's it's increased the number of times it's predicted down. So, that previously it was about 300, and now this is like over 400. So, you can see by making it more adaptive to to bet down, that when it actually does go down, that we make money here.

[00:14:13]So, that's in a in a sup- that's covering how to essentially do feature engineering to make your to create features that are more adaptive to non-stationarity, right? Uh but now we're going to look at a different form of learning algorithms, which are called online learning.

[00:14:32]Uh and essentially we're going to look at what's called this passive-aggressive regressor.

[00:14:37]So, it's what it's going to do is it's going to to continually change the weights of our linear regression.

[00:14:44]So, for example, it could like on one day time step, it could be saying, "Okay, this is a momentum type of strategy, a positive coefficient." And suddenly, maybe a few data points, it changes to be a mean reversion, so it has a negative a negative weight.

[00:15:03]>> [snorts] >> And the idea is is that how it how does it change the weight? So, it does it uses this what's called a passive-aggressive learning.

[00:15:11]And essentially, what happens is if it if it predicts correctly, it doesn't change the weight.

[00:15:17]Okay? But then if it if the model the predictions too deviates away too much, then it corrects it. So, this is a form of like error correction.

[00:15:28]And it's proportional to the to the amount of error in the prediction.

[00:15:33]Uh so, for example, here and it what's very important is that, for example, just all the the the learning that we've been doing our machine learning has just been all past all the data at once. It's like a full batch. And then we it's all That's another important thing is that this is uh the learning regression. If I go up, it's a closed form solution. So, you you pass all the data around. It's a closed form solution uh ordinary least squares. And then it all together what minimizes the the error.

[00:16:06]Okay? Uh but with this, let's just go back to here.

[00:16:10]This is more This online learning is more of a stream. We don't pass all the data at once. So, and this like complements uh non-stationarity in financial time we just get a new data point and we pass it in, make a prediction. If the prediction's wrong, we make a an adjustment to the weights.

[00:16:30]And then these are what's are called like hyper parameters. I won't go into what all these hyper parameters are. But again, we're just going to use the the previous uh the the current return to predict the future uh log return. Uh we've got this thing here that's a standard scalar, which is to scale it for the cuz we we I'll point this out. So, this is what's called a stochastic gradient descent. Uh this is like the most common uh machine learning algorithm the to to do the learning.

[00:17:00]So, we we've got a machine learning algorithm to to adjust our weights, do like error correction.

[00:17:05]Uh So, I think the most important thing here is that, you know, the train step is not it's a partial fit. So, we're just passing one data point at a time, make the fit it. So, basically, run the stochastic gradient descent, work out what the the weight and the biases at each time step because we're going to record it and display it in a Pandas data frame. Then we also take the the predicted value, and we take the the sign of it. That's our signal. And again, we just calculate what the the trade return would be, the log return more specifically.

[00:17:40]And and then we just put it into this records with cuz we're going to store in a Pandas data frame. And here out of the loop, we then create our data frame the results.

[00:17:52]And then we just take the cumulative trade log return to to display our gross equity curve.

[00:17:58]And and we're also just going to calculate the the directional hit rate, right?

[00:18:03]Without any form of like hyper optimization of the parameters, right?

[00:18:08]So, we can just look and see what's happening here. So, it's one like 50 just over 50% of the the time. And you can see if it's one or not. You can see what's very important here is that for example, in the the second tick, it the the weight is a momentum style model, right? But in the next tick, it's a mean reversion model.

[00:18:34]>> [cough and snorts] >> Pardon me.

[00:18:36]And yeah.

[00:18:40]This is and you can see it's changing, right? It it stays momentum, but then suddenly it changes to mean reversion, right? And this is the again the the beauty of of doing this on a linear regression because you know, there's just two parameters here to do the the learning. So, we don't have this thing what's called the curse of dimensionality. We have like so many parameters that it becomes like a an NP-hard problem.

[00:19:01]So, UK, so that's the and you can see the weights are adaptive, right? And in in our supervised, that weight is constant and it's invariant over time, right? And you can see here that it's adaptive and we can actually you we can let it all maybe I just show you the equity curve. When we look at the distribution here, it's actually nearly 50/50 going up and 50 going down, right?

[00:19:29]>> [snorts] >> So, uh and we look at the equity curve, uh it's not it doesn't the returns are we need to would have to calculate the the sharp of this, right? But um it looks a lot more smoother, right?

[00:19:43]Obviously, the returns aren't great as the supervised, but we haven't done any form of like hyper op hyperparameter optimization, which is these uh the models parameters. We haven't optimized any of these to to improve this, right? So, I've just uh picked this and again, this is this passive-aggressive learning curve. And then we have a look at the underlying time series, you can see here when it was like shooting down, it actually makes money when it was shooting down.

[00:20:10]And let's have another example. So, when it was like trending upwards, that's it.

[00:20:15]Also made money here as well. So, you can see here that it's made money both when it's trended down downwards and it's when it's open.

[00:20:26]When it's trended trending upwards.

[00:20:29]But we can make this even more adaptive by just choosing some good hyper parameters to to this.

[00:20:37]Okay, so that's online learning. So, now let's look into reinforcement learning.

[00:20:43]Uh So, I'm going the next video I'm going to do is going to be a video on reinforcement learning. So, it's a whole video dedicated to itself because uh would this video would be like 2 hours long if we just if I went into the into the depth of reinforcement learning.

[00:21:00]>> [snorts] >> So, reinforcement learning is quite similar to the online learning it where it's like if there's good behavior or good tradings, we we give it a reward, but if it's bad trades, we we penalize it. So, the idea is is that it can tell it makes good trades.

[00:21:17]Uh and we can uh formulate trading as uh So, oh, that's a good point. So, okay. So, with this um reinforcement learning, I just want you to imagine that's we know we are we're at a casino.

[00:21:31]We've got a Hawaii Hawaiian shirt on, uh sipping cocktails, and we're we're we're we're we've gone to a a a bandit. So, this is like a a bandit. We're going to go to like a two-armed bandit. And essentially, trading you can you can model it as a as a two-armed bandit problem. Okay? Which is we don't know the we pull on one uh essentially, let's say the right arm is to predict going up, and the left arm is to predict going down, right? And this this is uh and we don't know what the underlying distribution of the data is, right? Uh and it's partially observable. We only know we we don't know what the true distribution of the of the of the rewards, right? So, we need to play the game or play on this bandit to understand the true distribution. And obviously, it could also the distribution could completely change. It could be non-stationary, right? So, we just to just to gently break you in because we're going to just look at a basic two-armed bandit problem.

[00:22:34]>> [snorts] >> Uh so, yeah, oh, this is the for the actual algorithm that we're using uh we're going to use this reinforce algorithm, which is the foundation of all policy gradient learning algorithms is this reinforce algorithm. Because usually, when you how reinforcement learning is introduced is is with this like Q-learning, uh which is not you're not optimizing the uh, the strategy itself. You're setting up optimize in the value from Anyway, that's it. Maybe a bit out of scope. So, basically, what we're going to look at is where we're going to The first thing we're going to do is look at a partially observable stationary two-armed bandit.

[00:23:15]Okay, what the hell does that mean? So, basically, we're saying that it's a two-armed bandit. It essentially it's a coin toss, right? Uh, and, you know, you can either pick heads or tails. And this is a two-armed bandit because there's two choices, right? Can pick heads or tails. We don't know what the distribution. And this is a stationary biased coin. So, this coin is biased, so it's not 50/50. But, and by default here, it's like 70% probability of going heads. We don't know the underlying distribution. We can only uh, estimate by playing this game.

[00:23:51]>> [snorts] >> Okay? And also, this is the the policy.

[00:23:54]So, the policy is is that you pass in some state and it gives you an action.

[00:23:58]Uh, and the action here is essentially it's the it's the strategy. Uh, and it gives you an action. Like, what should you do? Should you go heads or should you go tails?

[00:24:08]Right.

[00:24:09]Uh, and that's the So, that that's our first uh, environment. And that's is this like stationary biased coin.

[00:24:19]So, what we're going to do is we're we we initially we set the the To make this deterministic, we set some seeds. We've got our policy, which is essentially the the the strategy of like choosing our actions based on the the state. Uh, this is actually stateless, so just put put that out there. So, this is like a dummy input. Uh, we do like a forward pass. So, we'll basically sample an action from the current policy {slash} strategy. Uh, and we then interact with the environment. Okay? If we take this action, like, what's the reward? And the reward is like, if it's heads, if you could if you guess correctly, then you get a dollar and if you lose, then you lose $1, right? We're just going to keep it like as a static reward because it just uh minimizes the noise, right?

[00:25:14]Uh and then we do this the the reinforcement uh reinforce algorithm, which is basically uh we take the negation of the log probability multiplied by the reward, and that's our loss function.

[00:25:25]And then what what what we do is we just do the run the optimization.

[00:25:29]Uh the but we it's imperative that we zero out the gradient because if we don't, the gradients get accumulated.

[00:25:37]But anyway, don't worry too much about the the the details here. And then after a certain amount of time steps, then we just log the the current rewards.

[00:25:46]Uh but this is the the experiment here.

[00:25:49]And I'm just uh show you like for example the the for this we've set it to 70% um we've run over like 2,000 episodes, and let's see what what is the strategy or the policy it's guessed.

[00:26:03]So, you can see here's the rewards over time. You can see it's like positive. It never has like negative rewards. Uh what's interesting it is is that you can see here that's the You can see like the policy has like a it believes like uh heads with a just over 60% probability of going heads.

[00:26:27]Oh, no, this is just I should also mention this is a stochastic strategy, right? So, it's sampling from a distribution and it's but it's changing here to say like okay, it's 100% that they should be going heads here.

[00:26:40]And you can see like it goes like okay, it's just over 60% to to predict heads. But as as it's learning is this policy gradient learning, it's converging to say like okay, you should just always go up and that's the optimal is just always bet up right because 70% of the time it goes heads. So, that's the for a stationary bandit, right? But that's just a toy example, right? And now we just add a bit more complexity and just a bit more of a real real world scenario, right? So, now what we're going to do is look at partially observable non-stationary two-armed bandit.

[00:27:21]And we're just going to simplify things here. We could maybe use like a cyclic sin function to, you know, to to cycle between different probabilities, right? But we're just going to keep it basic and just say like after a certain amount of time it's going to switch. So, before the switch, it the probability of heads is going to be 70% then after the switch after after 1,000 episodes by default, it's then going to be a 20 20% probability.

[00:27:54]So, the distribution is going to change over time, right?

[00:27:58]>> [snorts] >> And obviously we we can't observe this.

[00:28:01]We can only partially observe it by playing the game and fully understand it. So, for example, when you're doing like interviews and they're asking you like maybe it's run some market making this this is what essentially what you're doing. You're trying to work out what's the distribution of your of the the data like set prices based on on on that distribution. As as similar, right?

[00:28:23]Um Okay. So, that's our non-stationary biased coin. Ah, and here's a very important part is that we need to have some form of regularization. So, a very key important concept in reinforcement learning is exploration versus exploitation. You can't For example, if you just did exploitation the whole time >> [snorts] >> you'd end up in a local optima. And we you always want to make sure you never go into like a local optima. So, it's imagine like you're in a maze, right?

[00:28:56]Uh you could go down a route which is maybe the the shortest path so far, but it could, you know, like the long term is that it's a a dead end, right? Uh so, you need to to to constantly explore different uh options. And, you know, to to to ensure that it doesn't go into a local optima local optima, we use this what's called entropy.

[00:29:19]Uh so, this is what's called Shannon entropy. And as even if you don't understand the math, it doesn't we can build intuition here what what what it's uh doing. So, for example, on a coin toss, the the the high the max uncertainty is when the probability of heads is 50/50, right? And you know, that that's intuitive because uh and then the you know, the entropy of it is 1.0. But, however, if it's biased and it's biased like it's just heads 0% of the time, that means it's 100% tails, then the entropy is zero.

[00:29:58]So, essentially, what this entropy regularization is doing is it is ensuring that uh you you don't always just sample heads all the time or just sample tails all all the the time that you also uh because when the distribution changes, you may see that okay, you actually you you go heads, you get reward, but over time. Anyway, it might be better if I just show you the example, right? So, for example, this is our it's just pretty much the the same like just change the environment. Uh >> [snorts] >> And we add the entropy loss. So, now the loss is just uh two terms, which is the policy loss and the entropy. So, we calculate the entropy of the distribution so far, the empirical distribution.

[00:30:44]Uh And, you know, it's exactly the same, but we're just calculating the entropy, and then we you got this entropy beta, which is just a waiting of the entropy. So, we can make it less sensitive to entropy or more sensitive to entropy.

[00:30:59]And then it's just the summation of the policy loss and the entropy loss, right?

[00:31:03]Uh and again, just remember because with machine learning, it's about minimizing.

[00:31:08]So, we because we just take it then we just negate it, right? So, uh again, with the entropy beta, we just take the the the negation of the of the the the weight.

[00:31:21]And again, we do the the machine learning, it's exactly the same, okay?

[00:31:26]And we're going to run two experiments, one with the vanilla uh reinforce, which has no entropy regularization. And then we're going to want run same again, but with entropy regularization.

[00:31:40]And you're going to see something very interesting here.

[00:31:43]Okay. So, this coin Remember this coin uh bandit changes.

[00:31:49]So, just again, just imagine we're at the casino, we're either choosing one hand or the other. We notice like, okay, this hand is giving us money, the other one's not. So, we just choose the left arm, right? But after a certain amount of time, the dodgy casino changes time like changes and flip it, right?

[00:32:05]If our If you can see here what's happening with our vanilla without the entropy regularization, it's making money.

[00:32:13]But soon as the distribution changes, there's a regime change, it it loses money.

[00:32:18]And we can see what happens on the in this diagram here on the right. So, you can see with the vanilla, it it's got into a local optimum, which is to to go heads, which is the optimal for the for the left before the regime change.

[00:32:35]But when there's a regime Just make it very clear, this black line is when the regime change regime change happens. So, that the the distribution change. So, this is when the the probability of heads is 70% and this is when then it switches and then it comes to 20%. And you can see here that it just always stays at 1%. So, it does no exploration here. It's just pure exploitation. It found this local optima. It's got into it and there's no regularization, so it's just stays there. So, when the actual distribution of the returns, also the you know, the bandit changes, uh it doesn't adapt. Uh but, you can see here with the entropy regularization, it is never going to like 1% uh as only does here. But, as soon as the distribution changes and it's done the regime change, you can see it has adapted here. So, it goes all the way back So, it was predicting uh uh heads, but soon as the change, it notices there's a a change in the distribution of the of its rewards. So, the the policy has changed to to predict uh heads.

[00:33:46]>> [snorts] >> So, tails.

[00:33:49]And you can see like it does it here, right? Uh So, you can see this is adaptive and you can even by changing the the the beta uh the entropy beta that we can make this even like more quickly adapt even more.

[00:34:02]Uh so, and again, this is like it's a way more complicated, so I can't just do this all in one video. It needs a a dedicated video by itself, but I just wanted to show just some of the principles of like how to handle uh regime changes in uh using reinforcement learning.

[00:34:22]Okay, so that's the the end of the video. So, uh just to conclude, so we've like mathematically defined what is a regime change, which is the when it exhibits like non-stationary dynamics in a in a financial time or a any time series. Uh we've shown some avenues here to make your machine learning models adaptive. So, for example, if you're doing supervised learning that you can encode like hidden states, uh and this makes it more your supervised learning more adaptive. Uh we've also looked at online learning, which is actually quite similar to reinforcement learning, but there's some key fundamental differences. Uh using this passive-aggressive learning.

[00:34:59]So, it it penalizes it's like a form of like online error correction. So, it just the weights if it make if the the errors the predictions are wrong, it just the weights.

[00:35:12]And so, this allows us to to to adapt to any changes in the distribution of the the returns.

[00:35:20]Uh and then we also looked at this this policy gradient reinforcement learning.

[00:35:25]And that was just a quick look. And there's actually another key fundamental difference as well. With the online learning, it's just it doesn't consider the cumulative returns, right? So, it doesn't make a decision now that will give us a better future return, like it say long-term return. It's just always like looking, okay, what's the best return like now and here and now.

[00:35:47]Whereas with reinforcement learning learning, you can make some I don't know, maybe short-term bad decisions, but more long-term it's much better.

[00:35:57]So, imagine like again like a maze. You could easily if you just you could easily go into like a local optima, uh but then, you know, there's better to it's not the most optimal, right? And again with like trading, like if you had like three actions like to to buy or go long, short, or just hold or essentially do nothing, right?

[00:36:20]It could be that, you know, by holding that you make more returns, right?

[00:36:25]>> [snorts] >> So, but yeah, we'll look into this in more detail later, but then you we just looked at with Keith and Keith that we looked at the entropy regularization. So, this allows us to balance that it doesn't get to a local optima and it's a form of keeping not getting in a local optima to of exploitation that it it looks for any changes in the distribution of of the the data, right?

[00:36:52]Okay, so I think that's concludes everything. So, I hope that was useful.

[00:36:57]Please like, subscribe and also give me your comments as well. I'll be very interested to to read them. Okay, thank you very much. Cheers. Bye.