Modeling with the Law of Total Expectation

Added: 2026-05-30

[00:00:00]My life is forever changed by the kindness, support, and strength that you guys Quant Guild has given me over the last 48-72 hours. And while I can't see Java this second, and he remains in a stable condition, I am going to continue to be there for you guys and help you master your quantitative skills, my mission.

[00:00:32]And I'm going to do so today by talking about the law of total expectation.

[00:00:40]I hope to have more information later today about Java and how he's doing and when he's going to be able to come back home. And I will keep you guys updated as soon as I know more information, you will.

[00:00:56]So, let's start with this idea of the law of total expectation.

[00:01:06]This is one of my favorite ideas. I actually first really understood it in a class on stochastic processes.

[00:01:16]And it really changed the way that I think about statistics and how we take outcomes and scale them by probabilities in different regions of a a sample space.

[00:01:28]So, let me kind of just draw for you the the diagram that we typically work with.

[00:01:34]And that's this idea of just a sample space here.

[00:01:38]And we usually call this omega.

[00:01:42]Big omega.

[00:01:44]And this is just pretty much everything that can happen. If we sum the probability of this entire space, we're going to get one or 100%.

[00:01:56]It's pretty much that can happen.

[00:01:59]And within this sample space, right, whether we're looking at a dice roll or coin flips or heights or whatever, if it's discrete or continuous, it doesn't really matter, we can always introduce the idea of a partition.

[00:02:16]The idea of a partition is that we can perfectly separate this sample space into one of a set of categories.

[00:02:29]So, for example, if I'm using gender, right, we can separate this into male and female.

[00:02:41]All right, and please don't get mad at me right now for the for this.

[00:02:45]Um but the the two genders here can perfectly partition the the sample space and we're able then to use this law of total expectation to our advantage if we know something perfectly partitions the sample space.

[00:03:06]How might we do that? Well, we have this idea of an expectation and by definition the expectation is the expected value of, let's say, random variable X and we're going to call random variable X maybe the height.

[00:03:22]Okay, of people in a particular population. This could be the world.

[00:03:26]This could be a school. This could be a workplace. It doesn't really matter.

[00:03:31]This is going to be equal to and we have this idea of just the discrete and continuous expected value by definition. So, in the discrete sense, it's going to be the sum of all outcomes in a particular set.

[00:03:50]We'll call that A times the probability that A occurs. So, we're effectively normalizing all of the quantitative values. Well, not really normalizing, I guess you could think of it that way, but we're scaling all of the values by their corresponding probabilities. So, if we have, let's say, a height of 5'1" with a probability of 0.16, then we're going to multiply these two together, and we're going to do that for every single height. And that's what's going to give us the expected value.

[00:04:21]Now, of course, it doesn't really work too great there cuz it's more It's really reminds us more of like a continuous random variable, something like height. Something discrete we typically think about like a coin flip or a dice roll.

[00:04:36]But, if we're dealing with a continuous random variable, we're really dealing with a probability mass function. So, this is going to be equal to then, instead, the integral, which is just a continuous summation anyway, of X times the probability mass function of random variable X.

[00:04:56]Right? Parameterized by X, DX.

[00:05:00]Okay? And we're just going to integrate across the entire support chi. There's a lot of X's there. They're not all equivalent. This is going to be the X to parameterize the function and the integrand. This is random variable X in that case, that's the height. And this is going to be the support for the probability density function.

[00:05:20]So, this is by definition the expectation. But, it turns out that a lot of the times it's easier to compute the expectation conditionally than it is to just compute it unconditionally. So, this is an unconditional expectation.

[00:05:39]If I have just the expected value of height, so, X, remember, is going to be height, and I say that it's like call it 5'5".

[00:05:49]>> [snorts] >> We don't know if that's the average height of males, females, we we have no idea, right? It's just a compression of everything in this sample space, right? You can think of each points as a different individual in this population, and they're going to live in their corresponding partition appropriately, and we're effectively just compressing all that information down into this one value here.

[00:06:17]Okay.

[00:06:19]So, how can we use the law of total expectation to our advantage here? What exactly is it anyway? Well, it turns out that this is going to be equal to the expected value by definition here.

[00:06:31]If we use a couple of our axioms from probability and statistics, it turns out that this is going to be equal to the expected value of height given a particular partition.

[00:06:49]So, in this case we have a partition of male and female.

[00:06:53]So, I'm going to put male here.

[00:06:55]And then what I'm going to do is I'm going to take this conditional expectation. This isn't all that difficult to interpret. It's just the average height given that you're looking at males, right? So, instead of looking at this entire population, we're just going to look at males.

[00:07:11]Okay.

[00:07:13]So, that's one component of this unconditional expectation. It's effectively a decomposition. So, what are we going to do then? Well, we're going to scale it by the probability of observing a male.

[00:07:27]And I've always looked at it like, here's the thing that you want in an unconditional way.

[00:07:34]And then here's the condition.

[00:07:36]We got to find the expected value, then we have to find the corresponding probability for the condition, and this is how we're just gluing together pieces of the overall unconditional expectation.

[00:07:47]So now when we add, right? What do we need to add to get the rest of the space? Well, male is male and female partition the entire space, right? That makes up 100% of the sample space.

[00:07:59]So we're just going to need to glue the female piece together now. So it's going to be the expected value of X given female.

[00:08:08]And then times the probability of being a female.

[00:08:12]And this is actually going to equal if we find each of these components individually. Remember how we found the original 5.5? Well, if we glue all these pieces together, we're going to get that same 5.5.

[00:08:25]All right?

[00:08:27]That's this idea of the law of total expectation.

[00:08:32]Now, in a frequentist sense, this is a very important idea here.

[00:08:39]In a frequentist sense, right? If we continue to sample from the same population distribution, the same data generating distribution, these are all going to converge.

[00:08:51]Right? Because this is a compression of probability and outcome in a discrete or continuous sense, right? It's the same idea. We're just conditioning on a subset of the sample space here, which is just effectively giving us a new omega. Right? Same thing with female.

[00:09:10]It's just effectively giving us a new omega. We're just normalizing then by the quantity of males and quantity of females in those partitions, right?

[00:09:19]They're subsets of the sample space. But in the frequentist sense, these are all going to converge to their true values.

[00:09:29]So what do I mean by that? Well, if you have a small sample, right? We can think of it like this.

[00:09:36]If I have a small sample, then I might observe height as I continue to take more samples, right? I'm going to observe Let's take a look at the expected value of height given male. I'm going to observe this over the quantity of samples that I have. So, this on the Y axis here is expected value of height given male, and on the X axis here, we have the number of males that I've sampled in this particular population omega.

[00:10:07]You're going to find that as you continue to take samples, that this is going to eventually cumulatively converge to the true theoretical expectation, and that is by the law of large numbers. Asymptotically in a frequentist sense, the law of large numbers says that all probabilities, statistics, and distributions are going to converge to their data generating distributions, right?

[00:10:40]Now, what's the problem here?

[00:10:42]In practice, right? This is kind of true, and it's useful for the most part for the statistics and probabilities that we need in practice.

[00:10:52]However, the problem is over time, this theoretical expectation changes.

[00:10:59]And that's because the data generating distribution changes in kind of a broader sense, there really is no data generating distribution. We're imposing that structure on the space, and this is kind of the problem with using probability and statistics in finance just out the gate super aggressively is you're not going to have this nice frequentist convergence, right?

[00:11:22]Asymptotically in finance, if you look at stock returns, if you look at anything like that, you're not going to have probabilities, distributions, and statistics converge because the data generating distribution is changing over time.

[00:11:38]The way that you can think about this is imagine I have a school.

[00:11:43]See if I can draw.

[00:11:46]Yeah, crushed it. I have a school in call it the 1900s, maybe like 1902.

[00:11:53]And I have a school today, right?

[00:11:56]Completely different schools.

[00:11:59]Maybe a school in 2026.

[00:12:03]If I take a look at the expected value of height given male in 1902, is this going to be equal to the expected value of height given male in 2026?

[00:12:21]I mean, absolutely not, right? There's so many different factors that go into this, right? Are these in the same location? Time plays a a significant role in the population's average height, right? Over time. There's so many things that are going to change the distribution of males over time and their corresponding heights. And that's why in real life we don't see these statistics, probabilities, so on converge.

[00:12:53]There is no fixed data generating distribution. Yes, we have the central limit theorem, but the central limit theorem only holds under specific conditions, and in real life, when you have this time variation, this non-stationarity, that's when you get misled and you're not actually looking at correct probability statistics and distributions.

[00:13:16]So, these are most certainly not equal, right? It's going to be a function of a a variety of different things.

[00:13:24]But, nevertheless, we can see then, right, in the overall expectation, if we have all that information. So, let's say we have this information.

[00:13:35]Let's say we have the probability of male in both time frames. We have then the same for females, the expected value of height given female times the probability of female.

[00:13:49]The expected value of female and the probability of female in both time periods. Right? Hopefully, you see now in the stock return sense what we're building to. Clearly here, the two unconditional expected values, the expected value of height in 2026 is not going to be equal to the expected value of height in 1902, which is why your stock returns in the early 2000s are on average are not going to be equal to your stock returns now, right? They're companies. It's amorphous. We're not dealing with frequency-based probability statistics and distributions in real life. Instead, we're imposing that structure to try to make sound decisions under uncertainty.

[00:14:41]What you're seeing here is that if I can decompose this expectation into these different legs, I can see what changed most dramatically over this time period. Was it the average height of males? Is that what changed dramatically?

[00:14:58]Was it the probability of being a male?

[00:15:01]Is that what changed dramatically?

[00:15:03]Was it the average height of female? Is that what changed dramatically? Or the probability of being a female?

[00:15:11]All of these go into the unconditional expectations. So, when there's significant deviation in the unconditional expectation, you can decompose into the individual parts, and over time, you can see what is drifting, what is changing. You can do distribution distance measures, the list goes on, but keep in mind, right? We don't have this frequency interpretation that we do in the classroom. It's more amorphous, right? So, what do I mean by that? Well, what's really nice about the law of total expectation is, so long as you can partition the sample space, you're going to be able to decompose anything, even if it is more amorphous, right? Even if it's extremely chaotic, even if the distributions change violently over time, it doesn't change the fact that this is true. It does change the fact that it may not be stable over time, which is exactly why you perform this sort of analysis here in a temporal sense, but let's apply this idea now to trading, right? So, if I have a Let's call it a win.

[00:16:23]and a loss in terms of a trade, right? And this is a sample space of all of my trades, then I'm going to be able to partition the sample space perfectly by whether or not a trade was a win or a loss. This is one of the first things that I talk to any sort of anyone who comes to me for professional consulting, this is one of the first things I ask them to show me is the distribution stability over time of their their strategy, right? If if they are trading some sort of inefficiency, um whatever it is, but trades are going to live in one of these two one of these two partitions, right? A subset of the overall sample space.

[00:17:12]You're going to have your winning trades, you're going to have your losing trades, right? Now, the particularly keen among you will notice that this is likely to be more granular, right? It's probably not going to just be your wins and your losses. You're also going to have like sub regimes. So, you're going to break this up into like a low a mid and a high volatility regime. And then you'll even segment again um by other conditional factors, macro factors, political factors, whatever. And then you'll look at your win and loss. You can apply the law of total expectation in the exact same way. As long as you're dealing with a partition of the sample space. So, more formally here, right? A partition is just going to be if I take the union of all of these different events, then I'm going to get the original sample space. And if I take the intersection of all of these events such that I'm dealing with two different events, then we are going to be completely disjoint. And there's not going to be any overlap. And that's what that's what you see in all of these partitions that I'm drawing, right? I'm drawing them effectively so they have no overlap.

[00:18:31]They're just slicing and dicing this original sample space here.

[00:18:36]So, let's take a look then at the idea of the Let's take a look now at the idea of the expectation of trading P&L.

[00:18:50]Right? So, each one of these is going to be a trade. That means that we're going to have by the law of total expectation, the expected value of your P&L given a winning trade times the probability of a winning trade plus the expected value of your P&L given a losing trade times the probability of a losing trade.

[00:19:11]And [snorts] the problem here is that these are typically extremely unstable, right?

[00:19:18]>> [snorts] >> So, what you're going to have to do is you're going to have to continue to condition on different elements, volatility regimes, strategies to try to create reasonable stability. That is what your trading edge is effectively.

[00:19:32]If you have an edge, then this is going to be reasonably stable over time. We often talk about discretionary trading being performance, it's swings at bat.

[00:19:42]Even still, if you have an edge and you decompose your trades over time, you should see reasonable stability here, right? The mean should be on average positive with reasonably constrained variance such that you're not blowing out or such that you actually have like structurally orthogonal risk to just pure market exposure that you're taking advantage of.

[00:20:08]But, that goes into >> [snorts] >> other types of analysis that you would do.

[00:20:13]So, the thing here to keep in mind is we don't have that nice frequentist convergence, right? If I have n trades here and I'm looking at the expected P&L of my winning trades, right? As I make more trades, this doesn't have to converge to anything, right? Now, we say here with height that it did.

[00:20:36]The only The only reason why I'm saying it does with height is because it's likely more stationary, at least ephemerally, [snorts] than it is for something like your trades.

[00:20:48]Unless you have a really good strategy, then it's going to exhibit that kind of stability, right? But, if I take the average height of people in a workplace, people in a school, that tomorrow isn't really going to change all that violently, that distribution, right?

[00:21:04]But, over 10, 20, 50, 100 years, then you're going to observe those significant differences, right? Now, in finance, we actually observe those significant differences quite quickly, and distributions are going to change very aggressively um over the short, mid, and even long term. Just depends on what it is you're trying to trade and what it is you're doing.

[00:21:28]All right? So, just keep that in mind in the context of the law of total expectation. You can always decompose an unconditional expectation into a series of conditional expectations, and each conditional expectation is scaled by their corresponding probability. In a frequentist sense, if we're in the classroom, so to speak, uh in the lab, and we're just drawing from fixed distributions, then if I just keep increasing the amount of either people that I'm taking the average height of, or I keep increasing the total number of trades, >> [snorts] >> eventually I'm going to eventually I'm going to converge to the the true values of all of these, the probability statistics, and even distributions. But if we're not in the classroom, then convergence is not guaranteed, and that's the entire game.

[00:22:19]The entire game is trying to figure out stability, and trying to make the best decisions that you possibly can in a forward-looking sense without that information, knowing that this information is somewhat flawed, right?

[00:22:32]And you can use backtesting, walk forward testing to try to figure out how flawed it is. Obviously, that there is bias in doing that. It's historical data, right? All all empirical research in that sense is is data mining, I heard um I heard that from a from somebody recently, um just to to keep their privacy.

[00:22:57]I and I I I I loved uh I loved that idea.

[00:23:01]Um and if you've actually uncovered something structural, then you can continue to to take advantage of it.

[00:23:07]So, [snorts] that's the idea of the law total expectation. Um, I hope you enjoyed this video. I hope you learned something. I would love to continue to do um, more discussions on probability and statistics if there's interest kind of continuing to bridge this gap between the classroom and and practice.

[00:23:27]Um, thank you guys for all of the support.

[00:23:31]It means so much to me. I I I really thank thank you.

[00:23:39]Thank you.

[00:23:41]As soon as I know more, I'll let you know more.

[00:23:43]Um, and thank you guys for being there for me.

[00:23:51]Um, tha- thank you for watching this video and uh, I'll I'll see you I'll see you in the next video.