Shannon’s entropy and why - info theory p2 - how AI and computer stuff works

[00:00:00]Okay, so in the last video we essentially went over this equation and how we come up with this equation of information as a function of probability is equal to negative log of probability.

[00:00:13]Please understand this before we move on with the next concept. Now, the next concept we're going to be talking about is what's called entropy. Specifically, Shannon's entropy.

[00:00:27]Okay.

[00:00:28]And this is given by the equation of H is equal to negative summation PI log PI.

[00:00:42]Okay. So, let's go back to information in general. Specifically, let's ask the question of what happens when there is a constant stream of information or there's multiple events of information. For example, let's say we have an AI model that's reading some sort of context. Obviously, there's going to be multiple events or pieces of information in that context. So, how do we deal with that? Well, we're going to deal with that with what's called entropy. Specifically, what we're going to be looking for is the average amount of information coming from a specific source.

[00:01:24]>> [snorts] >> Okay. So, let's get started by essentially deriving it. So, first let's go over expected value, okay? So, expected value it's pretty similar it's pretty easy to understand, right? The expected value of let's say a you know a list of numbers for example, right?

[00:01:42]Well, the expected value of that list of numbers is going to be the summation, right? Of P of the probability of each you know number in that list or stream or whatever it is with X itself.

[00:02:01]Right? This is the expected value. This is essentially how it works, right? So, for example, let's say we have this input stream over here with the numbers 1 2 3 4 2 3 1. Right? So, what we see is this is a stream of 1 2 3 4 5 6 7 numbers. Right? So, it's a stream of seven numbers and one we see has 2/7 probability.

[00:02:33]Two, right? Has a 2/7 probability.

[00:02:37]Three has a 2/7 probability and four has a 1/7 probability. Correct? So, the expected value of this essentially like input stream over here, it's going to be well, the summation of each of these numbers times the probability of each.

[00:02:55]So, it's going to be or I mean, we can even say this in this example, for example, it's going to be 2/7 uh times one plus 2/7 times two plus 2/7 times three plus 1/7 times four.

[00:03:17]Okay? Great. So, we came across this concept of expected value. It's pretty easy to understand. So, how can we apply this to information or a stream of information? How do we get the average surprise factor of information from a specific source? We apply a very similar similar concept. In fact, all it is is just the expected value, but well, with subbing in information. So, H is equal to negative, right?

[00:03:49]Negative summation or actually, here. If we just plug it into this here, it'd be this.

[00:03:56]It's a summation uh from uh we have over here P I, right? The probability of each, you know, number, symbol, or whatever it is in that stream of information. And then we can sub in I P.

[00:04:13]Right? And then we see there's a negative sign here, right? So, we can put that negative out front. And then we come across the derivation of this.

[00:04:27]Okay, great. Now, you might be wondering, what the heck is the point of any of this? Well, I mean, think about it. Uh the thing is is when we have, you know, some sort of input stream or some sort of information, what are we really trying to do? We're trying to reduce the uncertainty, right? An AI model, what is it doing? Well, what it's doing is it's trying to reduce the uncertainty in each output, right? That it brings, right? We It It wants to be as uncertain or as less uncertain as possible. Now, we see here, entropy is really just a measure of uncertainty. In fact, in physics, entropy is known as disorder. In uh in a context of computer science, it would be known as uh a state of uh again, disorder or uncertainty, right? So, that is why the concept applies to AI models, because we're trying to essentially minimize entropy.

[00:05:28]And we'll We're going to see how uh that applies a little bit later. But, this is how the derivation of Shannon's entropy works. And we're going to see how useful that can be in terms of training AI models uh later with other concepts, like cross entropy.