How Entropy Can Stop AI Training From Exploding

[00:00:00]Imagine training a very large AI model for weeks. The hardware and the power bill are expensive and every additional hour carries a real cost. Early on, everything can look completely normal.

[00:00:12]The error keeps coming down, the curves look clean, and it appears that the model is learning exactly the way it should. Then, the whole process can fail very quickly. The loss jumps, the model's outputs collapse into nonsense, and training blows up, effectively ruining the entire run.

[00:00:30]>> It might sound like pure bad luck, some bizarre one-off failure.

[00:00:35]But one of the stranger things modern AI has taught us is that these collapses often aren't accidents in any meaningful sense.

[00:00:42]In transformer models, the architecture underlying systems like chat GPT and many image and speech models, training can be fragile by default. A very small change, as simple as pushing the learning rate a bit too high, can move a model from making steady progress to complete breakdown.

[00:01:00]>> What's striking is that researchers have begun to pin down why this happens. The clue turns out to be a concrete internal quantity, something you can actually measure, and one that behaves a lot like an advanced warning sign. Better still, there is a fairly direct way to control it.

[00:01:18]If we limit how much the model can stretch information as it moves from layer to layer, we can control this quantity, too. To see why this becomes a problem, it helps to start with what training is actually doing.

[00:01:30]A neural network contains an enormous of adjustable parameters, millions or even billions of small settings. Training is the process of changing those settings little by little so that the model makes fewer errors.

[00:01:44]One useful way to picture this is as moving downhill through a foggy landscape, always trying to take the next step in whatever direction seems to reduce the error. That picture is helpful, but it can also point us in the wrong direction.

[00:01:57]It's easy to look at a loss curve coming down and assume the model is steadily making its way toward a good solution.

[00:02:04]But, that intuition does not always hold. You can have a system that is decreasing its loss while still moving toward failure.

[00:02:11]The descent can look smooth and well-behaved right up until the point where the entire process becomes unstable. A useful way to think about it is a pencil standing on its tip.

[00:02:21]For a short moment, it can appear completely motionless. Nothing obvious is going wrong. But, that calm is misleading. The pencil is not actually in a safe state. It's sitting in an arrangement where even the tiniest disturbance is enough to send it toppling over.

[00:02:37]Some AI training runs behave in much the same way. The metrics look clean and uneventful, but beneath those smooth curves, the model may already be sliding into a brittle and dangerous regime.

[00:02:48]What, then, is actually shifting under the hood?

[00:02:52]To answer that, we need one of the core ideas behind transformers, attention.

[00:02:56]Before we zoom into attention, quick note. If you want to go deeper on physics-based machine learning, come join our Discord server where we'll learn physics-based ML, share resources and papers, and pressure test new ideas.

[00:03:09]And it's not just discussion. We're also collaborating to build next-generation AI models together. The link is in the description.

[00:03:18]All right. So, with that in mind, let's talk about attention and why it can act like an early warning signal for training instability. Attention is the mechanism that allows the model to determine which parts of the input are important when interpreting a word, an image patch, or a segment of audio.

[00:03:34]Take the sentence, "The trophy didn't fit in the suitcase because it was too big."

[00:03:39]What does it point to here? The trophy or the suitcase?

[00:03:45]Most people can infer that it refers to the trophy. A transformer does something similar. It looks back over the earlier words, weighs them against one another, and decides which ones matter most for resolving the meaning. This is why people so often talk about attention as a spotlight.

[00:04:03]At each word, the model casts a kind of internal beam back over the context that came before.

[00:04:10]In a well-behaved model, that beam is usually not razor thin. It may land more heavily on a few words that matter most, but it still keeps multiple candidates in play.

[00:04:21]What it is not doing is locking onto a single position with blind confidence.

[00:04:27]The first big idea is this. The width of that spotlight is not just a vague intuition.

[00:04:34]We can actually put a number on how wide or narrow it is. The quantity we're focusing on here is attention entropy.

[00:04:42]Entropy, originally introduced by physicists, also appears in machine learning using the same formula as in physics. It is a measure of how spread out something is. If the entropy is high, attention is being distributed over many different positions, more like a broad and gentle spotlight. If the entropy is low, that attention has collapsed onto a very small region, more like a laser locked onto a single point.

[00:05:08]If that still feels a little abstract, take studying for an exam as a concrete case.

[00:05:13]Real understanding means your attention has been distributed across the important ideas and the relationships between them.

[00:05:21]Collapse is the opposite. You've locked onto one sentence, memorized it exactly, and let everything else drop away.

[00:05:29]That can produce a brief feeling of confidence, but the result is fragile knowledge. Change the question even slightly, and the whole thing breaks.

[00:05:37]This is very close to what researchers observed inside unstable transformers.

[00:05:42]Right before training starts to go off the rails, attention is often no longer spread out and adaptable. It contracts.

[00:05:50]Then it contracts again. In the extreme case, nearly all of the weight can end up concentrated on a single token or a single patch. The model becomes fixated.

[00:06:01]There's a name for this phenomenon.

[00:06:03]Attention entropy collapse.

[00:06:06]Why is that dangerous? Because a transformer does its work by pulling together information from many different places.

[00:06:13]If the attention pattern collapses down to a single sharp peak, the model stops combining context in the richer way we want. Instead of weighing several useful signals at once, it starts acting like a system that has fixated on one detail and tuned out everything else. Once this starts happening, the updates during training can become unstable.

[00:06:33]The gradients, which are the signals that tell the model how to adjust its weights, grow more erratic and more sensitive. Very small changes in the input can lead to disproportionately large changes in the update.

[00:06:45]Instead of making steady progress, the training process begins to overreact.

[00:06:50]The loss might start bouncing up and down, or it may abruptly spike upward.

[00:06:54]What's surprising is not just that this low entropy regime appears near instability, but how reliably it does.

[00:07:01]Change the task, data set, or transformer variant, and the same pattern keeps showing up. Right before failure, or sometimes exactly as failure is setting in, attention becomes sharply peaked. But that leads to a basic scientific question. Is attention collapse actually the thing driving the failure, or are we looking at something downstream, more like smoke that appears after the fire is already burning?

[00:07:27]This is where things start to get interesting. The researchers added a temperature parameter to the attention mechanism. This idea shows up all over science, but here you can think of it as controlling how sharp or diffuse the attention pattern becomes. Turn the temperature up, and the attention weights spread out becoming softer. Turn the temperature down and they narrow becoming sharper and more concentrated.

[00:07:49]A useful way to think about it is like adjusting a camera lens.

[00:07:53]Turn it in one direction and the image softens with individual details blending. Turn it in the other and everything snaps into sharp almost unforgiving clarity.

[00:08:03]Attention behaves in much the same way.

[00:08:05]As we lower the temperature, the distribution becomes more concentrated sharpening the focus until a single location can nearly take over entirely.

[00:08:13]So the researchers ran a version of the same experiment but with the process intentionally broken at specific points.

[00:08:20]During selected moments in training, they lowered the temperature on purpose so the models attention would become more concentrated.

[00:08:27]Put more directly, they were deliberately triggering entropy collapse.

[00:08:32]And what happened? Right on schedule, training started to fall apart. That suggests the collapse is not just a byproduct of whatever else is going wrong. It is one of the things actually driving the failure.

[00:08:44]If we force attention to collapse, we can make training unstable.

[00:08:49]That takes the argument from these two things happen together to something much closer to a causal story. So now we can ask the more interesting question. Why does that collapse leave the whole system so fragile?

[00:09:01]To see why, picture the models error surface again as a piece of terrain.

[00:09:06]Some regions are broad gentle valleys where you can take a step and still be moving in basically the right direction.

[00:09:12]Other regions are narrow ridges or steep drop-offs. In those sharper parts of the landscape, a step that looks small can be enough to throw you far off course.

[00:09:21]Researchers found that attention collapse happens when the model drifts into steeper more fragile parts of the landscape.

[00:09:28]As attention gets more and more concentrated, the optimization problem becomes risky.

[00:09:33]The model is no longer moving through a broad stable valley. It is now making its way along the edge of a cliff.

[00:09:40]Timing matters here, too. If collapse shows up early in training, the model can run straight into an almost vertical wall before it has formed any durable internal structure. And from there, it may not come back at all.

[00:09:53]If the same thing happens later, there is sometimes enough structure already built up for the model to pull itself back out.

[00:10:00]But even in that case, the situation is still risky. This helps clear up another version of the same mistake. People often take a larger model to mean a more stable one simply because it is more capable.

[00:10:13]But greater capability is not the same thing as greater robustness.

[00:10:18]As a system grows, it can also develop stronger internal feedback loops.

[00:10:24]And if those loops start amplifying themselves instead of damping out, the larger model may end up being even more sensitive. The natural next question is what sets that self-reinforcing cycle in motion in the first place?

[00:10:39]Let's go one step further into a part of the model that sounds more mathematical than it really is.

[00:10:45]Inside the model are matrices, which are just collections of numbers that take one set of signals and turn it into another.

[00:10:53]The simplest way to think about a matrix is as a kind of mixing mechanism.

[00:10:58]Information comes in and the matrix recombines it into a new form. Now, take the most extreme version of the same question. How much can this machine increase a signal?

[00:11:09]The largest possible amount is called the spectral norm.

[00:11:13]The name itself is not especially important. What matters is the picture behind it. It tells us the biggest stretch this layer can apply. If you imagine a rubber sheet covered with little arrows, you can get a pretty good picture of what's going on here.

[00:11:27]Pull on the sheet and not every arrow changes by the same amount. Some barely get longer while others stretch much more.

[00:11:35]The spectral norm is just the largest of those possible stretches, the maximum amount the sheet can expand something in any direction. Why does that matter?

[00:11:44]Because once some layers begin stretching their signals too much, the attention scores that come later can get pushed to extremes.

[00:11:52]When those scores become extreme, attention turns sharply concentrated.

[00:11:57]Sharply concentrated attention means lower entropy.

[00:12:01]Lower entropy gives us unstable spiky gradients. And those unstable gradients can in turn drive the weights toward even more aggressive stretching. That is the runaway loop. Stretching things further doesn't just make the situation worse in one step. As the stretching increases, attention becomes more acute.

[00:12:21]That more acute attention makes each new update land with more force.

[00:12:26]Those harsher updates then push the stretching even further and the process repeats. What you get is a positive feedback loop.

[00:12:35]The easiest way to picture it is a microphone placed too close to a speaker.

[00:12:39]A tiny sound gets picked up, amplified, fed back into the system, amplified again, and very quickly turns into a painful squeal. Modern optimizers, the algorithms that update a model's weights, are incredibly powerful. In many cases, that power is exactly what makes training go faster and work better.

[00:12:58]But that same ability to adapt quickly can also create problems. In some settings, if there is nothing keeping things in check, rapid updates don't just move learning along, they can also amplify unstable directions and feed the loop further. The question then is how the cycle actually gets broken.

[00:13:15]The technique is called sigma reparam, but the label is not really the important part. Here is the idea.

[00:13:22]We want each layer to stop arbitrarily amplifying the signals that pass through it. The method works in a fairly direct way. We measure how much a layer can stretch its input, essentially the maximum amount of amplification it can produce, and then rescale that layer so this stretching stays bounded.

[00:13:40]The practical effect is that the unstable runaway behavior gets suppressed, while the model still keeps the part we actually want, the ability to learn useful transformation. One useful way to picture it is as a tether.

[00:13:53]The layer still has room to move. It can still grow stronger or weaker in whatever way helps with the task. What it cannot do is pull on the signal so aggressively that the entire system starts to come apart. Take a car engine with a governor.

[00:14:07]The engine still makes power and picks up speed. What the governor does is stop the engine from revving itself into failure.

[00:14:15]You're not rebuilding the car or changing what it is fundamentally, instead putting a limit in place to avoid one particular way it can break.

[00:14:23]What makes this approach feel so clean is that it does not appear to meaningfully restrict what the model can express.

[00:14:29]It is not imposing a blanket demand for simplicity or forcing the model into some weaker class.

[00:14:36]The point is narrower than that. The model can still be rich and flexible, just not in the particular direction that would allow this amplification loop to run away and concentrate all of the attention there. And there is a second, more satisfying part of the story here.

[00:14:50]Because this is not just a fortunate trick that happens to work in training.

[00:14:54]If we can put a bound on how much each layer is allowed to stretch the signal, then we can also put a bound on how concentrated the attention mechanism is allowed to become.

[00:15:04]In plain language, if we keep the amount of amplification under control, attention cannot collapse into an arbitrarily sharp spike. That matters because it changes what kind of intervention this is. It's not a matter of cycling through a bag of arbitrary tricks and keeping whichever one appears to help. It is much closer to placing a hard constraint on the system. One that follows from the model's own internal structure. In practice, this ends up mattering a great deal. Training large transformers usually comes with the same familiar collection of stabilizing techniques. Carefully chosen warm-up schedules, particular normalization setups, specific weight decay values, and special optimizer selections.

[00:15:47]Very often, these do work. But, they can also look like incantations.

[00:15:52]Leave out one piece and the whole run collapses. When attention collapse is kept from happening, a lot of those supporting tricks stop being quite so central.

[00:16:01]They do not become useless in every case, but they are no longer the only things keeping the whole setup from falling apart.

[00:16:09]Under those conditions, models are able to train reliably across configurations that would otherwise be unstable or simply fail. And this is not some narrow effect that only shows up on a single benchmark. Variants of this same approach have helped across image recognition, self-supervised learning, speech recognition, and machine translation.

[00:16:30]Even very deep transformers, on the scale of roughly 100 layers, which are famously difficult to optimize, become far easier to train. For a long time, training instability in AI has had this slightly superstitious quality to it. A run works or it doesn't. We adjust a few hyper-parameters, launch the experiment again, and see what happens. What this work points to is that at least part of that apparent randomness isn't randomness at all. It's the result of an internal failure mode that can be observed, tracked, and brought under control. Attention entropy works a bit like a pressure gauge.

[00:17:04]When it starts to collapse into a much narrower range, it can be an early sign that the system is under strain and the boiler is close to failing. And that is a very compelling way to frame the future of AI systems.

[00:17:17]As our models become larger and more expensive, relying on intuition, accumulated lore, and repeated trial and error is no longer enough.

[00:17:27]What we need are training methods that remain stable since now stability is built into them. Thanks for watching.

[00:17:34]Until the next video, take good care of yourself.