Every Step of Energy Based Image Generation Simply Explained | Unadjusted Langevin Algorithm

[00:00:00]Did you know images can come out of noise in more than one way? A lot of people now know the basic story of diffusion models. Start with noise, then gradually remove it until an image appears. That's one of the most famous ideas in modern generative AI. But it's not the only way. There is another route that feels more like physics. Imagine TV static slowly turning into a handwritten digit, a three or a seven. But instead of saying at each step remove a little noise, we say something stranger. Let the image roll downhill through an invisible landscape. That is the central idea behind energy-based models often called EBMS.

[00:00:40]In this video, we're going to unpack every step. what this energy landscape means, how a neural network learns it, how randomness actually helps instead of hurts, and how pure noise can eventually become a recognizable image. Along the way, you'll also see something beautiful, a method that grew out of physics can be used to train AI systems.

[00:01:04]>> A quick pause before we go further. This video has a twin on my other YouTube channel that lets you dig into the actual math, see the code, and generate your own images. I recommend watching this video to grasp the concepts. Then head over to the other video, download the GitHub repo, and start learning in depth for free. Now, back to the video.

[00:01:24]The big intuition starts with a landscape. Picture a hilly terrain with valleys, slopes, and mountains. In this picture, valleys represent good image patterns, images that look like real handwritten digits. High ground represents bad image patterns, images that look like random static or meaningless blobs. The word energy here is just a score. Low energy means the model thinks an image looks natural or realistic. High energy means the model thinks the image looks unnatural or unlikely. That's the first major shift in thinking. An energy-based model does not try to directly draw a digit. It doesn't say here is how to produce a five stroke by stroke. Instead, it learns to judge images. It learns a rule like this. Real images should sit lower in the landscape than fake-looking ones.

[00:02:17]This is a powerful idea because it turns image generation into a search problem.

[00:02:22]Rather than asking how do I create a digit from scratch, we ask where are the low valleys in image space and how can I move toward them. Before we move further away, let me clear up a common misunderstanding. Energy here does not mean electricity or power usage. It's much closer to how physicists use the word in statistical physics. Lower energy states tend to be more probable, especially when randomness and thermal motion are involved. That same relationship shows up here. Low energy images are the ones the model wants to treat as more probable. Now we need to ask what exactly is the space this landscape lives on. If we use IMNEST, the famous data set of handwritten digits, each image is 28 pixels x 28 pixels. That's 784 pixels total. So each image can be thought of as a list of 784 numbers. This means every image is a single point in a 784dimensional space. That sounds wild because we can't visualize 784 dimensions directly. But the idea is simpler than it sounds. In ordinary 2D space, a point has an x coordinate and a y-coordinate. In 3D space, it has x, y, and z. Here an image has 784 coordinates, one for each pixel value. If you change one pixel, you move in one direction in this giant space.

[00:03:49]Change a different pixel and you move in another direction. So the landscape is drawn over the space of all possible images. And that's an important concept.

[00:03:59]A handwritten five is not special because of one individual pixel, but because the entire pattern of 784 numbers lands in a region of this gigantic space that the model can learn as a low valley.

[00:04:14]Real digits cluster in certain structured regions. Random noise is scattered all over the place, usually up in the bad high energy terrain. At this point, a natural idea appears. If low energy images are better, then maybe we can turn these energies into probabilities. Images in valleys should be more likely. Images on mountain tops should be less likely. That's exactly the goal. But now we hit a huge obstacle to convert energy scores into actual probabilities. You have to compare one image against all possible images. You need a giant normalizing quantity that makes all the probabilities add up correctly. In physics, this quantity is called the partition function. Usually written as Z. Z is like the total weight of every possible image pattern. Once you know it, you can say exactly how probable a particular image is relative to all the others. The trouble is that in 784dimensional image space, computing the partition function Z is hopelessly hard. There are unimaginably many possible images. Even if you used a very coarse grid far rougher than real pixel values, the number of possibilities would still explode beyond anything you could calculate directly. So this becomes the central challenge. How can we work with the model without ever computing Z? That is the heart of energy based modeling.

[00:05:43]And the escape route comes from physics.

[00:05:45]If we can't count every possible state, maybe we can use the slope of the landscape instead. To understand that, imagine a tiny grain drifting in water.

[00:05:55]If the grain sits on a slope, it tends to slide downhill. But it does not move perfectly smoothly because water molecules keep bumping into it from random directions. So, its motion is a mix of two effects, a push downhill and random jiggling. Now, replace the grain's position with the current image.

[00:06:15]Replace the slope of the hill with how the energy changes when you tweak pixel values slightly. Then the story becomes the image gets nudged toward lower energy where more realistic patterns are. But it also gets randomly jiggled so it can keep exploring. That noisy downhill motion is the basic idea of dynamics, a tool from physics. It describes how systems move when they are influenced both by a force and by randomness. And here is an important subtle point.

[00:06:45]The randomness is not a bug. It is essential. If you only moved downhill greedily, you would likely fall into whichever nearby hole you happen to start near and stop there. The random kicks help the system explore nearby possibilities and avoid becoming too narrow-minded too quickly. So, how do we actually sample images with this idea?

[00:07:06]The rule is surprisingly simple. Repeat this many times. Look at which direction is downhill. Take a small step that way and then add a little random noise.

[00:07:17]That's the essence of the unadjusted language algorithm. Intuitively, the downhill step means move towards something more realistic. The noise step means don't just collapse immediately.

[00:07:28]Keep exploring the neighborhood. If you wanted to write the update in plain English, it would look like this.

[00:07:35]Next image equals current image minus a small downhill move plus a small random kick. Each part has a job. The current image is where you are now. The downhill move uses the slope of the energy landscape to improve realism. The random kick injects exploration. And here comes a really important insight. We only need the slope of the energy landscape, not the full normalized probability formula.

[00:08:06]For our generation purposes, we never had to calculate the impossible partition function Z. Why? Because slopes are local. If you add some giant constant to all energy values or if your probability formula includes a global normalizing term, the local downhill direction does not change. The partition function affects the absolute probability scale but not the slope with respect to the pixels of one image. That is the trick that makes energy based sampling possible. Exact probabilities are out of reach, but local directions are enough to guide a random walk toward realistic images. Of course, we still need the energy landscape itself. Where does it come from? This is where machine learning enters. We do not handdesign the energy landscape. Instead, we let a neural network learn it from examples.

[00:08:58]The setup is simple. The input is a 784dimensional image. The output is a single number, the images energy. A convolutional neural network is a natural choice here because handwritten digits are made of local patterns, edges, curves, short strokes, loops. Convolutions are very good at spotting exactly those kinds of local visual features. So the network can gradually learn what combinations of strokes tend to look digit-like. Before feeding images into the model, the pixels are often rescaled to lie between -1 and + one. Centering the data this way tends to make learning and sampling more stable. Since the lange van sampler keeps adjusting pixel values, it helps if the image values live in a nicely balanced range. Another practical detail is the activation function. Instead of using something sharp and kinkedked like Rayu, we often use a smoother activation such as Celu. The reason is intuitive.

[00:09:57]The sampler depends on slopes. If the landscape is smoother, the downhill guidance is cleaner and less jumpy. At the very beginning of training, the model has not learned anything useful yet. So, if you feed it a real handwritten digit and a random noise image, it may assign them very similar energy scores. That's expected. The landscape starts out basically unshaped.

[00:10:19]Training then has a very clear goal.

[00:10:22]Push real digits down and push fake samples up. In one sentence, training teaches the model to assign lower energy to real data and higher energy to the images produced by its current imperfect sampler. Suppose the model sees a real eight and some noisy blob. The real eight should end up lower in the landscape. The blob should end up higher. So at each training step, we compare two groups. Real Mness digits from the data set and nonsense or imperfect images produced by the model's own sampler. The update rule is conceptually simple. Lower the average energy of real images. Raise the average energy of model generated images. That's enough to shape the landscape. Notice what the model is not doing. It is not learning labels like this is a three or this is a seven. It is not mainly solving a classification task. Instead, it is learning from contrast. Make reality cheaper. Make your current mistakes more expensive. Also, it is not just memorizing exact training images.

[00:11:24]The goal is broader than that. The model is shaping a whole energy surface over image space, trying to carve out low valleys around realistic patterns and higher regions around unrealistic ones.

[00:11:36]There is a subtle training problem.

[00:11:38]Every training step requires model generated images, also called samples, because those are the images we want to push upward in energy. How do we get those samples? A first idea is start from pure noise every single time and run language dynamics. That sounds clean, but it is often too slow. If you restart from scratch at every training step, it may take many language steps before the sample gets anywhere near a realistic digit. A second idea is start from real training images. That sounds efficient, but it creates another problem. If you begin near a real digit, the sample may barely move. Then the model mostly sees images very close to the data and never really explores the wider image space. That can fool the model into a false sense of success. It might learn to shape the landscape only near real digits while failing to assign high energy to truly junky images far away from the data. Good training needs something in between. The samples should be somewhat realistic so they are informative, but they should not always stay glued to the true data. A clever practical fix is to use a persistent replay buffer. Here's the idea. Keep a large pool of images in memory. Instead of restarting every sample from scratch during every training step, store the current sampler states and keep updating them over time. A good analogy is to imagine thousands of hikers moving around the landscape. If every day they teleported back to random starting points, they would spend most of their time lost in bad regions. Instead, let them continue from where they were yesterday. Over time, they accumulate many steps and explore much more efficiently. This works because each buffer image gets a few more langu steps at every training iteration. Across many training rounds, those steps add up, so you get stronger samples without paying the full cost from scratch each time. To keep the system honest, you also occasionally replace a small fraction of buffer images with pure noise, maybe around 5% of them. That way, the model is forced to confront ugly, unrealistic inputs, too, instead of only refining already decent samples. In practice, people might use a buffer of around 8,192 images and run about 60 lounge steps per training iteration. The exact numbers are not the main point. The big idea is that the replay buffer makes EBM training much more workable in the real world. So what does the optimizer, the thing adjusting the neural network weights, actually see? A convenient training loss that the optimizer has to minimize to train the model is just this average energy on real images minus average energy on model images, also called samples. That's a simple surrogate loss, meaning a practical standin for the deeper theory. If this quantity goes down, it usually means the model is pulling real data lower relative to its generated samples. One more technical detail matters here. When we compute this loss, we usually treat the sampled images as fixed snapshots.

[00:14:52]In deep learning language, we detach them. That means we do not back propagate through the entire history of all the long sampling steps. In plain English, the optimizer is told, "Here are some current samples. Compare them to real data and update the energy function accordingly." It does not need to trace through every random kick and every tiny movement that produced those samples. That keeps training manageable and lets automatic differentiation software handle the bookkeeping without blowing up the computation. There is also often a small regularization term, a tiny penalty that stops energy values from drifting to absurdly large magnitudes. Without this, the model can sometimes become numerically unstable.

[00:15:36]And now after training the energybased model we arrive at the payoff watching a digit emerge from noise. Take a completely random image just static. Now run long dynamics for a long time maybe 2,000 steps. At first the image looks meaningless. Then faint strokes begin to appear. A curved region gets slightly darker. A vertical line starts to stabilize. Over more steps. The shape becomes clearer. Finally, a recognizable digit settles in. One advantage of the energy-based models is that you can interpret them directly. The image is drifting into a low energy valley of the learned landscape. The model is not drawing in the usual artistic sense. It is not planning a picture stroke by stroke with a pen. It is searching, moving through 784dimensional image space toward patterns that its learned physics says are natural. That's a big conceptual shift. Generation here does not mean predict the next pixel or decode a hidden vector. It means find a point in image space that lies in a low energy region. A final sanity check is to ask, did the landscape really learn to separate digits from noise? Before training, if you plot energy histograms for real digits and for random noise, they may overlap heavily. The model cannot yet tell the difference.

[00:17:02]After training, those histograms should separate. Real digits should mostly receive lower energy. Random noise should mostly receive higher energy.

[00:17:12]Intuitively, that means the model has succeeded in digging valleys where real digits live. Random junk now sits higher up the terrain. Visually and conceptually, a good model makes real data feel comfortable and stable, while noise feels uncomfortable and unstable.

[00:17:31]There is an important caveat though.

[00:17:33]Separation alone does not guarantee perfect image generation. A model can distinguish data from noise better than chance and still have imperfect sampling behavior. But strong separation is still meaningful evidence that the energy function has learned something real.

[00:17:51]Stepping back, why does all this matter beyond the mnest handwritten data set?

[00:17:56]Imnest is just a sandbox. It is a simple environment where we can clearly see the ideas at work. The deeper principles go much further. Energy-based models offer a different philosophy from direct generators. Instead of learning how to draw, they learn what good examples look like. They define a landscape of preference over possible inputs. This connects deeply to physics. Energy landscapes, thermal randomness, equilibrium behavior, and stochastic motion are the mathematical backbone of the method. There is also an interesting connection to other modern AI systems.

[00:18:35]Diffusion models also involve noise and gradual refinement, but EBMS place the main emphasis on a learned energy function and on sampling by noisy dynamics over that energy landscape. So in the end, the whole picture is surprisingly elegant. We built a machine that can create images by combining two ideas. A neural network that scores how realistic an image is by assigning it an energy and a physics inspired random walk that explores image space and hunts for low energy patterns. I hope you enjoyed this video. And one more thing before you go, if you would like to work with me directly, I offer paid one-on-one help for various needs. Maybe you are trying to break into R&D as a data scientist, build a standout portfolio project, or implement physics based models in your work. If so, the link in the description lets you book a call. Take good care of yourself, and I'll see you in the next video.