MoE Training Explained (Part 1): How Mixture of Experts Models Actually Learn

本站添加: 2026-05-27

[00:00:00]Let's talk about how mixture of experts models are trained. You know, the intuition behind it, not really the math behind it, but just the intuition how they how do they get trained? Now, MOEs are these new kind of neural networks where you use a collection of [music] experts instead of using a single neural network, you can think about the neural network as a collection of experts which are not all used to answer a question.

[00:00:26]Only a subset of them are used to answer a question. And for example, when you're asking a medical question, it is a different question than, you know, a legal question in real estate. So, the expertise you need for answering the medical question is different a different expert can answer that question. Now, the question becomes, you know, how do you train such a network?

[00:00:46]Do you actually come up with, you know, this expert will handle medical questions or scientific questions and this other question one will answer business questions. No, actually that is not how it is trained because in the beginning we uh humans you know, humans do not make that judgment. The people who are training these models, they do not make the judgment uh what exactly these expertises would be. They just come up with the idea that there will be nine experts, okay?

[00:01:14]Now, let's say uh you know, they start with random weights and there is something called a router. The router's job is to decide given the question, right? Given the tokens and decides which of the [music] experts to choose.

[00:01:26]And usually, you know, out of let's say there are nine experts, they could be choosing only two or three experts [music] to actually answer any question, right? And one of these experts could be a generalist which has knowledge about uh all the domains and the other experts could be specialists.