How are large language models trained?

[00:00:06]You might have heard people say things like large language models are trained on massive amounts of text or large language models predict the next token in a sequence. But what does any of that actually mean? How were large language models trained in practice? In this video, we'll walk through the steps it takes to train large language models that behave as helpful assistants, follow instructions, use tools, and just generally respond with the right kind of vibe. Step one is pre-training. During this phase, the model is trained on a task called next token prediction.

[00:00:41]This essentially means the model is given a sequence of text and its task is to predict what comes next.

[00:00:47]One of the advantages of model pre-training is that it's a type of self-supervised learning, meaning you don't need to go through the effort of labeling the whole training data set.

[00:00:57]Instead, you take a lot of data from publicly available sources and you create the pre-training data set by chunking this data into inputs and outputs. For example, if this document about house plants was part of your pre-training data set, you could chunk it into a data set that looks something like this.

[00:01:18]The model's training task is to correctly predict the next word in the sequence.

[00:01:23]In order to do this effectively across a diverse set of inputs, the LLM implicitly learns patterns about language, writing styles, and even some world knowledge, like how the Monstera is an easy-to-care-for house plant.

[00:01:38]While the labeling strategy isn't super complicated, what is complicated about pre-training is doing this at scale.

[00:01:45]This phase of LLM training is a massive engineering challenge.

[00:01:50]Because models are too large for a single chip, engineers must use complex parallelism to split the model and data across thousands of interconnected GPUs.

[00:02:00]They also need to implement strategies for automated recovery systems to handle crashes and massive pipelines to pre-process the petabytes of training data.

[00:02:11]But, if you do all of that successfully, you have a model that can take any prompt and use its vast world knowledge, understanding of language, and sense of how different kinds of documents work to produce a response.

[00:02:24]Now, unless you're a developer who actually trains LLMs, it's unlikely you've interacted directly with a model that's only gone through pre-training.

[00:02:33]Even though pre-trained models, particularly the more recent and sophisticated ones, are fairly capable, they can also be unpredictable and unaligned.

[00:02:43]So, there's still a lot of work that needs to happen to get the LLM to be ready for use in a production environment. And that is where model post-training comes in.

[00:02:53]Post-training is the phase where we refine model quality and focus on alignment so that the generated answers are safe, well-formatted, helpful, more accurate, and pleasant to engage with.

[00:03:05]Often, post-training involves two different training strategies, supervised fine-tuning or SFT, and reinforcement learning or RL.

[00:03:15]In SFT, we train the model on gold-standard interactions that demonstrate the behavior we want it to learn.

[00:03:21]During RL, the model generates many answers and a separate reward model grades them.

[00:03:28]The reward model gives points for helpful, safe, and accurate responses, pushing the model towards the behaviors we want, and penalizes responses we don't want. Let's dive into both of these in more detail. So, let's say you wanted the model to learn how to create a friendly, conversational response to a question like a helpful assistant. To do this, you might create a data set of prompts formatted as questions like, "How do I make gluten-free muffins and the corresponding answers that demonstrate the helpful assistant style.

[00:03:58]During SFT, the model is fed the prompt, it produces a response, and then the loss is measured between the predicted response and the gold standard response, which is the label, and that's used to update the model's weights.

[00:04:11]This is basically classic ML model training, and by showing the model these ideal examples, it learns how to produce responses in line with the behavior we care about.

[00:04:21]You might follow a similar process for teaching the model how to use tools, creating a data set of prompts that require tools, and the syntax of the corresponding tool call.

[00:04:30]Or you could use SFT for improving the model's ability to solve math problems or really any other capability you want the model to improve on.

[00:04:38]Now, SFT is a good starting place, but it teaches the model to memorize what good looks like and doesn't always generalize to new, unseen variations of a task.

[00:04:48]To do that, we need a mechanism for the model to practice producing good responses and receiving feedback, iterating and improving over time. And that is where RL comes in.

[00:04:59]In RL, you have the model generate responses to prompts that are salient to the task you want the model to learn.

[00:05:06]The model's responses are scored by a separate model called the reward model.

[00:05:11]And based on the scores, the LLM updates its weights and improves over time.

[00:05:16]How you train this reward model is out of scope for now, but there are resources in the description below that provide some more details.

[00:05:23]One way of thinking about the differences between SFT and RL is that SFT sets the floor. It ensures that the model is at least competent. It understands the basic format of a list, a code block, or an answer to a question. RL raises the ceiling. SFT can only ever be as good as the curated training data, but RL allows the model to surpass this data by exploring millions of different ways to answer a prompt and keeping only the ones that get the highest scores, the model can find more efficient or clearer ways of responding to prompts than what was in the original SFT training set. In practice, the balance between SFT and RL is still an open question, and there's no standard ratio of how much to do either step.

[00:06:06]But while the exact training recipe will differ between different state-of-the-art models, the high-level components are the same. And at least for now, that means some combination of SFT followed by RL.

[00:06:18]So far, we've talked through the core components of how LLMs are trained. But there's one missing piece. How do we measure any of this?

[00:06:27]Anytime we train a model, we need to have some way of quantifying how good the model is at the task we care about.

[00:06:34]Are the experiments we're running actually improving the model's ability or just wasting valuable compute?

[00:06:40]Measuring success in pre-training is fairly straightforward.

[00:06:44]All we care about is whether the model is getting better at predicting the next word.

[00:06:49]But while pre-training focuses on how well does the model predict the next word, post-training focuses on questions of alignment, like how well does the model follow instructions and behave like a safe, friendly, and helpful assistant.

[00:07:03]And all of that can be a little harder to measure. For closed-domain problems like math or coding, you might test the model's abilities on a data set of problems and check if the model produced a correct answer.

[00:07:15]This is the score you track towards to see if SFT and RL are actually improving the model skills. The more questions in your test data set that the model gets right, the better it's doing.

[00:07:26]For measuring things like is the model being collaborative or insightful or helpful or any other task that's hard to quantify, you first need to have a clear definition of what those terms mean and what would constitute a good response.

[00:07:40]For example, here's how you might define what it means for a model to be helpful.

[00:07:44]When you have that defined, you can use this criteria to evaluate the model's response.

[00:07:49]First, you curate a test set of relevant prompts.

[00:07:52]In the case of helpfulness, pretty much any prompt could be considered relevant because we always want a model to respond in a helpful way.

[00:08:00]But for measuring something like, "How funny is this model?" you'd want to make sure that your test set is made of prompts where you actually expect the model to respond in a humorous way.

[00:08:09]You're not going to get a great signal on how good the model is if the test prompts have nothing to do with humor.

[00:08:15]Once you have this test set of prompts, you pass it to the model you're evaluating and you get back responses.

[00:08:20]You then score those responses against the criteria that you defined earlier.

[00:08:25]In practice, you probably wouldn't do this completely manually. Instead, you would scale the process using a second LLM that works as a judge and does the scoring for you and have human spot-check the results.

[00:08:37]This judge model, known as an auto rater, evaluates the LLM's responses to a prompt and scores it according to your criteria.

[00:08:45]As you try out different data sets and training strategies, you'll use this evaluation to help you determine if you're on the right path and if the model is indeed getting better over time.

[00:08:55]You still definitely want to spot-check the results and do more robust testing before deploying the model in production, but using an auto rater with a set of relevant prompts and defined criteria gives you a number that quantifies the model's ability and can be computed relatively quickly so you can track progress through post-training. Putting all of this together, we start with model pre-training where we train LLMs to do next token prediction on large amounts of text.

[00:09:22]This initial training phase equips the model with a broad understanding of patterns, features, and relationships within the data.

[00:09:30]Then, we take this pre-trained model and we apply post-training techniques where we refine the quality, consistency, and alignment of the model so that it responds as a conversational pleasant assistant with improved skills in areas such as reasoning or using tools. A lot of data, experiments, and compute later and we have a trained large language model.

[00:09:54]Let me know in the comments what you want to learn about next. I'll read the research papers so you don't have to.

[00:10:03]>> [music]