LLM Fine Tuning Tutorial (Free Labs)

Added: 2026-05-06

[00:00:00]We know that the AI industry already spent billions of dollars building LLMs like GPT4, Claude, and Gemini that can do just about anything. But here's the ironic part. These models are built to be a generalist, not a specialist. We have millions of users across countless use cases where over 10% of the world is now using Chacha PT, for example. But when we look at how agents are starting to become more prevalent where they're now being applied for very specific tasks like only responding in JSON formats or acting as a drive-through agent that never breaks character or even playing an NPC in a game that speaks in medieval English, we need the interaction to be very consistent. And you might be wondering, wait, why can't we just use prompt engineering? Prompt engineering is basically the art of crafting the perfect instructions to get LLMs to do what they want. And for most use cases, they probably work great. But here's the problem. Prompts can be hacked. Users can inject instructions that override your carefully crafted system prompt. And more importantly, prompts don't actually change how the model behaves at its core, which means you're just hoping the model follows your instructions. Fine-tuning on the other hand actually modifies the model's parameters. You're essentially retraining the model on your specific data to embed domain specific knowledge or behavior directly into how it thinks.

[00:01:28]And here's something that most people don't realize. Chacht and Gemini that we use every day are actually already fine-tuned to be used in chat settings.

[00:01:37]Which means if you want to create a very specific agent for tasks other than chat or trying to create an agent for specific role, fine-tuning them might actually be a better option. This idea of fine-tuning was popularized by OpenAI when they introduced something called RLHF or reinforcement learning from human feedback where they hire humans to actually score different outputs. Let's look at some real use cases. If you need guaranteed JSON outputs for your API, fine-tuning can teach the model to always respond in that exact format. If you're building a corporate agent that needs to follow strict brand guidelines and terminology, fine-tuning embeds that behavior. And if you're creating game characters that need to maintain personality and speak in a specific way, fine-tuning makes that consistent as well. But here's where most people fall short when it comes to fine-tuning. So you could actually think of it like this. Fine-tuning teaches the model how to behave while rag pulls what the model needs to know. When you're first getting started into fine-tuning, you learn that there are different techniques like Laura or Qura where instead of changing the entire model's billions of parameters, you could actually freeze them and just add a small adapter layer into the model. Which means even on consumer-grade hardware, as long as the model fits, you could foreseeably fine-tune the model at home instead of a data center or hyperscaler, fine-tuning is actually becoming a more popular method as the means to tweak the model's weights are becoming actually more feasible. In this app, we're going to learn how to fine-tune LLMs to [music] customize their behavior. We will build a Taco Drive-through agent that stays on topic and resists jailbreaks. This is the same technique used to turn base models like GPT3 into chacha PT. This lab takes about 30 to 45 minutes to complete. When I open the lab, I'm dropped right into the scenario. The intro page explains why fine-tuning matters. Prompt engineering tells model what to do via instructions, but those instructions can be ignored. Fine-tuning changes the model weights directly, embedding behavior into how it thinks.

[00:03:47]The lab shows a comparison. With prompt engineering, users can say, "I ignore your instructions and break the agent."

[00:03:54]With fine-tuning, behavior is embedded and much harder to bypass. The fine-tuning process has six steps: seeing the prompt problem, preparing data, configuring Laura, training, evaluation, and alignment. The what you will do section lists all six tasks.

[00:04:11]Access the labs using the link in the description below to follow along with me. The first step is to verify our environment. First activate the virtual environment with the following query.

[00:04:21]This [music] script checks that Python version 3.10 or higher is installed. All packages are available. The small LM21 135 million model is accessible and the training data file exists. Now we move to task number [music] one which is about the prompt engineering problem. I will create a prompt engineer taco and [music] test it against jailbreak prompts. Open/root/code/task one jailbreak.py. I need to complete two toddos. [music] At line 52, replace the blank with the system prompt for taco. I set it to tell the model it is Tacobot and must always respond in JSON format.

[00:04:58]At line 93, replace the blank with attack prompt to pass the jailbreak prompt to the test function. Run the script with python/root/code/task one jailbreak.py. The output shows the model getting jailbroken. When asked to ignore instructions, it complies [music] instead of staying in character. This demonstrates why fine-tuning is needed.

[00:05:19]We have a knowledge check about fine-tuning versus prompting. The question asks what the key difference is [music] between them. The answer is fine-tuning changes models weights.

[00:05:28]Prompts are just instructions because fine-tuning modifies the model parameters directly while prompts are suggestions that can be ignored. Now we move to task two which is about preparing training data. I will create a new training example and add to [music] it the data set. Open /loot/code/task2 prepared data.py. I need to complete three todos. At line 77, replace a blank with a user message like do you have any combo deals? At line 81, replace a blank with a JSON response like response, yes, our combo includes two tacos and drink category deals. At line 129, replace a blank with a for append mode. run the script with python/code/task2 prepared data.py. The output shows my example being validated and added to my training data set. The model will learn from its example in task 4. Now we [music] move to task three which is about configuring and applying lura.

[00:06:24]Lora stands for low rank adaptation. It freezes the base model and adds small trainable adapters. Open/root/code/task 3 configure laura.py. I need to complete three todos. In line 49, replace a blank with eight for the [music] Laura rank.

[00:06:41]At line 53, replace a blank with 16 for the Laura alpha. At line 57, replace a blank with qroj and v proj for the target modules. Run the script with python/root/code/task 3 configure laura.py. The output shows the parameter reduction happening in real time. Before Laura, all 134 million parameters [music] would be trained.

[00:07:05]After Laura, only 460,000 parameters are trainable. That is a 99.7% reduction.

[00:07:13]The memory saving go from about 1,500 megabytes to just about 5 mgabytes. We have a knowledge check about Laura. The question asks what Laura freezes during training. The answer is the base model weights because Laura keeps all original model weights frozen and only train small adapter matrices added on [music] top. Now we move to task four which is about training with Laura. I will actually fine-tune the model and watch the loss decrease. Open/root/code/task 4 train Laura.py. I need to complete two toddos. In line 107, replace the blank with 50 for the number of training steps. At line 111, replace the blank with 2 e to the^ of -4 for the learning rate. Run the script with python/root/code/task fort train laura.py. Training takes about 5 to 8 minutes on CPU. The output shows the progress bar and the loss decreasing with each step. When training completes, this adapter is saved to /root/ Laura adapter. The adapter size is about 2 mgabytes compared to 500 megabytes for the full model. Now we move to task 5 which is about testing the fine-tuned agent. I will compare the base model versus my fine-tuned model on the topic relevance. Open /root/code/task 5 test agent.py. I need to complete two todos. At line 109, replace a blank with a normal prompt like what's your best seller. At line 132, replace a blank with an off-topic prompt like what's the capital of France. Run the script with python/root/code/task 5 test agent.py. [music] The output compares both models. The fine-tuned model should score higher on topic relevance because it learned to talk about tackles from the training data. Now we move to task six which is about creating DPO preference data. DPO stands for direct preference optimization. It uses pairs of responses. A chosen one we prefer and a rejected one we do not want.

[00:09:15]Open/root/code/task 6create dpo data.py. I need to complete three todos. At line 57, replace a blank with a customer scenario like I've been waiting 20 minutes for my food. At line 61, replace a blank with a chosen response that is helpful and apologetic.

[00:09:35]At line 65, replace a blank with a rejected response that is rude and dismissive. Run the script with python/root/code/task 6 create dpo data.py. The output validates my preference pair [music] and adds it to the data set. This is how companies align models to be helpful instead of harmful. We have a knowledge check about DPO. The question asks what DPO optimizes [music] the model for. The answer is human preference and helpfulness because DPO trains model to generate responses that humans prefer, making them helpful, harmless, and honest. Before wrapping up, I want to highlight a few things. Fine-tuning embeds behavior directly into model weights, making it much harder to bypass than prompts. Laura reduces trainable parameters by 99.7%, allowing fine-tuning on consumer hardware. DPO is a simpler alternative to RLHF for aligning models with human preferences. That's it. We started by seeing how prompt engineering fails against jailbreaks. We then prepared training data, configured Laura for efficient training, and fine-tune the model. We compared the base and fine-tuned models and finally create preference data for alignment. We now understand the complete fine-tuning pipeline used to create helpful AI assistance.