Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA

[00:00:00]Is fine-tuning large language models still needed today?

[00:00:05]Well let's take a well-known legal AI company because back in 2023 they fine- tuned their own model, their own custom AI, in partnership with the Frontier Lab, and in blind tests attorneys preferred this fine-tune model over the Frontier model at the time, which was GPT-4.

[00:00:29]They preferred it 97% of the time.

[00:00:34]A win for fine tuning.

[00:00:37]They built a custom AI that lawyers actually preferred over the off the shelf leading model.

[00:00:41]So the lesson is, if a general purpose model isn't quite right for some specific use case like legal work, then fine tune it.

[00:00:53]Right?

[00:00:54]Well, first let's define what fine tuning actually is.

[00:00:59]So we're going to start off with a base model, a base LLM.

[00:01:06]This is what comes off the shelf.

[00:01:09]So this is either working with a Frontier lab directly or maybe picking out an open source model.

[00:01:15]Now the base model is trained on a massive chunk of data.

[00:01:21]Effectively we're kind of scraping information off the internet.

[00:01:27]And using it to train that base model so there's lots of general knowledge that's going to be baked into the weights of the model.

[00:01:35]Now fine tuning takes that base model and it customizes it how by continuing its training, but now it's doing it on a much more focused data set so we've just got some specific documents here which are really focused in one particular area.

[00:01:53]So it might be legal contracts or maybe a an organization's internal support tickets, just like stuff that's not sitting around on the internet waiting to be scraped by base models.

[00:02:04]And then what this results in is we get a new model out of this, a fine-tuned model, and that fine-tuned model incorporates, well, the information from the focus data set plus all of the weights from the base model as well.

[00:02:24]They're now adjusted with that additional data.

[00:02:26]So this resulting model should get better at certain narrow tasks.

[00:02:31]That is the wonder of fine tuning.

[00:02:34]But in practice how well does it work?

[00:02:38]Well let's go back to this legal AI company.

[00:02:42]So, in 2025, they created their own legal benchmark.

[00:02:48]So benchmark would measure how effective their models were.

[00:02:53]At performing particular tasks and they tested their find tune system against the latest crop of Frontier AI models.

[00:03:03]So we've got the custom model here and then we've a bunch of Frontier AI models as well, kind of the mess general purpose models available at the time.

[00:03:14]Now what was the result of this benchmark?

[00:03:17]Well the result was that seven of the general purpose models.

[00:03:22]Had now surpassed the company's custom model in the benchmark.

[00:03:27]So this was by models that had never actually received any custom legal fine tuning, but they were still better.

[00:03:35]And Bloomberg saw something similar to this.

[00:03:37]They famously trained BloombergGPT from scratch and later evaluations found GPT-4 and ChatGPT outperforming Bloomberg GPT on many financial benchmarks.

[00:03:49]So where does that leave fine-tuning today?

[00:03:52]Is all that custom training worth doing when big frontier general models keep getting smarter on their own?

[00:04:01]Well to answer that I think it's worth considering how general models have in many cases caught up to custom trained ones.

[00:04:09]There's a few reasons for that and one it's just context windows have got really really big.

[00:04:16]So the original GPT-3 that used a token window of 2k, 2000 tokens, and today Frontier models they routinely handle much much more than that, like 1 million tokens plus of input.

[00:04:32]So if a if a model can read say 500 pages of legal documents directly in its prompt, then why bake those documents into the weights at all?

[00:04:42]Just pass the stuff that's contextually relevant when prompting.

[00:04:46]Now, second is reasoning models.

[00:04:50]They do extended thinking inference time working through a problem step by step before answering.

[00:04:58]So reasoning is coming from how hard the model thinks at the moment of the question at inference time rather than just from how it was trained months earlier.

[00:05:09]And then third is just cheaper inference, as in models are getting more efficient and smarter.

[00:05:16]So when the frontier model is constantly getting smarter and cheaper, training a custom version becomes a bit of a moving target, where by the time the fine tune model ships well the next frontier release may have leapfrogged it already.

[00:05:31]Okay so so general models have got better but if we're not adjusting weight..how do we make a general model behave like a specialist? Like a legal scholar for example.

[00:05:43]Well it turns out there is a whole stack of customization techniques that don't touch the model weights at all.

[00:05:52]And the first of those is RAG, retrieval augmented generation.

[00:05:57]Instead of training the documents into the model the application retrieves, that's the R in RAG, retrieves the documents at query time and then feeds them into the prompt.

[00:06:08]There is also the consideration of context, specifically context engineering.

[00:06:15]And the idea is that a good prompt is a carefully assembled bundle of context.

[00:06:21]The system prompt, the relevant data, maybe some format guidelines and the like, they're all kind of packaged together.

[00:06:27]And, the third thing I think we need to consider are agent skills.

[00:06:33]These MD files that you can create.

[00:06:36]So skills are folders of files that package up procedural knowledge, basically... How to do something and the tools to use to do it and the model loads them on demand when it sees a task back cause for them.

[00:06:50]So instead of fine-tuning a model to know how to write SQL queries against a very specific schema, a SQL agent skill can tell the model exactly what to do and any general purpose model can use that skill.

[00:07:05]So essentially what I'm trying to say is fine tuning isn't the only path to customization.

[00:07:11]There's a whole stack of options that work without ever actually touching model weights.

[00:07:17]Which is a good thing because fine tuning is not free.

[00:07:20]In addition to the training run itself there's a cost in collecting the examples, and evaluating results, and avoiding regressions, and there's a cost of maintaining the custom models as the frontier models move on.

[00:07:33]So all of this really begs the question.

[00:07:36]Does anyone still need to fine tune at all?

[00:07:40]Well, yes, but just for a much narrower set of reasons than back in 2023.

[00:07:46]Now there is a modern technique called LoRa, that's low rank adaptation, that lets a team fine tune by training a small adapter, and this adapter sits on top of an existing base model.

[00:08:06]So we've got the base model with its weights and then this adapter that sits on top of it.

[00:08:12]So most of the original weights are staying locked, and then really a lot of what gets labeled as fine tuning in production today is some flavor of LoRa or a related parameter efficient method.

[00:08:26]And fine tuning does still make sense in certain situations.

[00:08:29]Well, like for example, when we are concerned about reduced latency if that's super important.

[00:08:38]If a model has to respond in real time like a like a voice agent answering a phone call, well frontier reasoning models with all their thinking time are often too slow.

[00:08:48]So if real-time responsiveness is the constraint, small fine-tuned models might still be the way to go.

[00:08:57]Another reason is because of distillation.

[00:09:01]Now the idea is to take a huge front-end model.

[00:09:04]Generate high quality outputs from it then fine-tune a much smaller model on those outputs.

[00:09:10]So you're essentially generating reasoning traces from a large parameter teacher model and using that to fine- tune smaller models.

[00:09:19]And there's also reinforcement fine-tuning, RFT.

[00:09:26]So instead of training only on fixed correct answers, RFT uses a prompt data set plus a grader and the model samples candidate answers, the grader scores those answers and training updates the model to make high scoring answers more likely.

[00:09:43]Of course, the catch is that RFT only really works when the output can be programmatically graded, which is to say when there's a definitive right answer.

[00:09:51]So..so fine tuning isn't dead.

[00:09:55]From a practical decision framework today I think of it as the order going something like this.

[00:10:01]So start with a base model and then you can implement prompt and context engineering.

[00:10:10]If knowledge is fresh or proprietary well you might want to add in capabilities for RAG as well, and if procedural know-how is the thing that's missing... Well then at that point you might want to add in agent skills.

[00:10:28]And then maybe only reach for the fine tuning if there's a specific bottleneck that the rest of this stack can't solve.

[00:10:36]But what do you think?

[00:10:37]Does fine tuning still have its place?

[00:10:40]Let me know in the comments.