What happens when one of those articles changes? That was all I asked, and the room went quiet. A team had come to me last month wanting GPU budget to fine-tune a model on their support knowledge base. Forty thousand articles, updated daily, and they wanted the model to “know” all of it. Someone eventually said, well, we would retrain. How often, I asked. Every night, they said, and I watched them slowly realize they had just described a retrieval problem with a training bill attached.
This is the most common mistake I see, and it manages to be expensive in three ways at once. People reach for fine-tuning because it sounds like the serious option, the machine-learning option, while retrieval feels like plumbing. The instinct, especially among people who came up doing ML the old way, is to assume the real answer involves gradients. It usually does not.
The framework is three questions, not a vibe
Most advice on this collapses into “it depends,” which is true and useless. What it actually depends on is three questions, asked in order, the first settling most cases.
Is it knowledge or behavior? This is the whole ballgame, and everything else is downstream of it. Fine-tuning changes how a model behaves, whereas retrieval changes what it knows. If you want the model to have facts it does not currently have, your support articles, your product catalog, your internal policies, that is knowledge, and knowledge gets retrieved, not baked into weights. Fine-tuning teaches a model to act a certain way, adopt a tone, follow a structure, and it is genuinely bad at teaching a model that the return policy changed on Tuesday. People conflate these constantly because both feel like “making the model better at our thing,” but only one of them is fine-tuning.
How often does the knowledge change? Even on the knowledge side, the rate of change tells you how much you would regret freezing it into a model. A taxonomy that updates twice a year is one thing, while a catalog that changes hourly is another. Anything that moves faster than your retraining cadence cannot be fine-tuned without being permanently stale, and almost everything interesting in a real business moves faster than you want to retrain.
Do you need provenance? Nobody asks this until an auditor or an angry customer forces it. If you have to show where an answer came from, cite the source, prove the model did not invent it, you need retrieval, because a fine-tuned model cannot tell you which training example produced a given sentence. The knowledge is smeared across billions of parameters with no return address. In regulated work, payments, healthcare, anything with a compliance officer, this question alone ends the debate. I have sat in those reviews, and “The model just knows” is not an answer you can give a regulator.
Run those three in order and you find that what most teams want to fine-tune is knowledge, that it changes often, and that they will eventually need to cite it, three for three on retrieval.
Where fine-tuning actually earns it
If I only ever said retrieve, I would be selling the same one-size answer I am complaining about. Fine-tuning has real cases where it wins, and it would be dishonest to skip them.
Format, style, and tone come first, and they are the cleanest win. If you need every output to come back as a particular JSON shape, or in a specific house voice, or following a rigid template, that is behavior, and behavior is what fine-tuning is for. You can fight this with prompting and few-shot examples for a while, but past a certain consistency bar, a small fine-tuned model holds the format more reliably than a large prompted one, at a fraction of the token cost because you stop paying for the instructions on every call.
Narrow classification is the next one. If the job is to sort inputs into a fixed set of buckets, route this ticket, flag this transaction, label this document, a fine-tuned small model is often better and dramatically cheaper than a frontier model with a clever prompt. The task is closed and the labels do not change daily, so there is nothing to retrieve, just a decision boundary you want the model to learn. This is fine-tuning doing exactly what it is good at.
The third case is latency and cost on a fixed task. When you have one well-defined job running at volume, the economics flip. A retrieval call adds an embedding lookup, a vector search, and a fat context window on every request, while a fine-tuned small model with the behavior baked in answers in one short hop. At low volume the difference is noise, but at high volume on a task that does not change, that difference is the whole budget.
The three cases share one property: every one is about behavior on a stable task, never about supplying fresh, citable knowledge. The moment your classification task needs to reference a document that changed yesterday, you are back in retrieval territory and the fine-tune was a detour.
Why teams fine-tune anyway
Why does the GPU-budget request keep landing on my desk? Because fine-tuning lets you avoid building good retrieval, and good retrieval is genuinely hard. Chunking, embedding, the eval loop, the freshness pipeline, the permission filtering, none of it glamorous and all of it fiddly. Fine-tuning feels like you can hand the problem to the training run and walk away. Throw the documents at the model, let it learn them, done.
Handing it to the training run does not finish the problem so much as postpone it, and it comes back worse. The model is stale the day after you train it. It hallucinates facts in the gap between training examples, confidently, because confident wrong answers are the one output these models reliably produce. You cannot cite anything, and when the knowledge updates, you are back at the GPU queue retraining a model that will be stale again by the weekend. (The same team usually fine-tunes a second time, certain the first run just needed more epochs, which it did not.)
I have watched smart teams spend a quarter learning this the slow way. The fine-tune that was supposed to save them retrieval work ends up next to a retrieval system they had to build anyway, because they finally needed citations or freshness, so now they maintain both.
Fine-tuning and retrieval were never really competitors. Retrieval handles what the model needs to know right now, and fine-tuning handles how it should behave, persistently, on a task that holds still. The best systems I have built use both, retrieval for the knowledge, a light fine-tune for the format, each doing the job it is actually good at.
What sinks teams is not fine-tuning itself but reaching for it to dodge the retrieval work, then doing that work anyway, six months late, after the first answer with a fabricated citation goes out to a customer.
A few teams really do belong in the fine-tuning column. When they walk me through the case, it survives all three questions, and I send them off to train with a clear conscience. Most never get that far, because they are the team from last month, asking for a training budget and then going quiet as it dawns on them that what they have is forty thousand articles changing every day and a model that would be stale by the next morning. The teams that skip the questions buy the GPUs, and end up standing between a paid training bill and the retrieval system they had to build anyway.