The biggest single line on our inference bill was a transaction categorizer: millions of calls a day, sorting line items into a fixed set of buckets, every one of them hitting a frontier model over the network. Least glamorous thing we run, and it cost more than the rest of the platform’s inference combined, because that is what the prototype did and nobody had gone back to fix it.
None of that needed a frontier model, and it never had. What the task needed was a model that knew our forty-odd categories cold and answered in single-digit milliseconds, so we built one: a small open-weight model, fine-tuned on examples the frontier model itself generated. It now beats that frontier model on this workload: same accuracy on our eval set, a fraction of the cost, and latency low enough that I stopped getting paged about it.
This post is about where that trade works, where it absolutely does not, and the pipeline in between, including the line on the bill that does not show up until month three.
Where the small model wins
The pattern holds in one specific shape: a narrow task, a fixed output space, and high volume, all three at once. Drop any one and the math changes.
Classification is the cleanest case. Structured extraction fits too, pulling the same fifteen fields off a document type you see a thousand times a day, and so does narration in a fixed house style, turning a row of numbers into a sentence that always reads the same way. These tasks share a small, knowable output distribution. You can write down what “right” looks like, which means you can fine-tune toward it and measure when you get there.
The reason fine-tuning works here is not what people assume. The task gets narrower than the model’s general capability, not the model smarter, and you have been paying frontier prices for capability you throw away on every call. A frontier model categorizing a coffee purchase is a concert pianist playing chopsticks: it will nail every note, and you are renting the wrong instrument.
Where it does not work: anything that needs broad reasoning, open-ended knowledge, or a long tail you cannot enumerate. The moment a task starts requiring the model to know things you did not put in front of it, or to reason across a problem whose shape you have not seen, the small model falls off a cliff and the frontier model is worth every cent. I watched a team try to fine-tune a small model into a general support agent. It is a graveyard. The output space is the whole of human conversation, the eval bar is “vibes,” and you spend six months building something a frontier model does on day one.
The question I ask before any of the pipeline below is whether I can write the eval. If I can write down what correct output looks like for this task, it is probably a fine-tune candidate. When “correct” is a feeling, the answer is no.
The pipeline: distill, fine-tune, hold a bar
The whole thing is three moves: use the frontier model to build training data, fine-tune the small model on it, then hold an eval bar so you know when you are done and, more importantly, when you have regressed.
The distillation step holds most of the value, and it is almost embarrassingly direct. You already have a frontier model doing the task in production, so you log its inputs and outputs, clean them, and that becomes your training set. The expensive model teaches the cheap one, once, and then you stop paying the expensive one.
# Build a fine-tune set by distilling our prod frontier model.
# Real rule we learned the hard way: do NOT trust the teacher blindly.
# A frontier model is right ~95% of the time on this task, which means
# ~1 in 20 of your training labels is wrong unless you filter.
def build_training_rows(logged_calls, min_confidence=0.85):
rows = []
for call in logged_calls:
label = call.frontier_output # the "teacher" answer
if label.category not in VALID_CATEGORIES:
continue # teacher hallucinated a bucket
if label.confidence < min_confidence: # teacher itself was unsure
continue # send these to human review instead
rows.append({
"messages": [
{"role": "system", "content": CATEGORIZER_PROMPT},
{"role": "user", "content": call.line_item_text},
{"role": "assistant", "content": label.category},
]
})
return dedupe_on_input(rows) # near-dupes inflate your eval lift and lie to youThat confidence filter and the dedupe are not optional polish. The first version of this skipped both, and we fine-tuned a model that was confidently wrong on exactly the cases the teacher was confidently wrong on, behind an eval score that looked great because half the test set had been duplicated into the training set. (We caught it, and it was not a fun week.)
Next comes the fine-tune itself, and in 2025 it is refreshingly boring, which is a good thing: a few thousand to a few tens of thousands of clean examples, a parameter-efficient fine-tune so you are training adapters and not the whole model, and a small open-weight base. The training run is cheap and short. A long, expensive training run means you are probably either using too big a base or trying to teach the model knowledge it should be retrieving, not memorizing.
What actually decides whether you ship is the eval, not the training.
# The bar. A change ships only if it clears the frontier baseline
# on OUR data, not on a public benchmark we don't run on.
def evaluate(model, gold_set):
correct, frontier_correct = 0, 0
by_category = defaultdict(lambda: [0, 0]) # [right, total] per bucket
for ex in gold_set:
pred = model.classify(ex.text)
if pred == ex.label:
correct += 1
by_category[ex.label][0] += 1
by_category[ex.label][1] += 1
if ex.frontier_label == ex.label: # baseline, scored once, frozen
frontier_correct += 1
# Aggregate accuracy hides the failure that gets you fired:
# a small category that the fine-tune quietly forgot.
worst = min(by_category.items(), key=lambda kv: kv[1][0] / max(kv[1][1], 1))
return {
"accuracy": correct / len(gold_set),
"frontier_accuracy": frontier_correct / len(gold_set),
"worst_category": worst[0],
"worst_category_accuracy": worst[1][0] / max(worst[1][1], 1),
}That per-category breakdown is the line I care about most. Aggregate accuracy will tell you the model is fine while it has gone blind to your three rarest categories, the ones that are usually high-value (fraud-adjacent buckets, in our case). A fine-tune that lifts the average by getting better at the common case and worse at the rare one is a regression dressed as a win.
The gold set is a few hundred hand-checked examples that never touch training, frozen in place. Every candidate model runs against it, and the frontier baseline runs against it once and stays pinned as the line to beat. If the small model does not clear that line on our data, it does not ship, full stop. The goal was never a leaderboard, only beating one specific model on one specific job.
The maintenance bill
Every pitch for fine-tuning skips the maintenance bill, so I will not.
When you call a frontier API, somebody else owns the model. They retrain it, they fix it, they keep it current, and you get the upgrades for free. Fine-tune your own and you own a model the way you own a service: it can rot, and it will, without ever throwing an error.
The rot shows up as drift. Your input distribution moves: new merchant types, new document formats, new phrasings, a whole category of transactions that did not exist when you built the training set. A frontier model would shrug and handle it because it generalizes, while your narrow little specialist has never seen it and guesses, confidently, wrong. Nothing crashes; accuracy just bleeds out a basis point at a time until someone downstream notices the numbers are off.
Owning the model means owning the loop that keeps it honest: production monitoring on the live distribution, a steady trickle of fresh examples back through the teacher, periodic re-evaluation against a gold set you also have to keep current, and a retraining cadence. That is real engineering time, forever, weighed against the inference savings. On a low-volume task it never pays back. The whole case rests on volume: at millions of calls a day the inference savings dwarf the maintenance, and at a few thousand a day they do not come close.
I would still make the same call on this workload tomorrow. But I make it now with the maintenance line written into the proposal, next to the savings, in the same font. The teams that get burned are the ones who saw the cost-per-call drop, declared victory, and walked away from a model that needed feeding.
The small model does not win by being better; it wins because the task was always smaller than the tool, and we finally bought a tool the right size. Knowing which of our workloads are that shape is what the whole thing comes down to.