The default is expensive and often unnecessary
When a workflow needs a model, the path of least resistance is the strongest one available. It works, so nothing forces a re-examination. Meanwhile the cost per run is several times what the task requires, and the latency is several times what the experience allows.
The question worth asking on every model call is not which model is best. It is what is the smallest model that is reliably correct here.
Three task shapes where small models hold up
Classification with a known set of labels. Routing an inbound message to one of eight categories needs reading comprehension and instruction-following, not deep reasoning. The fixed label set also makes correctness measurable.
Extraction from semi-structured text. Pulling a date, an amount, and a reference out of a message is pattern recognition with tolerance for mess. The failure mode is usually a missing field rather than an invented one, which is the safer direction.
Short-form drafting where a person reviews the output. One or two sentences from supplied material, in the tone requested. The review step absorbs the quality gap.
Where they do not hold up
Multi-step reasoning where an early error compounds. Tasks needing broad world knowledge rather than reading the supplied text. Anything acted on without review where being subtly wrong is costly. Long inputs where the relevant detail is buried and easily missed.
The boundary is fuzzy and moves with each generation, which is exactly why it should be measured rather than assumed.
Measure it on your own inputs
Take thirty real examples and write down the correct output for each. Run them through a small model and a large one. Count the differences and read them.
Two outcomes are common. Either the small model is indistinguishable, and the choice is obvious. Or it fails on a recognisable subset — the longest inputs, the ambiguous ones — in which case route on that property rather than upgrading everything.
- Same accuracy: use the small model
- Fails on an identifiable subset: route, do not upgrade wholesale
- Fails unpredictably: use the larger model, and keep the test set for next time
Cost is not the only reason
Latency compounds in a pipeline. Three model calls at two seconds each is a six-second wait, which is the difference between a response feeling immediate and feeling broken.
Small models also make retries affordable. If a call is cheap and fast, you can run it twice and compare, or validate the output and re-ask when it fails a check. Those patterns buy back much of the reliability gap, and they are impractical when each call is expensive.
Keep the choice reversible
Whatever you pick, keep the model identifier in configuration rather than scattered through code, and keep the evaluation set. Models get deprecated, renamed, and repriced on someone else's schedule.
A pipeline that can swap models by changing one value, and prove the swap was safe by re-running thirty examples, treats that as routine maintenance. One that cannot treats it as a project.
Routing beats upgrading
When a small model fails on part of your workload, the reflex is to move everything to a larger one. That pays the larger price on the majority of cases that were already fine.
Routing is usually better: send the easy cases to the small model and the hard ones to the large one, on whatever property distinguishes them. Input length is often enough. So is a confidence signal, or a validation step that re-asks a stronger model when the first answer fails a check.
The result costs close to the small model's price with close to the large one's accuracy, and it degrades sensibly — if the router is wrong, you have paid too much rather than got it wrong.
Prompt quality closes more of the gap than model size
A small model with a clear, specific prompt and two examples routinely beats a larger one with a vague instruction. Examples matter disproportionately: they communicate format and edge-case handling more reliably than any description of them.
This is worth trying before concluding a small model cannot do the job. The cheapest experiment in this whole area is rewriting the prompt and re-running the same thirty inputs.
Latency is a product decision, not just a number
Where a person is waiting, model choice becomes a user experience decision rather than an infrastructure one. A form that takes four seconds to validate feels broken regardless of how good the validation is.
Small models make interactive use realistic. They also make it feasible to run the call on every keystroke-pause rather than on submit, which changes what the feature can be — a different thing from doing the same feature more cheaply.
What to keep after choosing
Whatever you land on, keep three artifacts: the evaluation set, the measured result per model, and the date. Together they are the answer to "why are we on this model", which someone will ask, and the starting point for re-checking when the next generation lands.