Fine-tuning vs. prompting: how to know which one you actually need

Most teams decide this backwards. They pick fine-tuning because the task feels specialized, or they avoid it because it sounds expensive, and either way the decision gets made before anyone measures anything.

It's a measurable question. Here's how to answer it in an afternoon.

Start with the failure mode, not the technique

The useful question isn't "is my task hard." It's "what is the model getting wrong, and why."

Run your task through a stock model with a good prompt. Pull 50 failures and sort them into buckets:

The model doesn't know something. It's missing a fact, a document, a policy, a product detail. This is a retrieval problem. Fine-tuning is a bad way to install facts into a model and a good way to make it confidently wrong about the ones you got slightly off in your training data.

The model knows, but ignores your instructions. Wrong format, wrong length, drifts out of the required schema, won't stay in the label set. Try prompting fixes first — few-shot examples, constrained decoding, a stricter output schema. If it still drifts after that, fine-tuning fixes this well. This is the single most reliable thing fine-tuning does.

The model doesn't understand your domain's boundaries. Your team's definition of "escalation" or "material change" or "P1" doesn't match the general-world definition, and no amount of prompt text pins it down because the distinction lives in thousands of judgment calls, not a rule. Fine-tuning is the right tool. This is what training on labeled examples is actually for.

It's too slow or too expensive. The output quality is fine, the economics aren't. Fine-tuning a small model to match a large model's output on your narrow task is a well-trodden path and often the highest-ROI version of this whole exercise.

If your failures are mostly bucket one, stop reading and go build retrieval. If they're in two, three, or four, keep going.

Prompting is probably still underexplored

Before you train, be honest about whether you've actually finished prompting. Most teams that say "prompting didn't work" tried a zero-shot instruction and moved on.

The version worth beating:

• 5–10 few-shot examples chosen to cover your edge cases, not your easy cases
• A stated output schema with constrained decoding if your stack supports it
• Retrieval of the relevant context, if the task needs any facts at all
• A model one tier larger than the one you're planning to fine-tune

That last one matters. If a bigger stock model with good retrieval solves your task, fine-tuning a smaller one is a cost optimization, not a capability decision — a different project with a different urgency.

Write down the number

Whatever bar you're trying to clear, define it before you train. Accuracy on a held-out set, F1 on your minority class, p95 latency, cost per thousand requests. Pick one primary metric and one constraint.

Then score your best prompted setup on it. That number is your baseline. Without it, "the fine-tune worked" is a vibe, and you'll have no way to know if the next iteration made things better or worse.

Build a held-out test set of a few hundred labeled examples that never touches training. Yes, this is the boring part. It's also the part that determines whether any of the rest was worth doing.

Fine-tuning has real prerequisites

Labeled data, enough of it, and consistent. A few hundred examples can move a narrow classification task. Generation tasks want more. But the ceiling isn't set by volume — it's set by consistency. If two annotators labeled the same case differently, you're teaching the model to be uncertain in exactly the place you wanted it to be sharp.

This is the most common reason a fine-tune disappoints, and it's invisible until you look. Before training, check for contradictory rows — identical or near-identical inputs with different labels — and check class balance. If 5% of your rows contradict each other, that's your ceiling, and no hyperparameter search will move it.

A task that's actually stable. If your label definitions change every quarter, you're signing up to retrain every quarter. Sometimes that's fine. Know it in advance.

Somewhere to run it. A fine-tuned open-weight model is a thing you now operate. If you have no inference infrastructure and no plans to get any, a hosted API with a good prompt may genuinely be the better engineering decision even at worse unit economics.

When fine-tuning clearly wins

Three cases where I'd skip straight to training:

Narrow, high-volume classification. Routing tickets, tagging documents, extracting fields from a form you've seen a million times. A small fine-tuned model will beat a large prompted one on cost and latency by a wide margin, and usually on accuracy too.

Format and consistency requirements that prompting won't hold. Structured output where a 2% malformed rate breaks a downstream system.

Data that can't leave your perimeter. If your compliance boundary rules out hosted APIs, prompting a large hosted model isn't an option you have, and the question becomes which open-weight model to train rather than whether to train.

When it doesn't

• Your task needs current facts. Retrieval, not training.
• You have under a hundred examples and no path to more.
• You haven't measured anything yet.
• The stock model already clears your bar and you're doing this because it seems more serious.

That last one is more common than people admit.

The order that works

1. Define one metric and one constraint. Write down the target.
2. Build a held-out test set. A few hundred labeled examples.
3. Score a stock model with a real prompt and retrieval. That's the baseline.
4. If it clears the bar, ship it and go do something else.
5. If not, audit your labels for contradictions and imbalance before you touch a GPU.
6. Train, score on the same held-out set, compare against the baseline honestly.
7. Check the cost and latency numbers too, not just accuracy.

Step 4 is the one teams skip, and it's the one that saves the most money.

FAQ

Should I fine-tune or just write a better prompt?

Sort 50 real failures by cause first. If the model is missing facts, that is a retrieval problem and fine-tuning will make it confidently wrong. If it knows the answer but ignores your format, or if it misses your team's specific judgment calls, fine-tuning is the right tool. Prompt engineering is almost always cheaper to try first, and most teams stop trying far too early.

How many labeled examples do I need to fine-tune a model?

A few hundred consistent examples can move a narrow classification task; generation tasks want more. Volume is rarely the binding constraint — consistency is. If annotators labeled near-identical inputs differently, that contradiction rate is your accuracy ceiling, and no amount of extra data or hyperparameter search will lift it.

Can fine-tuning teach a model new facts?

Not reliably. Fine-tuning adjusts behavior, format, and judgment far better than it installs knowledge, and any fact you got slightly wrong in the training data becomes something the model states confidently. Use retrieval for facts and fine-tuning for behavior.

How do I know whether a fine-tune actually worked?

Define one primary metric and one constraint before training, build a held-out test set of a few hundred labeled examples that never touches training, and score your best prompted setup on it first. That number is the baseline. Without it, "the fine-tune worked" is an impression rather than a measurement.

Is a fine-tuned small model cheaper than prompting a large one?

Usually yes, on a narrow, high-volume task — that is the highest-ROI case for fine-tuning. A small fine-tuned model typically beats a large prompted one on cost and latency by a wide margin, and often on accuracy too. It only pays off if you have somewhere to run it, because a fine-tuned open-weight model is infrastructure you now operate.

---

Miniclay runs this sequence as the default. We benchmark a stock model with retrieval and prompting on your task before any training job starts, and if it clears your bar, we tell you rather than billing you for a run you didn't need. Dataset contradiction and balance checks happen before GPU time is spent, not after. Run a job free.