Fine-tuning

What it is good for

Fine-tuning teaches form, not facts. The cases where it genuinely wins:

• A consistent output format the model keeps drifting away from, over thousands of calls.
• A specific voice or register — house style, a domain's terminology, a particular way of structuring a reply.
• A narrow, high-volume task where a small fine-tuned model matches a much larger general model at a fraction of the cost and latency. This is the strongest commercial argument.
• Shortening the prompt. If a 2,000-token instruction block is sent on every request, moving that behaviour into the weights saves it on every call.

Most fine-tuning today is parameter-efficient: LoRA and similar methods train a small set of additional weights rather than the whole network, which is why it can cost tens of dollars rather than thousands.

What it needs

Hundreds to a few thousand examples of input and correct output, and the quality of those examples sets the ceiling on the result. This is the part people underestimate: fine-tuning is a data-curation project with a short training job at the end. Fifty inconsistent examples will teach the model to be inconsistent, reliably.

You also need a held-out test set from before you start. Without it you cannot tell whether the tune improved anything or just made the model worse in a way that happens to please you on the three prompts you tried.

What to watch out for

Fine-tuning is a bad way to add knowledge, and this is the mistake that wastes the most money. Facts learned this way are absorbed unevenly, cannot be cited, cannot be updated without retraining, and tend to increase hallucination rather than reduce it — the model becomes more confident in the domain without becoming more correct. For knowledge, use retrieval-augmented generation.

Three further costs:

1. Catastrophic forgetting. Training hard on a narrow task degrades general ability. The model gets better at your format and worse at reasoning around it.
2. Maintenance. A fine-tune is pinned to one base model. When a better base arrives, you re-run the job — and re-run your evaluation.
3. Lock-in. A tune produced inside a hosted provider's platform generally cannot be moved elsewhere.

Order of attempts, in cost order: better prompt, then examples in the prompt, then retrieval, then fine-tuning. Most projects that reached for step four should have stopped at step three.