Blog ·

When fine-tuning pays off (and when Haiku is enough)

We sell fine-tuning, so take this with that in mind. For a lot of LLM features, a prompted Claude Haiku 5.5 or GPT-6 Luna is the right answer and fine-tuning is wasted effort. Both cost $0.10 per million input tokens (Anthropic, OpenAI). That is cheap enough that a small, prompted frontier model is a strong default.

Fine-tuning earns its place under specific conditions. Here is the test we use with customers before they train anything.

Question 1: Is it the same job every time?

Fine-tuning teaches a model one behaviour: route this ticket, pull these fields, label this comment. It works best when the inputs vary but the job does not.

The strongest public evidence comes from Predibase's LoRA Land study (arXiv 2405.00732). It fine-tuned 310 small models on 31 tasks. The best fine-tuned model per task averaged 0.756 against GPT-4's 0.661, and fine-tunes beat GPT-4 in 224 of 310 runs. GPT-4 kept winning broad tasks: on HumanEval code generation it scored 0.829 against 0.433, and on MMLU general knowledge 0.774 against 0.589.

Fine-tune when the task is narrow and repeated. Prompt when each request is a different kind of problem.

Question 2: Can you say what a right answer is?

If you cannot score an answer, you cannot tell whether the fine-tune helped. Labels, field values and routing decisions are easy to score. "Write a helpful reply" is harder. It needs a judge model or human review, and the result is noisier.

Honed scores classification with accuracy and macro F1, extraction with field accuracy, and open-ended answers with a blind judge that sees every model's answer shuffled and unlabelled. All three work. Classification and extraction give the clearest result.

Question 3: Is it about behaviour or about facts?

Fine-tuning changes how a model answers. It is a poor way to teach it new facts. Gekhman et al. found that models learn new facts slowly during fine-tuning, and once they do, they hallucinate more (arXiv 2405.05904). Ovadia et al. at Microsoft found retrieval beats fine-tuning for adding knowledge (arXiv 2312.05934).

If the model needs your product catalogue or this week's prices, give it retrieval. If it needs to learn your 14 ticket categories and the exact JSON your parser expects, fine-tune.

Question 4: Do you have about 100 good examples?

An EMNLP 2025 study found that small specialised models need about 100 labelled examples on average to match general LLMs on text classification (ACL Anthology). OpenAI's fine-tuning guide sets the minimum at 10 and suggests starting with 50 (OpenAI).

You also need examples you did not train on, so you can test. Honed holds back at least 100 for that. If you have fewer real rows, it writes extra examples to fill the training set and to top up the test set. Written test examples never enter training.

Question 5: Does the volume make the price matter?

At 1,000 calls a month, the cheapest model and the most expensive one cost about the same in absolute terms. At 1,000,000 calls a month, a few tenths of a cent per call is a line in the budget.

Here is a support-ticket classification call with 800 input tokens and 20 output tokens, at list prices:

Model Cost per 1,000 calls
Claude Haiku 5.5 $0.09
GPT-6 Luna $0.09
Gemini 3.8 Flash $0.68 (doubles on 1 January 2027)
Claude Sonnet 5.5 $1.80

Against Haiku, the saving from a fine-tune is small at any volume. The case for fine-tuning there is accuracy and stability. Against Sonnet, or against a prompt stuffed with 30 examples on every call, the cost difference is large.

A fine-tune on an open base does not change unless you retrain it. A hosted frontier model changes when its provider updates it, and can be retired with as little as 60 days' notice (Anthropic deprecations).

The short version

If… Use
The task changes from request to request A prompted frontier model
The model needs facts it does not have Retrieval
You cannot score answers yet A prompt, and build a test set first
Same job, scorable, ~100 examples, real volume Fine-tune, then compare

Even then, run your fine-tuned model and the prompted ones on the same held-out examples before you decide. Honed does that by default. On our support-routing run, the fine-tuned model reached 100% against 90.5% for Claude Haiku 5.5. Haiku was still about five times cheaper per call. The table shows both numbers, so you decide which one matters more for the job.

Sources