Blog ·

A fine-tuned small model vs Claude Haiku on support routing

Routing is a common first job for a fine-tune. A ticket comes in, and something has to decide which team, queue or workflow gets it. It is narrow, it runs on every ticket, and a wrong answer costs a person's time to fix. That makes it a good first test for fine-tuning.

We ran it end to end on Honed in production, the way a customer would, and timed every step.

The setup

Data. The Bitext customer-support dataset (CDLA-Sharing-1.0): about 27,000 customer messages labelled with 27 intents such as cancel_order, track_refund and change_shipping_address. We uploaded 2,000 rows.

Split. Honed dropped 35 empty or duplicate rows, held back 294 tickets for testing and trained on 1,671. No synthetic examples were needed, because the upload had enough real rows. The test tickets never entered training.

Models trained. With the default "most accurate" goal, Honed trained Inkling Small, DeepSeek V4.1 Flash and GLM 5.3 Flash in parallel on the same data, and kept GLM 5.3 Flash.

Models compared. Claude Haiku 5.5 and Claude Sonnet 5.5, prompted with the task and the label list, plus the untrained GLM 5.3 Flash.

Scoring. Accuracy and macro F1 on the held-out tickets. Macro F1 averages over the 27 intents, so a rare intent counts as much as a common one. Latency is the median after warm-up calls. Cost is the tokens each model used on the test set, at list price, per 1,000 calls.

The result

Model Accuracy Macro F1 Median latency Cost per 1,000 calls
Fine-tuned GLM 5.3 Flash 100.0% 100.0% 530 ms $0.21
Claude Sonnet 5.5 95.9% 95.8% 790 ms $0.89
Claude Haiku 5.5 90.5% 90.6% 580 ms $0.04
GLM 5.3 Flash, untrained 88.4% 88.1% 480 ms $0.22

The fine-tuned model got all 294 test tickets right. It was 4.1 points more accurate than Claude Sonnet 5.5, at about a quarter of Sonnet's cost per call and two thirds of its median latency. Against Claude Haiku 5.5 it was 9.5 points more accurate, and Haiku was about five times cheaper per call.

If 90.5% is good enough for your routing, Haiku is the cheaper answer. If each misrouted ticket costs an agent a few minutes, 9.5 points of accuracy are worth more than the price gap.

The timings

Step Time
Upload and parsing 1 s
Questions shown and answered 45 s
Training data ready 1 min 34 s
Training, three models in parallel 1 h 43 min
Evaluation 2 min 10 s
Upload to result table 1 h 49 min

Training took 95% of the time, because the "most accurate" goal trains the three largest bases. With the "lowest cost" goal, our runs on other tasks took 13 to 26 minutes end to end. The compute for this run cost us $13.77, almost all of it training. Nobody touched a GPU, a training script or a config file. The person running it answered the questions and picked the models to compare against.

How this compares to published work

In Predibase's LoRA Land study, fine-tuned 7B models beat GPT-4 in 224 of 310 runs, with the largest gains on narrow classification (arXiv 2405.00732). An EMNLP 2025 study found that about 100 labelled examples is the average break-even point for small models against general LLMs on text classification (ACL Anthology). We trained on 1,671.

One caveat. 100% is a ceiling, not a promise. Bitext is partly synthetic and very clean, and its messages follow templates. Real tickets are messier: typos, two questions in one message, signatures and quoted replies. Expect lower absolute numbers on your own data, and test on your own tickets before you rely on the gap.

Sources