Blog ·

7 open models compared for fine-tuning

Choosing a base model used to mean reading leaderboards. Leaderboards measure broad ability: maths, code, trivia, long reasoning. A fine-tune for one job uses very little of that. What matters is how well the model learns your task from a few hundred examples, how fast it answers, and what each call costs.

Honed trains on seven open bases. Below is what each one is, and how each did on the same tasks.

The seven

Model Lab Size Licence Honed goal
gpt-oss 20B OpenAI 21B total, 3.6B active (mixture of experts) Apache 2.0 Lowest cost
gpt-oss 120B OpenAI 117B total, 5.1B active Apache 2.0 Balanced
Nemotron 3.5 Lightning 30B NVIDIA 30B total, 3B active OpenMDW-1.1 Lowest cost
Nemotron 3 Super 120B NVIDIA 120B total, mixture of experts See model card Balanced
Inkling Small Thinking Machines Large, reads images See model card Most accurate, Balanced
DeepSeek V4.1 Flash DeepSeek Large mixture of experts MIT Most accurate
GLM 5.3 Flash Z.ai 320B total, 18B active MIT Most accurate

Sizes and licences come from the model cards: gpt-oss, Nemotron 3.5 Lightning, DeepSeek V4.1 Flash, GLM 5.3 Flash.

"Active" parameters matter more than total ones for speed and cost. A mixture-of-experts model routes each token through a small part of its weights. gpt-oss 20B runs 3.6 billion parameters per token, which is why it is fast and cheap to serve.

How they behave when you fine-tune them

gpt-oss 20B is the default for high-volume classification and routing. It is small enough to answer quickly and learns a fixed label set well. Untrained, it does not know your labels, so its score on a narrow task can be low. The fine-tune closes that gap.

Nemotron 3.5 Lightning 30B is the other low-cost option, built for long context (up to 1 million tokens) with a hybrid Mamba and mixture-of-experts design. It is a good pick when inputs are long: full email threads, contracts, call transcripts.

gpt-oss 120B and Nemotron 3 Super 120B are the middle tier. Pick them when the task needs more judgement than a label lookup, such as risk triage with several factors or extraction from messy documents.

DeepSeek V4.1 Flash, GLM 5.3 Flash and Inkling Small are the largest. They cost more per call and are slower. They are the ones to try when accuracy is all that matters, or when the output is free text that a judge scores. Inkling Small also reads images.

Results on the same tasks

We ran four tasks with the "lowest cost" goal, so on each one Honed trained gpt-oss 20B and Nemotron 3.5 Lightning 30B in parallel on the same data. These are their scores on held-out data, with the best prompted frontier model for reference:

Task gpt-oss 20B Nemotron 3.5 Lightning 30B Best prompted model
Receipt extraction (field accuracy) 72.7% 70.1% Claude Sonnet 5.5, 51.6%
Comment moderation (accuracy) 72.4% 72.0% GPT-6 Luna, 44.4%
Support replies, distilled (judge score) 79.5% 78.8% Claude Sonnet 5.5, 70.0%
Lead scoring on synthetic data (accuracy) 53.1% 58.4% Claude Haiku 5.5, 50.9%

These are the scores Honed uses to pick the winner. The winner's final eval can differ by a point or two; on receipts it was 71.1%.

The two small bases finished within 3 points of each other on three tasks, and both beat every prompted model on those three. On the lead-scoring set, no model beat the share of the majority class (about 59%), because that dataset carries little signal. We report it anyway.

We have not yet run the balanced and most-accurate goals on these tasks, so the larger five models have no row here. We will add them when we do.

How to choose

Start with the goal. Honed's three goals each train two or three candidates in parallel on the same data and keep the most accurate. On an exact tie, it keeps the cheaper one:

  • Lowest cost: gpt-oss 20B and Nemotron 3.5 Lightning 30B.
  • Balanced: gpt-oss 120B, Nemotron 3 Super 120B and Inkling Small.
  • Most accurate: Inkling Small, DeepSeek V4.1 Flash and GLM 5.3 Flash.

Our default is to start with lowest cost. On the three tasks above, the lowest-cost models beat every prompted frontier model. If yours does not, the result table shows by how much, and you retrain with a bigger goal on the same data.

Sources