Blog ·

Distill your GPT logs into a model you own

Distillation means training a small model to copy a large one on a single job. Hinton, Vinyals and Dean named the idea in 2015 (arXiv 1503.02531). Google showed in 2023 that a 770-million-parameter T5 trained on a large model's answers and reasoning could beat few-shot PaLM 540B on the tasks it was trained for (Distilling step-by-step, arXiv 2305.02301).

For a product team, the practical version works like this. If your app has been calling a frontier model for months, your logs hold thousands of inputs with answers you were happy enough to ship. That is a training set.

Check the terms first

Read your own contract. This is not legal advice.

  • OpenAI (Services Agreement §3.3(e), effective 1 January 2026): you may not use Output to develop models that compete with OpenAI, except for a "Permitted Exception". That exception covers models mainly used to classify or organise data, such as classifiers and embeddings, that you do not sell or distribute (OpenAI Services Agreement).
  • Anthropic (Commercial Terms §D.4): you may not use the services to train competing AI models without Anthropic's approval (Anthropic).

An internal classifier built from GPT outputs sits inside OpenAI's exception. A model that generates text and is delivered to a client is a grey zone under both sets of terms. Two cleaner routes exist:

  1. Use answers your own people wrote or corrected. If agents edited the model's draft before sending, the edited version is your data.
  2. Use an open model with a permissive licence as the teacher. DeepSeek V4.1 Flash and GLM 5.3 Flash are MIT-licensed, and Nemotron 3.5 Lightning's OpenMDW licence places no restriction on outputs (DeepSeek, GLM, Nemotron).

The steps

1. Export the logs. Input and final output for one feature. 500 to 2,000 rows is plenty for a first run.

2. Filter. Drop calls where a person overrode the answer, where the user complained, or where your parser failed. You are copying the large model at its best.

3. Hold some back. Keep at least 100 rows aside as the test set.

4. Train and compare. Train a small model on the rest and run it, the teacher and a few other models on the held-out rows.

In Honed the distillation template follows these steps. Use it with a teacher whose terms allow it, or with answers your team wrote. Upload the logs as CSV or JSONL. Honed masks emails, phone numbers and IDs, trains open models on the rows, and runs the eval against the teacher and the other models you pick. For open-ended answers, a judge model scores every answer blind, with the answers shuffled and unlabelled.

What we measured

We took 800 customer messages from the Bitext support dataset, threw away Bitext's own replies, and had DeepSeek V4.1 Flash (MIT licence) write a reply to each one. Writing all 800 replies cost $0.06. Honed trained gpt-oss 20B on 678 of them and held out 120.

The judge was GPT-6 Luna. It scored every answer from 1 to 5 against the teacher's reply, with the answers shuffled and unlabelled. Luna was also one of the compared models, so read its own row with that in mind.

Model Judge score Median latency Cost per 1,000 calls
Student: fine-tuned gpt-oss 20B 77.5% 795 ms $0.07
Teacher: DeepSeek V4.1 Flash 76.7% 2,082 ms $0.08
Claude Sonnet 5.5 70.0% 2,918 ms $3.02
Claude Haiku 5.5 69.2% 1,167 ms $0.07
GPT-6 Luna (also the judge) 67.7% 2,137 ms $0.07
Gemini 3.8 Flash 65.7% 4,087 ms $1.61
gpt-oss 20B, untrained 52.5% 839 ms $0.08

The student scored the same as its teacher and answered 2.6 times faster at the median. The cost saving was small, because the teacher was already cheap. Against Claude Sonnet 5.5, the same student cost about 1/45 per call. The run took 18 minutes and cost us $1.19 in compute, plus the $0.06 for the teacher's replies.

On a request to change a shipping address, the student copied the teacher's style closely: it asked for the order number and said what happens if the parcel has already shipped. The untrained gpt-oss 20B invented an "Orders" menu that does not exist. On a payment error, Sonnet and Gemini wrote a general list of causes, where the teacher asked for the order number.

What to expect

A student rarely beats its teacher on everything. It can match the teacher on the one job, answer faster, and keep its weights when the teacher's provider ships an update. The cost saving depends on the teacher: small against DeepSeek, large against Sonnet. We expect the gap to be smallest on short, structured answers and largest on long, open-ended ones.

Sources