Cases · Run on
Distillation: a student that matches its teacher, 2.6× faster
A gpt-oss 20B trained on DeepSeek V4.1 Flash support answers scored 77.5% against the teacher’s 76.7%, with a median latency of 795 ms against 2,082 ms.
Setup
- Data: 800 customer messages from the Bitext customer-support chatbot dataset (CDLA-Sharing-1.0). The Bitext answers were discarded.
- The teacher, DeepSeek V4.1 Flash, wrote the answers: a support agent for an online store, at most four sentences.
- 678 training and 120 held-out test messages. No synthetic data.
- Goal: “Lowest cost”. Candidates on 18 Oct 2026: gpt-oss 20B and Nemotron 3.5 Lightning 30B. Winner: gpt-oss 20B.
- Judge: GPT-6 Luna scores each answer from 1 to 5 against the teacher’s answer.
Result
| Model | Judge score | Median latency | Cost per 1,000 calls |
|---|---|---|---|
| Student (gpt-oss 20B, fine-tuned) | 77.5% | 795 ms | $0.067 |
| gpt-oss 20B (untrained) | 52.5% | 839 ms | $0.076 |
| Teacher DeepSeek V4.1 Flash | 76.7% | 2,082 ms | $0.081 |
| Claude Haiku 5.5 | 69.2% | 1,167 ms | $0.069 |
| GPT-6 Luna (also the judge) | 67.7% | 2,137 ms | $0.071 |
| Gemini 3.8 Flash | 65.7% | 4,087 ms | $1.606 |
| Claude Sonnet 5.5 | 70.0% | 2,918 ms | $3.019 |
Time and cost
- Time: 18 min 04 s from upload to result. Training was 13 min 20 s.
- Run cost: $1.19, plus $0.057 for the teacher’s answers.
Notes
- The student scores the same as the teacher (77.5% against 76.7%), is 2.6× faster (median) and costs 83% of the teacher’s price per call.
- The saving is small because the teacher is already cheap.
- The judge rates similarity to the teacher’s answer. The teacher was asked again in the eval, so it does not score 100%.
- The judge, GPT-6 Luna, is also one of the compared models.
The full write-up: Distill your GPT logs into a model you own.