Cases · Run on
Comment moderation: 74.7% accuracy against 44.4% for the best prompted model
A fine-tuned gpt-oss 20B learned the raters’ threshold on Civil Comments and sorted comments into allow, review and remove far better than prompted frontier models.
Setup
- Data: Civil Comments validation split (CC0), comments of 20 to 1,000 characters, 500 each of allow, review and remove.
- Labels from the toxicity score: allow below 0.3, review from 0.3 to below 0.7, remove at 0.7 and above.
- 1,275 training and 225 held-out test comments. No synthetic data.
- Goal: “Lowest cost”. Candidates on 18 Oct 2026: gpt-oss 20B and Nemotron 3.5 Lightning 30B. Winner: gpt-oss 20B.
Result
| Model | Accuracy | Median latency | Cost per 1,000 calls |
|---|---|---|---|
| Your model (gpt-oss 20B, fine-tuned) | 74.7% | 372 ms | $0.050 |
| gpt-oss 20B (untrained) | 43.1% | 472 ms | $0.066 |
| Claude Haiku 5.5 | 41.3% | 637 ms | $0.062 |
| GPT-6 Luna | 44.4% | 1,350 ms | $0.053 |
| Gemini 3.8 Flash | 40.4% | 2,501 ms | $0.736 |
| Claude Sonnet 5.5 | 40.4% | 806 ms | $1.076 |
Time and cost
- Time: 25 min 58 s from upload to result. Training was 22 min 04 s.
- Run cost: $1.52.
Notes
- Three classes, so chance is 33%. Macro F1: 74.9% for your model.
- With the same written rules, the prompted models marked many comments as allow that the Civil Comments raters count as remove. The fine-tuned model learned the raters’ threshold.
- Example: “Only a complete moron would raise business taxes 20% …” is remove. The fine-tuned model said remove; Haiku, Gemini and Sonnet said allow.
The full write-up: Content moderation with a small model you own.