Blog ·

New model version, worse results: how to catch regressions

In 2023, researchers at Stanford and Berkeley asked GPT-4 the same questions in March and in June. In March it followed their answer-format instruction 99.5% of the time. In June it followed it 0.5% of the time. Directly runnable code fell from 52% to 10% (Chen, Zaharia, Zou, arXiv 2307.09009).

Arvind Narayanan and Sayash Kapoor read the same data and argued that GPT-4 did not get less capable. It changed behaviour. For example, the June model wrapped code in markdown fences, so the code no longer ran as pasted (Normal Tech).

For a product, the distinction does not help. If your parser expects raw code and gets a fenced block, the feature is broken either way.

It keeps happening

A CMU study tested prompts against evolving LLM APIs. In 58.8% of prompt and model combinations, accuracy fell after an API update, and in 70.2% of those it fell by more than 5 points (Ma et al., arXiv 2311.11123).

Bugs cause some of it. Anthropic published a postmortem in September 2025 for three infrastructure issues that lowered Claude's output quality for weeks. At the worst hour, 16% of Sonnet 4 requests were affected (Anthropic).

Planned changes cause the rest. Gemini API aliases move to newer models, so calls to gemini-3.7-flash now go to gemini-3.8-flash (Gemini models), with no change in your code.

Models also retire. The guaranteed time differs by provider:

Provider Guarantee
OpenAI 6 months for generally available models, 3 months for variants, as little as 2 weeks for previews (OpenAI)
Anthropic 60 days (Anthropic)
Google Cloud Stable models stay available for at least 12 months after release (Google Cloud)

On Anthropic's deprecations page, the last two retirements came 61 and 62 days after the notice. That leaves about two months to find, test and ship a replacement.

The habit that catches it

The fix is the same whether the change is a bug, an alias or a retirement. Keep a fixed test set and run it before any model change reaches users.

1. Freeze 100 to 300 real examples. Take them from production, with the answer you accept as correct. Cover the rare cases, not only the common ones. Never train on them.

2. Score them the same way every time. Exact match for labels, field-by-field match for extraction, and macro F1 so a class with 3% of traffic counts. Write down the format error rate separately.

3. Pin model versions in code. Call a dated snapshot where the provider offers one, not an alias that moves.

4. Run the set on every change. That includes a new model version, a prompt edit, a provider's deprecation email and your own retrain. If accuracy or format errors move by more than you can accept, the change does not ship.

5. Keep the baseline row. The table you compare against is the one from the last version you shipped.

Where fine-tuning fits

A fine-tuned open model does not change until you retrain it. The weights that scored a number on your test set last month score the same number today. That removes the drift from the provider's side. You still need the test set for your own changes.

Honed builds this habit into the product. Every model gets a held-out test set of at least 100 examples that never enters training. Each retrain runs on the same set, next to the frontier models you compare against, so you can see when a new version is worse. You can also run the eval again later on new data to check the live model.

With the lowest-cost goal, a run takes 13 to 26 minutes from upload to result.

Sources