LLM Regression Testing: What Breaks When You Swap Models

LLM regression testing is the practice of running a fixed set of cases through your system before and after a change, then comparing the results to catch behavior you didn't mean to alter. When the change is a model swap, most teams check one thing: are the answers still correct? That test passes a swap that truncates responses mid-sentence, quietly stops using your prompt cache, returns HTTP 200 with nothing in it, or costs a third more per request. It can also fail a swap that made your product better, for reasons that have nothing to do with the model.

We build LLM features for clients and for our own products, and we run a multi-model judge panel over this blog's own pipeline. This is what we've learned to measure.

Routing is the easy half

If you already route through a gateway, changing models is one line of config. We wrote about that side of the problem in provider-agnostic LLM routing and cost metering: how to keep the provider a runtime decision instead of a rewrite. That post answers which model. This one answers whether you can tell the swap was safe.

The incentive to swap is real and arithmetic. Anthropic's published pricing puts Claude Opus 5 at $5 per million input tokens and $25 per million output, against Claude Fable 5 at $10 and $50. Claude Sonnet 5 sits at $3 and $15, with an introductory $2 and $10 running through 31 August 2026 (pricing). When a model one tier down starts handling your workload, the finance case writes itself.

So you change the string, the eval suite goes green, and you ship. The eval suite going green is the part worth interrogating.

What actually breaks when you change the model

Everything in this section is documented by the vendor. None of it is hypothetical.

The failures you'll catch in five minutes

Some breakages announce themselves. A request carrying temperature, top_p, or top_k returns a 400 on Claude Opus 4.7 and later; so does thinking: {type: "enabled", budget_tokens: N}; so does an assistant-turn prefill on Opus 4.6 and Sonnet 4.6 (migration guide). We covered that class of breakage, along with tokenizer drift and model-scoped caches, in provider-agnostic LLM routing and cost metering, so this post won't rehearse it.

The prefill case does have a second-order cost worth naming. If you were prefilling {" to force JSON, the scaffolding around it is dead weight too: the stop sequences, the regex extraction, and the retry-on-parse-failure loop all exist to solve a problem the structured-outputs parameter now solves properly, which is the same pattern behind structuring model output with a JSON schema.

These are the good failures. They're loud, they surface on the first request, and nobody ships them.

The changes that pass every test you have

The dangerous ones return a 200 and a plausible-looking response. These four are the reason a correctness eval isn't enough.

Reasoning output went empty without changing shape. On Claude Opus 4.7, thinking blocks still arrive in the response stream, but the thinking field inside them is empty unless you opt in with display: "summarized". The field name didn't change. The block type didn't change. The schema didn't change. Anthropic's guide calls this out as a silent change from Opus 4.6, where summarized reasoning was the default, and names the symptom precisely: if you stream reasoning to users, the new default "appears as a long pause before output begins." A product that showed its work now shows a spinner, and every structural assertion you have still passes.

Thinking flipped to on by default. Omitting the thinking parameter on Claude Opus 5 produces a thinking request, where the same omission on Opus 4.8 and 4.7 produced a non-thinking one. The guide flags the consequence as a cost and truncation change rather than a behavior change: max_tokens caps thinking plus response text together, so a workload that sized that budget tightly around its answer "can now truncate mid-response."

Refusals arrive as success. On Claude Fable 5 and Claude Opus 5, a request declined by safety classifiers returns a successful HTTP 200 with stop_reason: "refusal". Code that reads content[0] unconditionally breaks. More to the point for testing: an integration test asserting a 200 and a well-formed envelope passes cleanly while your user gets nothing.

The economics move without the answers moving. Two mechanisms, both silent. Prompt-cache minimums don't run in one direction across generations: the minimum cacheable prefix is 512 tokens on Claude Opus 5 but 4,096 on Opus 4.6 and Haiku 4.5, so a prefix that cached before can stop caching after an upgrade, reporting cache_creation_input_tokens: 0 rather than an error (prompt caching). Tokenizers change too, by roughly 30% between Sonnet 4.6 and Sonnet 5 at unchanged per-token pricing. Both show up as a bill, never as a failed assertion, which is exactly why cost belongs in the eval rather than on a dashboard someone reads next month.

What changed	How it shows up	What catches it
Reasoning defaults to empty	Blank reasoning panel, long pause before text	Asserting on reasoning content, not just its presence
Thinking on by default	Answers truncate under a tight max_tokens	Rate of stop_reason: "max_tokens"
Refusal returns HTTP 200	Empty response, status assertions still green	stop_reason distribution across the eval set
Cache minimum differs by model	Cost climbs, quality identical	cache_read_input_tokens per request
Tokenizer changed	Budgets, ceilings, and dashboards all drift	Re-running token counts per model

Not one of those rows is visible to a correctness eval.

Your eval can be wrong in both directions

Correctness evals have a subtler problem. The eval is an instrument, and swapping the model recalibrates the instrument.

When the score falls because the model got better

This is the case that should change how you read eval output, and it's documented rather than theoretical.

A code-review harness tuned for an earlier model can report lower recall on a stronger one. Anthropic's guide explains the mechanism: when a review prompt says "only report high-severity issues," or "be conservative," or "don't nitpick," the newer model follows that instruction more faithfully. It investigates just as thoroughly, finds the bugs, and then declines to report the ones it judges below your stated bar. Precision rises. Measured recall falls. Underlying bug-finding improved.

Read that again in terms of your CI gate. The number went down. The product got better. The cause was an instruction you wrote.

The recommended fix inverts where the filtering happens: have the model report every finding with a confidence level and an estimated severity, then filter in a separate downstream step. Coverage becomes the model's job and ranking becomes yours. That's better engineering regardless of which model you're on, but you only discover you needed it when a swap exposes it.

Your prompt is part of the system under test

The eval guides we read while researching this piece all hold the prompt fixed and treat the model as the only variable. That measures the wrong thing, because a prompt tuned for one model can be actively harmful on the next.

Two documented examples from the same source. Self-verification instructions, the "double-check your answer before responding" family, now cause over-verification on Claude Opus 5, which verifies its own work unprompted. Removing them reduces the over-verification with no capability loss. The guide notes this inverts standard prompting advice, which is worth sitting with if you maintain a shared prompt library with a global self-check rule.

Delegation reversed direction between adjacent versions. Opus 4.8 under-reached for subagents and needed prompting to delegate more. Opus 5 reaches for them freely, which multiplies cost and latency, so the guidance you added one version ago now needs to come out. A workaround has a version range, and nothing in your repo records what that range is.

Verbosity is worth checking separately, since the obvious knob doesn't control it. Default response text is longer on Opus 5, and the guide is explicit that effort is not the lever: changing it moves thinking volume without reliably changing visible output length. Prompting is what shortens output. If your UI has a fixed-height answer card, that's a layout bug arriving through a model upgrade.

Don't let the model grade itself

LLM-as-judge is the standard answer for scoring open-ended output, and it carries measurable biases. Position bias, the tendency to favor an answer based on where it sits in the prompt, is well enough established to have dedicated studies proposing metrics for it. Self-preference bias is the one that matters most here: judges score their own outputs more favorably than an independent evaluator would. If the model you're evaluating is also the model doing the judging, the comparison tilts toward the new model by construction.

We run a three-model judge panel with a hard gate over this blog's own writing pipeline, and the failure modes have been instructive.

The panel contradicts itself. On one post, two judges disagreed on the same draft in opposite directions: one wanted the piece cut by a quarter to a third, another wanted it expanded. We recorded the split and trimmed only genuine repetition, which came to about 3%.

Judges fail you for the wrong reason. On the same post, two judges raised a hard-fail accuracy flag. The draft was right. The claim in our internal brief had been abridged, so the judges were scoring the draft against an incomplete version of its own source. We were one automated gate away from rewriting correct prose to match a summary error. On another axis, all three marked us down for a missing meta description that our pipeline deliberately assembles at a later step, scoring an artifact that didn't exist yet.

None of this makes LLM-as-judge unusable. It makes the judge a component with its own failure modes, which means: pin the judge model independently of the model under test and hold it fixed across the comparison, randomize presentation order, and keep the raw per-case scores rather than only the aggregate, so a disagreement is visible instead of averaged away.

What LLM regression testing has to measure for a model swap

Correctness is necessary and nowhere near sufficient. A swap eval should record a full row of metrics per case, not a single score.

Dimension	Why it moves on a swap	Gate or watch
Task correctness	The obvious one	Gate
Schema parse rate	Structured-output behavior and format drift	Gate
Refusal rate	Classifier changes, and refusals return 200	Gate
stop_reason distribution	Surfaces truncation and refusal at once	Gate
Error rate by type	New 400s from removed parameters	Gate
Input tokens	Tokenizer changes	Watch
Output tokens	Verbosity and thinking defaults	Watch
Cache read tokens	Cache minimums differ per model	Watch
p50 / p95 latency	Thinking defaults and effort levels	Watch
Cost per completed task	The number the swap was supposed to improve	Watch
Response length in characters	Product-visible, no natural assertion	Watch
Tool-call rate	Agentic eagerness varies by version	Watch

Gate on the dimensions where a regression is unambiguous. Watch the rest with a threshold that pages someone rather than blocking a deploy, because a modest latency increase bought with a large cost reduction may well be a trade you'd take, and that's a judgment call rather than a test failure.

In CI those two categories behave differently. A gate metric moving past its threshold fails the job the same way a unit test does, because a schema that stopped parsing or a refusal rate that tripled is not a trade-off anyone chose. A watch metric writes its delta into the build output and warns, so the numbers land in the pull request where someone can weigh them. Keep one exception in mind when you write the gate: if the regression is on a dimension your prompt shapes, like how many findings a reviewer reports, treat it as a review trigger rather than an automatic block. That's the shape the recall inversion takes, and a hard gate there will reject the better model.

Concretely, the runner captures the whole row:

import time
from dataclasses import dataclass

@dataclass
class Result:
    case_id: str
    model: str
    ok: bool            # your task-level checker
    parsed: bool        # output satisfied the response schema
    refused: bool
    input_tokens: int
    output_tokens: int
    cached_tokens: int
    latency_ms: int
    tool_calls: int
    chars: int

def run_case(client, model, case) -> Result:
    started = time.perf_counter()
    resp = client.messages.create(model=model, **case.request)
    latency_ms = int((time.perf_counter() - started) * 1000)

    refused = resp.stop_reason == "refusal"
    text = "".join(b.text for b in resp.content if b.type == "text")

    return Result(
        case_id=case.id,
        model=model,
        ok=(not refused) and case.check(text),
        parsed=(not refused) and case.validates(text),
        refused=refused,
        input_tokens=resp.usage.input_tokens,
        output_tokens=resp.usage.output_tokens,
        cached_tokens=resp.usage.cache_read_input_tokens,
        latency_ms=latency_ms,
        tool_calls=sum(1 for b in resp.content if b.type == "tool_use"),
        chars=len(text),
    )

The usage field names above are Anthropic's; other providers expose the same quantities under different keys. The shape is what matters. One row per case, per model, with the metrics beside the score.

Then compare per case rather than in aggregate:

def paired_deltas(baseline: list[Result], candidate: list[Result], field: str):
    """Per-case differences. Both models must have run the same cases."""
    by_id = {r.case_id: r for r in baseline}
    return [
        getattr(c, field) - getattr(by_id[c.case_id], field)
        for c in candidate if c.case_id in by_id
    ]

How many examples, and is the difference real?

The published advice disagrees with itself. One 2026 evaluation-tools guide recommends starting with 20 to 50 representative examples to keep CI fast. Another states that fifty examples detects large regressions, and 200 gives statistical confidence on a 3% to 5% quality change. Both are reasonable starting points and neither mentions the caveat.

The caveat is that the confidence intervals most eval tooling computes are built on the Central Limit Theorem, and a 2025 position paper argues they shouldn't be trusted below roughly a few hundred datapoints. That's precisely the range both recommendations sit in. So if your suite of fifty cases moves a few points either way, the honest reading is that you don't yet know whether anything happened.

Two things help. The first is framing, from Anthropic's Adding Error Bars to Evals: evaluations are experiments, and an eval score is a sample statistic rather than a measurement. The companion post puts the question directly, asking whether a benchmark difference is real "or could one model simply have gotten lucky in the choice of questions." The second is method. When both models answer the same cases, compare per-case differences instead of independent means. Pairing removes the variance that comes from some questions being harder than others, which is why paired_deltas above keys on case_id.

There's also a noise floor underneath all of this. Temperature 0 doesn't buy you bit-for-bit reproducibility, because floating-point arithmetic isn't associative and server-side batching changes how operations get grouped. Thinking Machines Lab's Defeating Nondeterminism in LLM Inference is precise about the distinction: the same script can be run-to-run deterministic while still not being invariant across hardware, software, or batch conditions, and the fix is batch-invariant kernels rather than a sampling parameter.

The practical move is cheap. Run your baseline model twice over the eval set and measure the spread. That spread is your noise floor, and any candidate delta smaller than it is not a result.

Keeping the comparison fair

Parameter names don't transfer between models, so a bare model-string swap changes more than the model.

Sonnet 4.6 defaults effort to high where Sonnet 4.5 had no such parameter at all, which means switching the string and nothing else can raise both latency and token usage on its own. Across generations the calibration shifts again: Anthropic documents Sonnet 5 at medium as comparable to Sonnet 4.6 at high, and Sonnet 5 at high as comparable to Sonnet 4.6 at max. Their advice for benchmarking is to match by observed thinking length rather than by parameter name, which is a good general rule. The same word means different amounts of work on either side of the swap.

Three habits keep the experiment clean. Log every request parameter alongside every result, so you can tell later whether you compared like with like. Change the prompt as a separate, subsequent experiment, since a model swap and a prompt rewrite in one commit confounds two variables you'll want to reason about independently. And shadow-run the candidate against live traffic before you canary it, because your eval set encodes the failures you already know about while production is where you find the others. Feed each one you catch back into the set, and after a few months it describes your real failure modes rather than the ones you guessed at the start. Pair that curated set with a small random sample of live prompts on each run: the golden set is the deterministic gate, and the random draw is what surfaces the failure you haven't written a case for yet.

Frequently asked questions

What is LLM regression testing? Running a fixed set of cases through your system before and after a change and comparing the results, to catch behavior you didn't intend to alter. For a model swap it should cover output format, refusal rate, token usage, latency, and cost alongside task correctness.

How many examples does an LLM eval set need? Published guidance ranges from 20 to 50 for catching obvious regressions up to 200 for detecting a few percentage points of quality change. Below roughly a few hundred cases, the standard confidence intervals are unreliable, so prefer paired comparisons and treat small deltas as unresolved rather than real.

Can I use an LLM as the judge when comparing two models? Yes, with two conditions. Pin the judge to a model that isn't part of the comparison, since judges score their own output more favorably, and randomize the order in which candidates are presented to limit position bias.

Why did my eval score drop after upgrading to a supposedly better model? One documented cause is that the newer model follows your harness instructions more literally. A prompt saying "only report high-severity issues" gets obeyed more faithfully, so fewer findings are reported and measured recall falls even though the model found more bugs.

Does temperature 0 make LLM output reproducible? No. Floating-point non-associativity and server-side batching mean output can vary with how requests are grouped, so identical inputs can produce different results under different load. Measure your noise floor rather than assuming determinism.

The eval you should have had anyway

Nothing here is specific to swapping models. Format validity, refusal rate, token accounting, and a latency distribution are worth watching whether or not you ever change the string in your config. Useful LLM regression testing measures the properties your users actually experience, and a model swap is simply the event that makes their absence expensive, usually at the point where you've already shipped.

If you're building LLM features and want a team that has already run into these edges, that's the work we do at Vantaso. You can reach us at contact@vantaso.org.