How Many Test Cases Does an LLM Evaluation Need? A Practical Power Guide

An evaluation with 50 examples can be useful and still be incapable of detecting the regression your release gate claims to prevent. The problem is not that 50 is always too small. It is that sample size only makes sense relative to baseline performance, outcome variability, and the smallest change that matters.

Treat evaluation design as a measurement problem. Before generating more test cases, define the decision the score will support.

Start With the Minimum Detectable Effect

Suppose a production classifier is 95% accurate and a one-point decline would be costly. Detecting a change from 95% to 94% requires far more examples than detecting a collapse to 80%.

The minimum detectable effect is the smallest difference that should change a release decision. Set it from product consequences: extra human review, failed transactions, safety incidents, churn, or cost. A statistically detectable difference that has no practical consequence is not a useful gate.

For binary pass/fail metrics, required size depends heavily on the baseline rate. Rare failures need many observations before their rate can be estimated precisely. For continuous judge scores, variance matters: noisy ratings require either more cases or a better measurement process.

Prefer Paired Comparisons

Evaluate the candidate and baseline on the same inputs. Then compare each pair rather than comparing two unrelated averages. Pairing controls for example difficulty and often improves statistical power.

For a binary outcome, focus on discordant pairs: cases where only one system passes. McNemar-style analysis is more informative than treating both samples as independent. For continuous scores, analyze per-case differences and their confidence interval.

Keep generation settings and tool data comparable. If the candidate sees fresher retrieval results or a different tool sandbox, the test is measuring a bundle change, which must be documented explicitly.

Account for Sampling Variance

One model output per prompt measures a particular stochastic draw. If product behavior varies materially across runs, repeat selected cases with independent seeds or sampling. Do not multiply every case automatically; decide whether breadth across intents or repeated measurement of variance is more valuable.

Clustered data needs care. Ten thousand near-duplicate support tickets do not provide ten thousand independent pieces of evidence. Estimate uncertainty by customer, conversation, document, or other unit that matches how failures correlate in production.

Report confidence intervals with point estimates. “96.2% pass rate” implies more certainty than the evidence supports when the interval spans the release threshold.

Size Each Critical Slice

An overall sample can look large while important cohorts remain tiny. Language, customer tier, safety category, tool type, document length, and region may each have distinct behavior.

Pre-register critical slices and allocate cases deliberately. If a slice is too rare to evaluate statistically, use targeted stress tests and describe them as such. Do not let a strong majority cohort erase a severe minority regression.

Avoid testing dozens of slices and reporting only the worst one without correction or replication. Multiple comparisons produce apparent discoveries by chance. Separate exploratory findings from confirmatory release gates.

Use Sequential Testing Carefully

Teams often inspect results while an evaluation is still running and stop when the preferred model pulls ahead. Repeated peeking inflates false positives under ordinary fixed-sample tests.

Use a predefined sample size or a valid sequential design with stopping boundaries. Keep a holdout set for the final decision so prompt tuning does not overfit the visible suite.

For a first estimate, run a pilot to measure baseline rate, score variance, and pair disagreement. Feed those values into a power calculation, record assumptions, and revisit the design when production traffic changes.

There is no universal “enough cases” number. A defensible evaluation connects sample size to a minimum meaningful change, the variability of the metric, the structure of the data, and the cohorts the product must protect. That connection is what turns a benchmark score into release evidence.