AI Evals That Catch Regressions: Build a Harness Before You Ship

The most dangerous AI demo is the one that works five times in a row.

Five good answers create confidence. They do not create evidence. Change the model, prompt, retrieval settings, or tool description and the same feature may quietly get worse for an important slice of users.

Traditional software tests compare deterministic outputs. AI systems need a broader evaluation harness: a repeatable set of cases, scoring rules, runtime checks, and release thresholds that measure whether the product still behaves acceptably.

Without that harness, every AI release becomes a taste test.

Define the Job Before the Score

An evaluation starts with a product claim.

“Our assistant answers questions” is too vague. “Our assistant answers billing-policy questions using the current policy document, cites the relevant section, and never invents a refund” is testable.

Write the job as observable behavior:

• What inputs should the system handle?
• What facts must appear?
• What behaviors are forbidden?
• Which tools should it use?
• What latency and cost are acceptable?
• When should it decline or ask for clarification?

Separate hard requirements from preferences. A valid order number is a hard requirement. A friendly tone is usually a preference. The scoring system should punish factual or safety failures much more heavily than stylistic differences.

Build a Dataset From Real Failure Modes

Do not start with a hundred perfectly written examples invented by the engineering team.

Start with real user requests, support tickets, red-team cases, and failures discovered during development. Remove sensitive data, label each case, and preserve the reason it matters.

A balanced dataset should include:

• Typical requests that represent most traffic
• Edge cases with missing or conflicting information
• Adversarial or policy-sensitive requests
• Long inputs and noisy documents
• Requests that require tools
• Requests the system should refuse
• Previously fixed bugs

Keep a small smoke set for fast checks on every change and a larger regression set for release candidates. Add a new test whenever a production failure reveals a category you did not cover.

The dataset is a living record of what quality means for the product.

Use More Than One Kind of Grader

No single grader is reliable enough for every dimension.

Deterministic Checks

Use code for anything code can verify:

• JSON schema validity
• Required citations
• Correct identifiers
• Allowed tool names
• URL format
• Maximum length
• Presence of forbidden strings
• Numerical consistency

These checks are cheap, fast, and explainable.

Reference-Based Checks

For tasks with a known answer, compare the output against expected facts. Exact string matching is often too strict, so normalize values or check a set of required claims.

A support answer may have many acceptable phrasings but still need to mention the correct eligibility window and escalation path.

Model-Based Graders

Use a model grader for qualities that require judgment: completeness, groundedness, tone, or whether an explanation follows evidence.

Give the grader a narrow rubric and require structured output. Ask for a score plus a short reason tied to specific criteria. Do not ask, “Is this answer good?” Ask, “Does the answer make any claim not supported by the provided policy excerpts?”

Calibrate model graders against human labels. If the grader disagrees with reviewers on important cases, change the rubric or use a stronger grader.

Human Review

Humans remain necessary for ambiguous, high-impact, and novel failures. Review a sample of passing and failing cases, not just the worst outputs. This catches graders that have become too lenient.

Score the Whole System

An AI feature is more than its final text.

For an agent, capture the trajectory:

• Did it choose the correct tool?
• Were tool arguments valid?
• Did it recover from a tool error?
• Did it stop after completing the task?
• Did it ask for approval before a sensitive action?
• Did it use tool evidence accurately in the answer?

A final answer can look correct even when the agent took an unsafe path. Conversely, an agent may reach the right result with too many expensive calls.

Track quality, latency, token usage, tool count, and estimated cost together. A release that improves the score by one point while tripling latency may still be a bad product decision.

Create a Clear Scoring Contract

Use a compact rubric that the team can understand.

For example:

Dimension	Weight	Pass condition
Factual correctness	35%	No unsupported material claims
Task completion	25%	All required fields or actions completed
Policy compliance	20%	No prohibited behavior
Grounding	10%	Claims trace to supplied evidence
Style	5%	Clear and on-brand
Efficiency	5%	Within cost and latency budget

Add automatic failure conditions. A fabricated refund, leaked secret, wrong destructive action, or missing approval should fail the case regardless of the average score.

Then set release gates. A candidate might need at least 95% pass rate on critical cases, no safety regressions, and no more than a defined increase in cost.

Compare Changes With Paired Runs

Always compare a candidate against a baseline on the same dataset.

Run both versions with controlled settings. Record the model snapshot, prompt version, retrieval configuration, tool definitions, and temperature. Store outputs so reviewers can inspect side-by-side differences.

Look beyond the average:

• Which categories improved?
• Which categories regressed?
• Did failures move toward higher-risk cases?
• Did variance increase?
• Did latency or tool use change?

Averages hide tradeoffs. A new prompt may improve general helpfulness while making refusals less reliable. A new model may score higher overall but struggle with a specific language or document type.

Use confidence intervals or repeated runs for high-variance tasks. One run per case can confuse random variation with a real improvement.

Test Retrieval Separately

If the system uses RAG, evaluate retrieval and generation as two connected components.

For retrieval, measure whether the correct source appears in the top results. Track precision, recall, ranking quality, and freshness. Include cases where no relevant document exists.

For generation, provide the correct evidence directly and test whether the model uses it faithfully.

This separation tells you where the failure lives. If the correct policy never enters context, rewriting the answer prompt will not help. If retrieval is perfect but the model ignores the passage, then the generation layer needs work.

Add Online Signals Carefully

Offline evals protect releases. Production signals show what users actually experience.

Useful online metrics include:

• Task completion
• Correction or retry rate
• Escalation rate
• Tool errors
• User abandonment
• Explicit feedback
• Human audit findings

Do not treat thumbs-up rate as a complete quality score. Feedback is sparse and biased. Users may approve a confident answer that is factually wrong.

Log enough information to reproduce failures while masking sensitive data. Sample traffic for review based on risk, novelty, and uncertainty.

A Minimal Evaluation Harness

You do not need a giant platform to begin.

Use a versioned dataset such as JSON Lines. Write a runner that calls the full application stack. Save the input, assembled context, tool trace, output, latency, and token usage. Apply deterministic checks, then model-based graders where needed. Produce a report that compares the candidate against the current production baseline.

A useful folder might look like this:

evals/
  cases/
    smoke.jsonl
    regression.jsonl
    safety.jsonl
  rubrics/
    groundedness.md
    task-completion.md
  baselines/
  reports/

Run the smoke set in continuous integration. Run the full regression suite before changing a model, prompt, retrieval system, or tool contract. Require a human sign-off for high-risk changes.

The Release Rule That Matters

An evaluation system should make one question easy to answer:

Is this version safer and more useful for our users than the version already running?

That requires a baseline, representative cases, explicit scoring rules, and the discipline to block a release when a shiny demo hides a serious regression.

Start with twenty painful cases. Automate the obvious checks. Review the ambiguous ones. Add every important failure back into the dataset.

The harness will never prove that an AI system is perfect. It can prove that your team is no longer shipping blind.