Deterministic Evaluation Seeds for LLM Apps: Reproduce the Failure

If your red evaluation turns green when you rerun it, you have not fixed anything. You have rolled the dice again.

Deterministic evaluation seeds for LLM apps help, but a seed is only one part of the replay contract. A production answer also depends on the exact model snapshot, prompt, retrieved documents, tool responses, sampling settings, clock, and application code. Capture the whole run or accept that your “test suite” is theater.

What Does a Seed Actually Control?

A seed asks a model provider to make sampling more repeatable for the same request. It does not guarantee mathematical identity across infrastructure changes, model revisions, or different providers. Some APIs expose a backend fingerprint or model version for exactly this reason.

Use seeds to reduce random drift, not to pretend a probabilistic service became a pure function.

Input	Must capture?	Why it changes the run
seed and temperature	Yes	Controls sampling behavior
model identifier or snapshot	Yes	Weights and serving stacks change
system and user messages	Yes	One hidden instruction changes everything
tool schemas	Yes	The model plans around available actions
retrieved chunks and order	Yes	RAG context is part of the prompt
tool outputs	Yes	External state is rarely deterministic
time, locale, feature flags	Yes	Application logic can branch on them

How Do You Build a Replay Envelope?

Create one immutable object for every evaluation case. Store hashes for large or sensitive payloads, plus protected raw artifacts where policy allows.

{
  "case_id": "refund-policy-017",
  "seed": 731942,
  "model": "provider/model-snapshot",
  "temperature": 0,
  "prompt_sha256": "…",
  "tool_schema_sha256": "…",
  "retrieval_snapshot": "kb-2026-08-14T04:50Z",
  "fixture_set": "support-tools-v12",
  "clock": "2026-08-14T05:00:00Z",
  "app_commit": "8c91f2a"
}

Do not log secrets just because replay is convenient. Replace credentials, personal data, and privileged tool responses with fixtures. The replay should preserve behavior without duplicating production risk.

How Do You Freeze Retrieval and Tools?

Vector search is a frequent source of hidden nondeterminism. Re-indexing changes embeddings, chunk identifiers, rankings, and ties. For offline evaluation, persist the exact retrieved chunk IDs, text hashes, scores, and order. Run two modes:

1. Frozen-context replay tests model and prompt behavior against the original context.
2. Live-retrieval replay tests whether the current knowledge pipeline still finds the right evidence.

Tool calls need the same separation. A payment lookup, search endpoint, or current-time function should become a versioned fixture in a deterministic run. Keep a smaller integration suite for live endpoints. Mixing both modes produces failures nobody can diagnose.

How Should You Score Probabilistic Output?

Exact string matching is fine for IDs, schema keys, and safety refusals. It is terrible for prose. Use layered assertions:

• validate the output schema;
• check required facts against trusted references;
• check forbidden actions deterministically;
• measure citation support;
• compare task outcomes;
• use a model grader only after simpler checks.

Record grader prompts and seeds too. A nondeterministic judge can make a stable answer look flaky.

expect(result.toolCalls).toEqual([]);
expect(result.answer).toContain("human review");
expect(result.citations).toSatisfyAll(isAllowedSource);
expect(await groundedness(result, frozenDocs)).toBeGreaterThan(0.9);

What Are the Common Failure Modes?

The classic mistake is setting temperature to zero and declaring victory. Providers may still return small differences. Another mistake is keeping only the final prompt while dropping retrieval and tool traces. The prompt is not the run.

Teams also overwrite fixtures in place. That destroys the baseline and turns regression testing into moving-target testing. Version fixture sets, approve changes in code review, and show a semantic diff: documents added, tool fields changed, and assertions affected.

Finally, do not run one seed forever. Use one canonical seed for debugging, then a small seed panel for robustness. A fix that works only for seed 731942 is not a fix.

What Should the CI Pipeline Do?

Run cheap structural checks on every pull request. Run frozen replay cases for known incidents. Add a small robustness matrix for critical flows. Schedule larger live-retrieval and provider comparisons outside the merge path.

When a case fails, CI should print the replay command:

npm run eval:replay -- --case refund-policy-017 --seed 731942 --fixtures support-tools-v12

That one line changes the culture. Engineers stop arguing about screenshots and start debugging an artifact.

The takeaway is blunt: a seed is useful, but the real unit of reproducibility is the entire execution envelope. Capture it once, replay it exactly, and turn every production failure into a permanent regression test.