LLM Schema Drift in Production: Detect Broken Contracts Before Users Do

Your parser did not throw an error. The response matched JSON Schema. The user still received the wrong action because priority: "high" quietly changed from “handle today” to “put near the top.” That is LLM schema drift in production: the shape remains valid while the meaning, frequency, or operating assumptions change.

Syntactic validation is necessary. It is not a production contract.

Define the Contract You Actually Depend On

Start with a versioned JSON Schema, but document semantics next to every consequential field. Specify allowed values, null behavior, units, time zones, ordering, confidence meaning, and what consumers do when data is absent.

For example, a due_at string is incomplete as a contract. Is it UTC? Can it be in the past? Does missing mean “no deadline” or “unknown”? Does the downstream scheduler act immediately?

Create three layers:

1. Shape checks validate types, required fields, and enums.
2. Semantic checks enforce domain rules such as ranges and cross-field consistency.
3. Behavior checks verify that downstream systems make the expected decision.

If a field can trigger money movement, publication, deletion, or a customer message, test the behavior—not only the JSON.

Capture a Field-Level Baseline

Before changing a model or prompt, record a baseline over representative traffic. Track missing rate, null rate, enum distribution, string length, numerical percentiles, retry rate, and validation failures by field.

Do not log sensitive values just to gain observability. Hash stable identifiers, bucket numeric values where appropriate, and retain samples only under a deliberate privacy policy.

A useful drift dashboard compares the candidate against the current production version:

Signal	Example alert
Missing field	risk_reason rises from 2% to 18%
Enum mix	unknown doubles after a prompt change
Length	Summaries become 4× longer
Cross-field rule	approved=true appears with confidence<0.5
Downstream action	Auto-escalations fall despite stable input mix

The last row matters most. A model can preserve every surface metric while changing the decision boundary.

Run Shadow Parsing Before the Cutover

Send a copy of production inputs to the candidate model or prompt without letting it affect users. Parse both outputs and compare them at the field level. Exact equality is often too strict for prose, so use task-specific comparators.

For classifications, measure agreement and inspect the confusion matrix. For extracted entities, compare normalized sets. For numerical fields, define tolerances. For free text, evaluate required facts and forbidden claims rather than generic similarity.

Store the schema version, prompt version, model snapshot, tool versions, and comparator version with every run. Without that lineage, a regression investigation becomes archaeology.

Add Canaries for Semantic Drift

Build a small suite of inputs whose correct behavior is stable and operationally important. Include boundary cases: missing evidence, conflicting dates, an empty retrieval result, a value exactly at a threshold, and malicious text inside a document.

Run canaries continuously, not only during releases. Providers can update serving infrastructure, safety behavior, or model aliases. Your own retrieval corpus and tools also change.

Canaries should test downstream effects in a safe environment. If the output says send_email=true, assert that the approval gate still intercepts it. If a model returns a currency amount, confirm the consumer interprets the same currency and precision.

Roll Out With Versioned Consumers

Never assume every consumer upgrades simultaneously. Accept the old and new schema during a migration window, or place a translation layer between the model and the rest of the product.

Version fields explicitly when meaning changes. Adding schema_version: 2 is cheaper than debugging an invisible reinterpretation across five services. Reject unknown major versions and tolerate additive optional fields only when consumers are designed for them.

Use a staged rollout: internal traffic, 1%, 10%, then wider exposure. Define automatic rollback thresholds in advance. A rollback based only on HTTP errors will miss semantic failures, so include business signals such as manual correction rate, escalation rate, and task completion.

Common Failure Modes

The first is trusting “strict mode” as a complete guarantee. Strict structured output constrains format; it does not make a field truthful or operationally consistent.

The second is evaluating only happy paths. Drift appears first in ambiguous, sparse, multilingual, and adversarial inputs.

The third is changing model, prompt, tools, and schema together. That makes attribution nearly impossible. Change one major layer at a time or run a factorial test with enough traffic to separate effects.

The fourth is keeping no raw evidence. Privacy-safe traces should still preserve which evidence supported each important field, allowing reviewers to distinguish extraction drift from retrieval drift.

The Takeaway

Treat structured LLM output like an evolving external API. Version the contract, measure field distributions, shadow-test candidates, exercise downstream behavior, and roll back on semantic signals. The parser is your first guardrail; production correctness begins after it passes.