Tool Contract Testing for AI Agents: Stop Shipping Schema Roulette

An AI agent does not fail only when the model is wrong. It fails when the model calls create_invoice, the service expects a renamed field, the adapter swallows a 422, and the agent confidently tells the user the invoice exists.

That is not an intelligence problem. It is a contract problem wearing an AI costume.

Tool contracts include the schema shown to the model, the validation performed by the runtime, the request sent to the provider, and the response translated back into model context. If those four layers drift, your agent is playing schema roulette.

Define the Contract as Executable Data

Start with a versioned JSON Schema for every tool input and output. Descriptions matter because the model reads them, but descriptions are not enforcement. Validate at runtime before an external call and again after the provider responds.

An input contract should specify:

• required and optional properties
• enum values and mutually exclusive branches
• formats for dates, IDs, email addresses, and money
• length and numeric boundaries
• whether extra properties are rejected
• idempotency and confirmation fields

The output contract deserves equal attention. Define success, partial success, retryable failure, permanent failure, and unknown state. A boolean success plus an unstructured message is how ambiguity enters the control loop.

Use stable machine codes such as RATE_LIMITED, PERMISSION_DENIED, and ALREADY_EXISTS. The human-readable message can change without breaking policy.

Test Four Boundaries, Not One

Most teams test the provider API and assume the agent path is covered. It is not.

Model-to-runtime tests verify that tool definitions are visible, names are stable, and generated arguments pass validation. Use a corpus of real user intents, including missing data and ambiguous requests. The desired result may be a clarifying question, not a call.

Runtime-to-adapter tests feed valid and invalid payloads directly into the tool layer. Confirm that invalid inputs never reach external systems and that normalization does not silently change meaning.

Adapter-to-provider tests run against a sandbox or recorded provider contract. Detect renamed fields, new enum values, pagination changes, and altered error bodies.

Response-to-model tests verify that external results become accurate model-visible facts. If a provider returns “accepted for processing,” the adapter must not translate that into “completed.”

Each boundary needs fixtures that are easy to read and hard to reinterpret.

Add Golden Cases for Dangerous Semantics

Schema validity cannot catch every semantic error. Both 100 and 10000 may be valid numbers. A date can match ISO format and still represent the wrong timezone.

Create golden contract cases for high-risk details:

• currency stored in major versus minor units
• local time versus UTC
• inclusive versus exclusive date ranges
• account ID versus display name
• dry-run versus committed action
• draft versus published state
• soft delete versus permanent delete

For each case, assert the exact request sent and exact normalized result returned. Property-based tests can then explore nearby values: zero, negative numbers, maximum length, Unicode, duplicate IDs, and daylight-saving transitions.

Test Failure as a First-Class Output

Agents become dangerous when failure information is vague. Simulate timeouts before the provider receives a request, after it receives a request, and after it completes the action but before your adapter receives confirmation.

Those are three different states. The last one is the nasty case: retrying may duplicate the action. Require idempotency keys for mutations and persist them outside model context.

Test rate limits, expired credentials, revoked permissions, malformed provider responses, partial batch completion, and dependency outages. Confirm which errors the agent may retry and how many times. The model should not invent retry policy from prose.

Also test cancellation. If the user withdraws a request while an action is running, can the runtime stop it? If not, does the final status explain what may still complete?

Prevent Schema Drift in CI

Store tool definitions beside adapter code. Generate model-facing schemas from the same source when possible. If generation is impossible, hash both and fail CI when one changes without an explicit compatibility decision.

Run consumer-driven contract tests against provider sandboxes on a schedule. Production APIs drift even when documentation does not. Alert on new response fields, missing required fields, and previously unseen error codes.

Version tools when meaning changes. Adding an optional field may be backward compatible. Changing amount from dollars to cents is not. Keep old versions during migration, route deliberately, and measure usage before removal.

Evaluate the Agent on Outcomes

Finally, run end-to-end scenarios with a deterministic fake tool server. The server should record every call and return scripted states. Score whether the agent chose the correct tool, supplied valid arguments, respected confirmation rules, handled failure, and described the final state accurately.

The model can vary its wording. The contract cannot vary its meaning.

Reliable agents are not built by hoping a smarter model interprets messy APIs correctly. They are built by turning tool boundaries into executable, versioned agreements—and breaking the build whenever those agreements drift.