AI Agent Tool Contract Testing: Catch Breaking Changes Before Production

Your API can remain technically online while every agent quietly becomes incompetent. Rename customer_id to account_id, tighten an enum, or change an error shape and the model may keep calling the tool—just incorrectly. AI agent tool contract testing catches that drift before users discover it.

The goal is not to test model intelligence. It is to prove that the declared tool, the runtime implementation and the agent’s expected recovery behavior still agree.

Prerequisites

You need the machine-readable tool schema, a staging implementation and a small set of recorded calls with secrets removed. JSON Schema examples are used below, but the method works with MCP tools, function calling and internal RPC wrappers.

Step 1: version the whole contract

A tool contract includes more than its input JSON:

• tool name and description;
• required and optional arguments;
• output and error shapes;
• permissions and side effects;
• idempotency behavior;
• timeouts and rate limits.

Store this manifest beside the service code.

{
  "name": "create_invoice",
  "version": "2.1.0",
  "sideEffect": "financial_record_create",
  "idempotencyKey": true,
  "inputSchema": {
    "type": "object",
    "required": ["account_id", "amount_cents", "currency"]
  }
}

Use semantic versioning as communication, not magic. Removing a field or changing meaning is major; adding an optional field is minor; description clarification is patch.

Step 2: validate provider and consumer expectations

Provider tests confirm that the service honors the schema. Consumer tests confirm that the agent integration still sends valid calls and understands responses.

def test_create_invoice_contract(tool_client, schema):
    result = tool_client.create_invoice(
        account_id="test-account",
        amount_cents=2500,
        currency="USD",
        idempotency_key="contract-001",
    )
    validate(instance=result, schema=schema["outputSchema"])

Do not let the test create real financial records. Point it at an isolated tenant or a deterministic simulator.

Step 3: keep golden tool-choice cases

Create short prompts where the correct action is unambiguous. Record the expected tool name and structural constraints, not the exact natural-language reasoning.

- prompt: "Create a $25 invoice for test account A in USD"
  expected_tool: create_invoice
  required_args: [account_id, amount_cents, currency, idempotency_key]
- prompt: "Show me invoice 123"
  expected_tool: get_invoice
  forbidden_tools: [create_invoice]

Run these against the production model configuration with temperature and model version recorded. Allow small nondeterminism by retrying or scoring a batch, but fail if dangerous tool selection appears even once.

Step 4: test compatibility both directions

New agents may reach old tool servers during rolling deploys, and old agents may reach new servers. Test N against N, N against N-1 and N-1 against N.

For an argument rename, support an overlap window:

{
  "anyOf": [
    {"required": ["account_id"]},
    {"required": ["customer_id"]}
  ]
}

Log use of the old field, migrate callers, then remove it in a major version. Silent coercion without telemetry creates permanent ambiguity.

Step 5: make errors part of the contract

Agents often recover from RATE_LIMITED differently than PERMISSION_DENIED. If the server replaces stable codes with free-form prose, recovery becomes guesswork.

{
  "ok": false,
  "error": {
    "code": "PERMISSION_DENIED",
    "retryable": false,
    "request_id": "req_test_42"
  }
}

Test timeout, validation, conflict, rate-limit and permission paths. Verify that retryable errors use bounded backoff and non-retryable errors escalate instead of looping.

Step 6: verify side-effect safety

For write tools, send the same idempotency key twice and assert that only one record exists. Simulate a timeout after the server commits but before the client receives the response. This is the failure that creates duplicate emails, tickets and payments.

Also test approval boundaries. A schema change must not accidentally convert a read-only tool into a write path or remove a confirmation flag.

Step 7: put the suite in the deployment gate

Run static schema diffing on every pull request. Run provider tests on service changes and a small model-backed golden suite before production. Nightly jobs can cover larger prompt variations.

Fail the gate on removed required fields, widened permissions, changed side effects, unknown error codes or statistically meaningful drops in valid calls.

Common pitfalls

Snapshotting entire model responses makes tests brittle. Assert the call structure and safety properties instead. Testing only happy paths misses the retry loops that cause real incidents. Using live production accounts makes tests dangerous. And validating JSON without validating meaning can still pass amount_cents: 25 when the user meant $25.

Expected result

After implementation, a breaking schema change produces a clear CI failure naming the incompatible field and affected consumer. A model upgrade produces a scorecard of tool-selection, argument-validity and safety regressions. Runtime errors carry stable codes that the agent can handle predictably.

The practical takeaway: tool schemas are APIs with probabilistic callers. Contract tests provide the deterministic boundary that keeps an agent system deployable.