Build a Coding Agent Evaluation Harness: Test Patches Before They Reach Production

Coding agents are very good at producing a patch that looks finished.

That is not the same as producing a patch you should merge.

A polished diff can compile while solving the wrong problem. It can make the target test pass by weakening the assertion, introduce a security regression in an unrelated path, or quietly rewrite far more code than the task required. The dangerous part is not that agents fail. It is that their failures often resemble competent engineering from six feet away.

The answer is an evaluation harness: a repeatable system that gives an agent a task inside a controlled repository, runs the resulting patch through independent checks, and scores the outcome against evidence the agent did not see.

Start With a Task Contract

Every evaluation needs a contract that separates the problem from the proof.

The task description should contain what a developer would reasonably receive: the bug report, feature request, relevant constraints, and repository state. It should not reveal the hidden tests or the exact implementation expected.

A useful task record includes:

• repository and commit SHA
• issue text
• allowed directories
• setup command
• public test command
• time and resource limits
• expected output format
• hidden verification checks

Pinning the commit matters. If the repository moves between runs, scores stop being comparable. Container images, package-lock files, tool versions, and environment variables should be pinned for the same reason.

Keep the success criteria behavioral. “Change parser.ts” is an implementation instruction. “The parser must accept nested arrays while preserving the existing error format” is a contract. Agents should be free to find a good patch, but the evaluator should be strict about observable behavior.

Build Tasks From Real Failures

Synthetic toy tasks are convenient and often misleading. A model that fixes a five-line function may still collapse when the repository has migrations, generated types, flaky integration tests, and an unfamiliar build system.

Collect tasks from resolved issues and incidents. Rewind the repository to the commit before the fix, remove the original patch, and preserve tests that demonstrate both the bug and its surrounding behavior.

Avoid leaking the answer through filenames, comments, fixtures, or commit messages. If the hidden test is named test_fix_null_pointer_by_adding_guard, the benchmark is measuring search skills more than engineering.

Define More Than Pass or Fail

A binary score is useful but incomplete. Track:

• task success
• regression-test pass rate
• build and type-check status
• lint violations
• security-policy violations
• changed lines and files
• wall-clock time
• model and tool cost
• number of retries

These metrics reveal tradeoffs. One agent may solve more tasks by rewriting large sections of the repository. Another may solve slightly fewer while producing small, reviewable patches. A team deciding what can run autonomously should care about both.

Run Every Attempt in a Disposable Sandbox

Never evaluate a coding agent in a working directory that matters.

Create a fresh container or isolated virtual machine for each attempt. Mount only the repository snapshot and the minimum credentials required for the task. Most benchmark tasks need no production credentials at all.

Disable outbound network access unless the task explicitly requires it. Package installation should come from a controlled cache or an allowlisted registry. Otherwise an agent can accidentally depend on a changing remote resource, leak repository data, or download a tool that makes the run impossible to reproduce.

Set hard limits for time, memory, CPU, disk, process count, and output size. A broken test loop should end as a failed attempt, not become a surprise infrastructure bill.

Capture the Full Execution Trace

Store the initial prompt, tool calls, command results, final diff, test output, timing, and environment metadata. Redact secrets before the trace leaves the sandbox.

The trace is not just for debugging the model. It helps you debug the benchmark. If every agent fails while installing dependencies, the task is testing your setup script. If the hidden test depends on the current date, you have built a lottery.

Preserve the final Git diff separately from the agent’s explanation. The explanation can be persuasive and completely wrong. The repository state is the evidence.

Prevent Evaluation Tampering

Agents should not be able to edit the evaluator, hidden tests, CI configuration, or baseline snapshots. Keep those assets outside the writable repository mount and copy them in only after the agent finishes.

Check for suspicious changes before running tests:

• deleted or skipped tests
• altered test discovery
• relaxed assertions
• disabled type checks
• new shell hooks
• modifications outside the allowed path

Do not reward a green test suite obtained by turning the test suite off.

Grade the Patch Independently

Evaluation should happen in stages, from cheap deterministic checks to slower judgment.

First, verify the repository is intact and the diff stays inside scope. Then run formatting, linting, type checks, and targeted tests. Next run the hidden regression suite and broader tests. Finally, apply security and maintainability checks.

Use an LLM grader only where deterministic evidence cannot answer the question. A model can help assess whether a migration plan is understandable or an error message meets a style rubric. It should not overrule a failing test or invent credit for code that does not run.

Use Hidden Tests Carefully

Hidden tests should verify requirements that were clear from the task, not surprise the agent with secret product decisions. Include:

• the reported failure
• boundary conditions
• representative regressions
• failure-mode behavior
• idempotency where relevant
• authorization and validation checks

Avoid testing one exact implementation. Two different patches can be equally correct. The harness should reward behavior, safety, and compatibility.

Mutation testing can expose weak tasks. Deliberately break the expected behavior in several ways and confirm the hidden suite catches those mutations. If obvious wrong patches pass, the benchmark is not ready.

Compare Against Human and Baseline Runs

Run the same tasks with a simple baseline and, where possible, with experienced developers. A benchmark where everyone scores zero is not automatically difficult; it may be broken.

Track results by task family. Repository navigation, API changes, dependency upgrades, UI bugs, concurrency failures, and security fixes exercise different capabilities. One global score hides where an agent is dependable.

Put the Harness in CI

Once the harness is stable, run it on every agent, prompt, tool, and model change.

Keep a small smoke suite for pull requests and a larger nightly suite for deeper comparison. Store results with the exact configuration: model version, system prompt hash, tool schema, sandbox image, and repository commits.

Set merge thresholds before looking at the new score. Otherwise every regression acquires an excuse.

A practical release gate might require:

1. no drop in task success beyond a small tolerance
2. zero new security-policy violations
3. no meaningful increase in out-of-scope edits
4. cost and latency within budget
5. manual review of newly changed failures

When a production incident involves an agent-generated patch, turn it into a new evaluation task. The harness should accumulate scar tissue.

The Bottom Line

Do not measure a coding agent by how impressive its diff looks or how confidently it summarizes the work.

Give it a pinned repository and a realistic task. Run it in a disposable sandbox. Keep the proof hidden and independent. Score behavior, regressions, scope, security, cost, and time. Preserve every trace, and promote real failures into permanent tests.

The goal is not to prove that an agent can write code. That part is already obvious. The goal is to learn exactly when its patches deserve trust—and when they still need a human standing between the diff and production.