AI Coding Agent Observability: Trace Every Decision Before It Ships

Your AI coding agent does not need malice to break production. It only needs one hidden assumption, one skipped test, one tool call you never inspected, and one confident final answer.

That is why ai coding agent observability matters. Not because dashboards are cool. Dashboards are usually where engineering teams go to cosplay control. The real goal is simpler: capture every meaningful decision your coding agent makes before its code lands in your repo.

A good observability setup answers brutal questions fast:

• What prompt caused this change?
• Which files did the agent read before editing?
• Did it run tests or merely say it did?
• Which tool failed silently?
• Did the agent ignore a lint error?
• Why did it choose one implementation over another?
• Can we reproduce the run?

If you cannot answer those, you do not have an AI coding workflow. You have a very expensive intern with root access and vibes.

This tutorial shows how to instrument an AI coding agent workflow using traces, spans, structured events, and review gates. The examples use Python-style pseudocode and OpenTelemetry concepts, but the pattern works whether your agent is built with OpenAI Agents SDK, LangChain, LangGraph, a custom tool runner, or a homegrown script taped together at 1 a.m.

Prerequisites

You do not need to be a full-time observability engineer, but you should understand the basic moving parts.

You need:

• A coding agent or automation that can read files, call tools, edit code, and run commands
• A place to send traces, such as Langfuse, LangSmith, Honeycomb, Datadog, Grafana Tempo, Jaeger, or an OpenTelemetry collector
• A way to wrap each agent step in code
• Access to your CI logs or local test runner
• A policy for sensitive data, because traces can capture prompts, code, environment names, and tool outputs

You should know these terms:

• Trace: One end-to-end agent run, such as “fix issue #421” or “generate migration script.”
• Span: One operation inside the trace, such as reading files, calling an LLM, running tests, or applying a patch.
• Event: A timestamped fact inside a span, such as “test failed” or “approval requested.”
• Attribute: Searchable metadata, such as repo name, branch, model, commit SHA, ticket ID, or risk level.

The important mental model: a trace is the story, spans are the chapters, events are the receipts.

Step 1: Define What You Actually Need To See

Do not start by instrumenting everything. That creates a landfill of JSON and nobody reads it.

Start with the decisions that affect shipping.

For an AI coding agent, the minimum useful trace should include:

• User request
• System or policy instructions
• Repository and branch
• Files inspected
• Files changed
• LLM calls
• Tool calls
• Command outputs
• Test results
• Security checks
• Review decisions
• Final diff summary

A basic trace shape should look like this:

trace: agent_run
  span: intake_request
  span: plan_work
  span: inspect_repo
    span: read_file
    span: search_code
  span: edit_code
    span: apply_patch
  span: verify
    span: run_tests
    span: run_lint
    span: security_scan
  span: review_gate
  span: final_response

Expected result: every agent run has a single trace ID that lets you jump from the original request to the final code change.

If your agent can open pull requests, add one more requirement: the PR description should include the trace ID. That gives reviewers a direct path from diff to reasoning.

Step 2: Create A Trace For Every Agent Run

The first instrumentation rule is boring and non-negotiable: every run gets a trace.

Here is a minimal Python example using OpenTelemetry-style tracing:

from opentelemetry import trace

tracer = trace.get_tracer("egoistai.coding_agent")

def run_agent(request, repo, branch, user_id):
    with tracer.start_as_current_span("agent_run") as span:
        span.set_attribute("agent.request", request)
        span.set_attribute("repo.name", repo)
        span.set_attribute("repo.branch", branch)
        span.set_attribute("user.id", user_id)
        span.set_attribute("workflow.name", "ai_code_change")

        result = execute_agent_workflow(request, repo, branch)

        span.set_attribute("agent.status", result.status)
        span.set_attribute("agent.changed_files_count", len(result.changed_files))
        return result

That is not enough for production, but it gives every run a spine.

Expected result: your observability backend shows one trace per coding task, with attributes you can filter by repo, branch, user, and status.

Good attributes to add at the trace level:

span.set_attribute("model.name", "gpt-5")
span.set_attribute("agent.version", "2026.08.01")
span.set_attribute("ticket.id", "ENG-421")
span.set_attribute("deployment.env", "staging")
span.set_attribute("risk.level", "medium")

Do not put giant blobs in attributes. Attributes are for filtering and grouping. Put full prompts, command output, and diffs in span events or external artifacts.

Step 3: Trace The Agent’s Planning Step

The planning step is where bad work usually begins. If the agent misunderstands the request, everything after that is just a polished mistake.

Wrap the planning call:

def plan_work(request, context):
    with tracer.start_as_current_span("plan_work") as span:
        span.set_attribute("input.request_length", len(request))
        span.set_attribute("context.files_count", len(context.files))

        plan = llm.generate(
            system="You are a coding agent. Produce a concise implementation plan.",
            user=request,
            context=context.summary,
        )

        span.add_event("plan.generated", {
            "plan.text": plan.text,
            "plan.steps_count": len(plan.steps),
        })

        return plan

Expected result: when a run goes sideways, you can inspect the plan and see whether the agent was wrong from the start or failed later during execution.

A useful planning span captures:

• The summarized request
• Key constraints
• Assumptions
• Planned files to inspect
• Planned verification commands
• Risk estimate

Do not accept plans that say “update the code” or “fix the issue.” That is filler. A useful plan names files, commands, and expected outcomes.

Step 4: Record File Reads And Code Search

Most coding agents fail because they edit before understanding the repo. Observability should expose that.

When the agent searches code, trace it:

def search_code(query, path="."):
    with tracer.start_as_current_span("search_code") as span:
        span.set_attribute("search.query", query)
        span.set_attribute("search.path", path)

        matches = ripgrep(query, path)

        span.set_attribute("search.match_count", len(matches))
        span.add_event("search.completed", {
            "matches.preview": matches[:10],
        })

        return matches

When it reads a file, trace that too:

def read_file(path):
    with tracer.start_as_current_span("read_file") as span:
        span.set_attribute("file.path", path)

        content = open(path, "r", encoding="utf-8").read()

        span.set_attribute("file.bytes", len(content.encode("utf-8")))
        span.set_attribute("file.lines", content.count("\n") + 1)

        return content

Expected result: reviewers can see whether the agent inspected the right files before changing code.

This is also where you catch nonsense. If an agent modifies billing.ts after reading only README.md, your review gate should light up like a crime scene.

Add a simple heuristic:

def validate_context_before_edit(files_read, files_to_edit):
    missing_context = [
        path for path in files_to_edit
        if path not in files_read
    ]

    if missing_context:
        raise AgentPolicyError(
            f"Agent tried to edit files it did not read: {missing_context}"
        )

This rule is not perfect. Sometimes generated files or simple config edits are fine. But as a default, it kills a shocking amount of lazy agent behavior.

Step 5: Trace LLM Calls Without Leaking Secrets

LLM calls are the heart of the run, but they are also where sensitive data can spill into logs.

Capture enough to debug:

def call_model(messages, model, purpose):
    with tracer.start_as_current_span("llm_call") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.purpose", purpose)
        span.set_attribute("llm.message_count", len(messages))

        sanitized_messages = redact_messages(messages)

        span.add_event("llm.input", {
            "messages": sanitized_messages,
        })

        response = client.responses.create(
            model=model,
            input=messages,
        )

        span.set_attribute("llm.output_tokens", response.usage.output_tokens)
        span.set_attribute("llm.input_tokens", response.usage.input_tokens)

        span.add_event("llm.output", {
            "text": redact_text(response.output_text),
        })

        return response

Expected result: you can debug model behavior while reducing the chance that API keys, customer data, or private credentials end up in traces.

Your redaction layer should catch:

• API keys
• Bearer tokens
• Passwords
• Private keys
• Session cookies
• Customer email addresses if not needed
• Environment variable dumps
• Internal URLs if sensitive

A basic redactor:

import re

SECRET_PATTERNS = [
    re.compile(r"sk-[A-Za-z0-9_-]{20,}"),
    re.compile(r"Bearer\s+[A-Za-z0-9._-]+"),
    re.compile(r"-----BEGIN PRIVATE KEY-----.*?-----END PRIVATE KEY-----", re.S),
    re.compile(r"password\s*=\s*['\"][^'\"]+['\"]", re.I),
]

def redact_text(text):
    redacted = text
    for pattern in SECRET_PATTERNS:
        redacted = pattern.sub("[REDACTED]", redacted)
    return redacted

def redact_messages(messages):
    return [
        {
            **message,
            "content": redact_text(str(message.get("content", ""))),
        }
        for message in messages
    ]

OpenAI’s Agents SDK tracing documentation notes that tracing can capture generation and function inputs and outputs, and it provides controls for sensitive data capture. Use those controls. Do not discover your secret-handling policy after the first incident.

Step 6: Trace Tool Calls Like They Are Production APIs

A coding agent’s tools are its hands. If you only trace the model and ignore the tools, you are watching a chef think while refusing to look at the knife.

Every tool call should capture:

• Tool name
• Input arguments
• Start time
• End time
• Exit status
• Output summary
• Error message
• Retry count

Example:

def run_command(command, cwd):
    with tracer.start_as_current_span("tool.run_command") as span:
        span.set_attribute("tool.name", "run_command")
        span.set_attribute("command.cwd", cwd)
        span.set_attribute("command.text", command)

        result = subprocess.run(
            command,
            cwd=cwd,
            shell=True,
            text=True,
            capture_output=True,
            timeout=120,
        )

        span.set_attribute("command.exit_code", result.returncode)
        span.set_attribute("command.stdout_bytes", len(result.stdout))
        span.set_attribute("command.stderr_bytes", len(result.stderr))

        span.add_event("command.output", {
            "stdout.preview": result.stdout[-4000:],
            "stderr.preview": result.stderr[-4000:],
        })

        if result.returncode != 0:
            span.set_attribute("error", True)
            span.add_event("command.failed", {
                "error.message": result.stderr[-1000:],
            })

        return result

Expected result: no more “the agent said tests passed” without proof. You can inspect the actual command, exit code, and output.

One sharp rule: never let the agent summarize command success without storing the raw exit code. Natural language is not a test result.

Step 7: Capture Diffs As First-Class Evidence

The diff is the artifact that ships. Treat it as evidence, not decoration.

Trace every patch:

def apply_code_patch(patch, changed_files):
    with tracer.start_as_current_span("apply_patch") as span:
        span.set_attribute("patch.changed_files", len(changed_files))
        span.set_attribute("patch.bytes", len(patch.encode("utf-8")))

        for path in changed_files:
            span.add_event("file.changed", {
                "file.path": path,
            })

        result = apply_patch_to_repo(patch)

        span.set_attribute("patch.applied", result.success)

        if not result.success:
            span.set_attribute("error", True)
            span.add_event("patch.failed", {
                "error.message": result.error,
            })

        return result

Expected result: every code edit is tied to the reasoning and tool calls that led to it.

For larger diffs, do not stuff the entire patch into your tracing backend. Store the diff in object storage, CI artifacts, or the pull request itself, then put a link in the trace:

span.set_attribute("diff.artifact_url", artifact_url)
span.set_attribute("git.commit_sha", commit_sha)

This keeps tracing useful instead of turning it into a bloated document store.

Step 8: Add Verification Spans For Tests, Lint, Types, And Security

The verification phase should be painfully explicit.

Create separate spans for each check:

def verify_change(repo):
    checks = [
        ("tests", "npm test"),
        ("lint", "npm run lint"),
        ("types", "npm run typecheck"),
        ("security", "npm audit --audit-level=high"),
    ]

    results = []

    with tracer.start_as_current_span("verify") as span:
        for name, command in checks:
            result = run_check(name, command, repo)
            results.append(result)

        failed = [r.name for r in results if r.exit_code != 0]

        span.set_attribute("verify.failed_count", len(failed))
        span.set_attribute("verify.passed", len(failed) == 0)

        if failed:
            span.add_event("verify.failed", {
                "failed.checks": failed,
            })

    return results

And the individual check:

def run_check(name, command, repo):
    with tracer.start_as_current_span(f"check.{name}") as span:
        span.set_attribute("check.name", name)
        span.set_attribute("check.command", command)

        result = run_command(command, repo)

        span.set_attribute("check.exit_code", result.returncode)
        span.set_attribute("check.passed", result.returncode == 0)

        return CheckResult(name=name, exit_code=result.returncode)

Expected result: the trace shows exactly which checks ran and whether they passed.

For coding agents, “verified” should mean the command ran and returned a passing exit code. Not “the model believes this compiles.” Not “the output looked okay.” Not “the agent forgot but remained emotionally committed.”

Step 9: Add A Review Gate Before Shipping

Observability is not only for debugging after failure. It should stop bad changes before they merge.

A review gate reads the trace and decides whether the run is allowed to proceed.

Example policy:

def review_gate(trace_summary):
    blockers = []

    if trace_summary.changed_files_count > 0 and trace_summary.tests_run_count == 0:
        blockers.append("Code changed but no tests ran.")

    if trace_summary.failed_checks:
        blockers.append(f"Failed checks: {trace_summary.failed_checks}")

    if trace_summary.edited_unread_files:
        blockers.append(f"Edited files without reading them: {trace_summary.edited_unread_files}")

    if trace_summary.secret_detected:
        blockers.append("Potential secret detected in trace or diff.")

    if trace_summary.risk_level == "high" and not trace_summary.human_approved:
        blockers.append("High-risk change requires human approval.")

    return ReviewDecision(
        approved=not blockers,
        blockers=blockers,
    )

Expected result: the agent cannot quietly skip verification and still ship.

High-risk changes should include:

• Authentication
• Authorization
• Billing
• Payments
• Data deletion
• Database migrations
• Security headers
• Encryption
• Dependency upgrades
• Generated infrastructure config
• Anything touching customer data

When the gate blocks a run, store the decision in the trace:

span.add_event("review.blocked", {
    "blockers": blockers,
})
span.set_attribute("review.approved", False)

When it passes:

span.add_event("review.approved", {
    "approved_by": "policy:auto",
})
span.set_attribute("review.approved", True)

Step 10: Build A Useful Trace Summary

Nobody wants to click through 80 spans just to review a tiny CSS fix. Add a summary that gives reviewers the useful stuff first.

A good summary includes:

• Request
• Agent version
• Model
• Files read
• Files changed
• Commands run
• Checks passed or failed
• Risk level
• Human approval status
• Trace URL

Example PR comment:

## Agent Run Summary

Trace: https://observability.example.com/traces/trace_abc123

Request:
Fix the failing checkout total calculation test.

Files read:
- src/checkout/calculateTotal.ts
- src/checkout/calculateTotal.test.ts
- package.json

Files changed:
- src/checkout/calculateTotal.ts
- src/checkout/calculateTotal.test.ts

Verification:
- npm test: passed
- npm run lint: passed
- npm run typecheck: passed

Risk:
Medium - billing-adjacent calculation logic

Review gate:
Passed

Expected result: reviewers get the executive version, with a trace link for the ugly details.

This is where observability becomes practical. You are not asking humans to blindly trust the agent. You are giving them a receipt stack.

Common Pitfalls

Capturing Everything And Reviewing Nothing

Full traces are useful. Infinite traces are not.

If every run captures megabytes of unfiltered prompts, full file contents, and giant terminal outputs, the system becomes expensive and unreadable. Capture previews by default. Store big artifacts somewhere cheaper. Link to them.

Good default limits:

• Last 4,000 characters of command output
• File metadata instead of full file contents
• Full diff only as an external artifact
• Redacted prompts unless deep debugging is enabled
• Sampling for low-risk successful runs
• Full retention for failed or blocked runs

Forgetting Short-Lived Process Flushes

Many coding agents run as short-lived jobs in CI or task queues. They start, do work, and exit. If your tracing SDK exports in the background, the process may die before traces flush.

Add an explicit flush at the end:

try:
    result = run_agent(request, repo, branch, user_id)
finally:
    tracer_provider.force_flush(timeout_millis=5000)

Expected result: traces show up reliably even when the agent job exits quickly.

Treating Logs And Traces As The Same Thing

Logs say what happened. Traces show how one thing led to the next.

For coding agents, causality matters. You need to connect the user request, model reasoning, file reads, patch, tests, and final response. A pile of logs without a trace ID is a junk drawer.

Use logs for raw detail. Use traces for the workflow.

Hiding Failures Behind Friendly Summaries

Agents love to be helpful. Sometimes that means they turn “tests failed” into “there may be a minor issue with the test environment.”

No. Store the exit code. Store the stderr preview. Store the failed check name. Make failure boring and obvious.

Ignoring Privacy Until Legal Notices You Exist

Traces can contain sensitive code, customer context, credentials, and private prompts. Decide early what gets captured, redacted, sampled, and retained.

At minimum:

• Redact secrets before export
• Disable sensitive payload capture where your SDK allows it
• Separate development, staging, and production traces
• Restrict trace access by repo or team
• Set retention windows
• Avoid sending private customer data unless absolutely needed

Observability should reduce risk, not create a second breach surface with a nicer UI.

What Good Looks Like

A mature AI coding agent observability setup lets you answer these questions in under a minute:

• Show me every agent run that touched authentication last week.
• Which model produced this bad migration?
• Did the agent read the test file before editing the implementation?
• Which runs skipped type checks?
• How often do security-related changes require human intervention?
• Which tool causes the most failures?
• Are small dependency updates passing automatically?
• Which prompts lead to the highest rollback rate?

That last one matters. Once traces are structured, you can improve the agent itself. You can compare prompts, models, tools, repo instructions, and verification policies. Without traces, every failure is folklore.

Minimal Production Checklist

Before your coding agent can open or merge pull requests, require this:

• One trace per agent run
• Trace ID attached to PRs or commits
• Spans for planning, file inspection, edits, commands, tests, and review
• Raw command exit codes captured
• Sensitive data redaction enabled
• Failed checks marked as blockers
• High-risk files routed to human approval
• Trace flushing for short-lived jobs
• Retention policy defined
• Access control configured

That is the floor. Not the deluxe package. The floor.

Final Takeaway

AI coding agents are moving from toy demos into real engineering workflows. That means the standard has to change.

Do not ask, “Did the agent seem smart?”

Ask, “Can we trace what it did, why it did it, what it changed, what it verified, and who approved it?”

If the answer is no, keep the agent away from production. If the answer is yes, you have something worth scaling: not blind automation, but inspectable automation with receipts.