Deterministic Browser Replay for AI Agents: Debug Every Click and Failure

Your AI browser agent did not “randomly fail.” It clicked something, saw something, received something, and chose something. You just failed to capture the evidence.

That is the whole point of deterministic browser replay ai agents workflows: turn vague agent chaos into a repeatable debugging loop. Instead of reading a sad stack trace that says TimeoutError: locator.click, you get the actual page state, click target, network response, console output, model decision, and screenshot at the moment everything went sideways.

This tutorial shows you how to build that loop with Playwright, trace files, recorded network responses, seeded agent decisions, and structured logs.

No corporate observability fog machine. Just a practical system for answering: “What exactly did the agent do, and can I replay it?”

What Deterministic Browser Replay Actually Means

Full browser determinism is brutal. Modern web apps have timers, animations, ads, A/B tests, third-party scripts, service workers, flaky APIs, rate limits, auth sessions, and DOMs that mutate while you blink.

So do not chase perfect determinism. Chase useful determinism.

For AI browser agents, deterministic replay means you capture enough of the run to reproduce the failure under controlled conditions:

The starting URL and browser context
Viewport, locale, timezone, permissions, storage state, cookies
Agent instructions and tool definitions
Model outputs or action decisions
Browser actions: clicks, fills, keypresses, navigation
DOM snapshots, screenshots, console logs, and network activity
API responses or HAR files when possible
The final error, timeout, or assertion failure

The goal is simple: when an agent fails on step 47, you should not rerun the entire job and pray. You should open a trace, jump to step 47, inspect the before-and-after state, and replay the suspicious part with the same inputs.

Prerequisites

You need basic comfort with the command line and JavaScript or TypeScript. You do not need to be a Playwright expert.

Install Node.js 20 or newer, then create a small project:

mkdir agent-replay-demo
cd agent-replay-demo
npm init -y
npm install -D typescript tsx @types/node
npm install playwright
npx playwright install chromium

Create this minimal structure:

agent-replay-demo/
  src/
    agent.ts
    replay-log.ts
    run.ts
  traces/
  recordings/
  package.json

Add scripts to package.json:

{
  "scripts": {
    "agent": "tsx src/run.ts",
    "trace": "npx playwright show-trace traces/latest.zip"
  }
}

Expected result: npm run agent will eventually launch the agent run, and npm run trace will open the most recent Playwright trace.

Step 1: Stop Treating Agent Actions Like Magic

Most bad agent systems log the prompt and the final error. That is not enough.

A browser agent needs an action ledger. Every click, fill, navigation, wait, model decision, and observed page state should get a stable event ID.

Create src/replay-log.ts:

import fs from "node:fs";
import path from "node:path";

export type ReplayEvent = {
  id: number;
  timestamp: string;
  type: string;
  data: Record<string, unknown>;
};

export class ReplayLog {
  private events: ReplayEvent[] = [];
  private nextId = 1;

  constructor(private filePath: string) {
    fs.mkdirSync(path.dirname(filePath), { recursive: true });
  }

  add(type: string, data: Record<string, unknown> = {}) {
    const event: ReplayEvent = {
      id: this.nextId++,
      timestamp: new Date().toISOString(),
      type,
      data
    };

    this.events.push(event);
    fs.writeFileSync(this.filePath, JSON.stringify(this.events, null, 2));
    return event;
  }
}

This is deliberately boring. Boring logs save expensive debugging hours.

Expected result: each agent run writes a JSON file like recordings/latest.json with ordered events.

Step 2: Wrap Browser Actions

Do not let your agent call Playwright directly from everywhere. Put a tiny wrapper around browser actions so every operation is recorded.

Create src/agent.ts:

import { Page } from "playwright";
import { ReplayLog } from "./replay-log";

export class BrowserAgent {
  constructor(
    private page: Page,
    private log: ReplayLog
  ) {}

  async goto(url: string) {
    this.log.add("browser.goto.start", { url });
    await this.page.goto(url, { waitUntil: "domcontentloaded" });
    this.log.add("browser.goto.done", {
      url: this.page.url(),
      title: await this.page.title()
    });
  }

  async click(selector: string, reason: string) {
    this.log.add("browser.click.start", { selector, reason });

    const element = this.page.locator(selector).first();
    await element.highlight().catch(() => {});
    await element.click({ timeout: 10_000 });

    this.log.add("browser.click.done", {
      selector,
      url: this.page.url()
    });
  }

  async fill(selector: string, value: string, reason: string) {
    this.log.add("browser.fill.start", {
      selector,
      value: redact(value),
      reason
    });

    await this.page.locator(selector).first().fill(value);

    this.log.add("browser.fill.done", {
      selector,
      url: this.page.url()
    });
  }

  async snapshot(label: string) {
    this.log.add("browser.snapshot", {
      label,
      url: this.page.url(),
      title: await this.page.title(),
      bodyText: (await this.page.locator("body").innerText()).slice(0, 2000)
    });
  }
}

function redact(value: string) {
  if (value.length > 80) return "[redacted-long-value]";
  return value.replace(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi, "[email]");
}

The reason field matters. An AI agent should explain why it clicked a button or filled a field. When debugging, “clicked button:nth-child(3)” is weak. “Clicked the pricing tab because the user asked for subscription details” is useful.

Expected result: every action now has intent attached to it.

Step 3: Enable Playwright Tracing

Playwright tracing gives you the visual debugging layer: actions, snapshots, screenshots, console output, source locations, and network activity. This is where agent debugging stops being guesswork.

Create src/run.ts:

import { chromium } from "playwright";
import { BrowserAgent } from "./agent";
import { ReplayLog } from "./replay-log";

async function main() {
  const log = new ReplayLog("recordings/latest.json");

  const browser = await chromium.launch({
    headless: false
  });

  const context = await browser.newContext({
    viewport: { width: 1280, height: 800 },
    locale: "en-US",
    timezoneId: "America/New_York"
  });

  await context.tracing.start({
    screenshots: true,
    snapshots: true,
    sources: true
  });

  const page = await context.newPage();

  page.on("console", (message) => {
    log.add("browser.console", {
      type: message.type(),
      text: message.text()
    });
  });

  page.on("pageerror", (error) => {
    log.add("browser.pageerror", {
      message: error.message,
      stack: error.stack
    });
  });

  const agent = new BrowserAgent(page, log);

  try {
    log.add("run.start", {
      goal: "Find the Playwright Trace Viewer documentation"
    });

    await agent.goto("https://playwright.dev/");
    await agent.snapshot("home");

    await agent.click("text=Docs", "Open documentation");
    await agent.snapshot("docs");

    await agent.click("text=Trace viewer", "Open trace viewer documentation");
    await agent.snapshot("trace-viewer");

    log.add("run.done", { status: "success" });
  } catch (error) {
    log.add("run.error", {
      message: error instanceof Error ? error.message : String(error),
      stack: error instanceof Error ? error.stack : undefined
    });

    throw error;
  } finally {
    await context.tracing.stop({ path: "traces/latest.zip" });
    await browser.close();
  }
}

main().catch((error) => {
  console.error(error);
  process.exit(1);
});

Run it:

npm run agent
npm run trace

Expected result: Playwright opens a trace viewer. You can click each action and inspect what the browser saw before and after that action. You should also have recordings/latest.json with your custom agent event stream.

Step 4: Capture Model Decisions Separately

If your browser agent uses an LLM, the model is part of the runtime. Log it like one.

Do not only log the raw final answer. Capture the decision boundary:

Input goal
Condensed page observation
Available actions
Model output
Parsed action
Validation result
Retry count

Example event:

{
  "id": 12,
  "timestamp": "2026-08-03T10:05:21.314Z",
  "type": "agent.decision",
  "data": {
    "step": 4,
    "observationHash": "sha256:0f4d...",
    "availableActions": ["click", "fill", "goto", "stop"],
    "modelOutput": {
      "action": "click",
      "selector": "text=Trace viewer",
      "reason": "The user asked to inspect trace debugging docs."
    },
    "temperature": 0,
    "seed": 12345
  }
}

For deterministic replay, temperature should usually be 0 during debug runs. If your model provider supports a seed parameter, log it. If it does not, store the selected action and replay from that action instead of asking the model again.

That distinction is important. You are not trying to prove the model will always produce the same token sequence. You are trying to reproduce the browser behavior caused by a specific model decision.

Step 5: Add a Replay Mode

Now split execution into two modes:

live: ask the model what to do
replay: load previous decisions and execute them again

A simple replay file can look like this:

[
  {
    "action": "goto",
    "url": "https://playwright.dev/"
  },
  {
    "action": "click",
    "selector": "text=Docs",
    "reason": "Open documentation"
  },
  {
    "action": "click",
    "selector": "text=Trace viewer",
    "reason": "Open trace viewer documentation"
  }
]

Then your runner can execute those recorded actions:

type RecordedAction =
  | { action: "goto"; url: string }
  | { action: "click"; selector: string; reason: string }
  | { action: "fill"; selector: string; value: string; reason: string };

async function replayActions(agent: BrowserAgent, actions: RecordedAction[]) {
  for (const action of actions) {
    if (action.action === "goto") {
      await agent.goto(action.url);
    }

    if (action.action === "click") {
      await agent.click(action.selector, action.reason);
    }

    if (action.action === "fill") {
      await agent.fill(action.selector, action.value, action.reason);
    }

    await agent.snapshot(`after-${action.action}`);
  }
}

Expected result: you can rerun the same browser path without invoking the model. That is the first big debugging win.

When the replay fails, you know the problem is probably browser state, page drift, network drift, timing, auth, or selectors. When the replay succeeds, but the live agent fails, the problem is probably observation quality, prompt design, model choice, tool schema, or action validation.

That split saves you from debugging everything at once.

Step 6: Make Network Less Slippery

Browser agents often fail because the page changed underneath them. A pricing page loads different copy. A dashboard API returns an empty state. A login flow triggers a captcha. A recommendation endpoint returns a different card order.

For critical workflows, capture network traffic.

Playwright can record HAR files from a browser context:

const context = await browser.newContext({
  viewport: { width: 1280, height: 800 },
  recordHar: {
    path: "recordings/latest.har",
    content: "embed"
  }
});

For replay, route requests from the HAR:

await context.routeFromHAR("recordings/latest.har", {
  update: false
});

Expected result: many API responses become stable during replay. Your agent sees the same backend data it saw during the original run.

Use this carefully. HAR replay is excellent for debugging a specific failure. It is not a replacement for live integration tests, because it can hide real production changes.

Step 7: Save Browser State

A replay that starts logged out when the original run started logged in is not a replay. It is a different test wearing the same jacket.

Save storage state after setup:

await context.storageState({
  path: "recordings/storage-state.json"
});

Use it during replay:

const context = await browser.newContext({
  storageState: "recordings/storage-state.json",
  viewport: { width: 1280, height: 800 },
  locale: "en-US",
  timezoneId: "America/New_York"
});

Expected result: cookies and local storage are restored, so auth-dependent workflows start from the same baseline.

Do not commit real user cookies or session tokens. Add this to .gitignore:

recordings/*.har
recordings/*.json
traces/*.zip

If you need to share traces internally, scrub them first. Browser traces and HAR files can contain URLs, request bodies, headers, customer data, and auth material.

Step 8: Use Stable Selectors, Not Vibes

If your agent clicks by visible text only, replay will be fragile. If it clicks generated class names, replay will be worse.

Prefer selectors in this order:

Accessibility role and name
Test IDs
Stable labels
Text selectors for content-like pages
CSS only when the structure is intentionally stable

Good:

await page.getByRole("button", { name: "Submit" }).click();
await page.getByTestId("checkout-submit").click();
await page.getByLabel("Email").fill("[email protected]");

Risky:

await page.locator(".css-1x9k3a").click();
await page.locator("div > div:nth-child(4) > button").click();

For AI agents, add a selector validation layer. Before executing a click, check how many elements match. If it is zero, fail with a useful error. If it is more than one, ask the agent to disambiguate or choose a stricter locator.

async function assertSingleMatch(page: Page, selector: string) {
  const count = await page.locator(selector).count();

  if (count !== 1) {
    throw new Error(`Selector "${selector}" matched ${count} elements`);
  }
}

Expected result: failures happen before the bad click, not five steps later when the page is already in nonsense mode.

Step 9: Add Screenshots at Decision Points

Playwright traces already include screenshots when tracing is enabled, but explicit screenshots are still useful for quick triage, issue reports, and CI artifacts.

await page.screenshot({
  path: `recordings/step-${stepNumber}.png`,
  fullPage: true
});

Take screenshots at these moments:

Before the first model decision
Before every browser action selected by the model
After navigation
On timeout
On final failure

Expected result: someone can scan the failure without opening the full trace immediately.

Common Pitfalls

Pitfall 1: Replaying the Model Instead of the Action

If you ask the model again during replay, you are running a new experiment. Useful sometimes, but not deterministic.

Fix it by storing the chosen action and replaying that exact action.

Pitfall 2: Forgetting the Starting State

Viewport, timezone, locale, permissions, cookies, local storage, and URL all matter. A mobile viewport can hide a button behind a menu. A different timezone can change a date picker. Missing cookies can send the agent into login purgatory.

Fix it by storing context configuration and storage state with every run.

Pitfall 3: Logging Secrets

Agent traces can leak everything: bearer tokens, customer names, prompts, emails, cookies, internal URLs, and request bodies.

Fix it with redaction at the logging boundary. Redact before writing files. Do not rely on people remembering to clean files later.

Pitfall 4: Trusting Text Selectors Too Much

Text changes. Marketing pages especially love changing text because someone discovered a new adjective.

Fix it with roles, labels, and test IDs where you control the app. Use text selectors only when browsing third-party content where the text is the target.

Pitfall 5: Treating Replay as a Test Suite

Replay is a debugging weapon. It reproduces one captured run. A test suite verifies expected behavior across fresh runs.

Use both. Replay explains the corpse. Tests prevent the next one.

A Practical Debugging Workflow

When an agent run fails, do this:

Open recordings/latest.json
Find the last agent.decision
Open the Playwright trace
Jump to the matching browser action
Inspect the before snapshot, after snapshot, console, and network tab
Check whether the selector matched the intended element
Replay the recorded action list without the model
If replay passes, debug the model decision layer
If replay fails, debug browser state, selectors, timing, or network drift

This gives you a clean fork in the road.

Browser problem? Fix selectors, waits, context, HAR, storage, or page handling.

Agent problem? Fix observation extraction, prompt constraints, tool schema, action validation, or retry policy.

Optional: Use Hosted Session Replay

If you run agents in hosted browsers, use the provider’s session tooling instead of reinventing everything. Browserbase, for example, provides a Session Inspector for watching live sessions, viewing recordings, and inspecting logs.

Hosted session replay is especially useful when:

Failures happen only in cloud infrastructure
You need to share a session with teammates
Local browser behavior differs from production
You run many parallel agents
You need recordings for customer support or audits

Still keep your own structured agent log. Hosted browser recordings show what happened in the browser. Your agent log explains why the agent did it.

What Good Looks Like

A mature deterministic replay setup produces a failure bundle like this:

failure-2026-08-03T10-05-21/
  agent-events.json
  actions.json
  trace.zip
  network.har
  storage-state.json
  screenshots/
    step-001.png
    step-002.png
    failure.png
  metadata.json

And metadata.json should include:

{
  "runId": "run_2026_08_03_100521",
  "goal": "Compare plan pricing and summarize limits",
  "browser": "chromium",
  "viewport": { "width": 1280, "height": 800 },
  "locale": "en-US",
  "timezone": "America/New_York",
  "agentVersion": "0.7.3",
  "promptVersion": "pricing-research-v4",
  "modelSettings": {
    "temperature": 0,
    "seed": 12345
  }
}

Expected result: another person can replay, inspect, and reason about the failure without asking you what happened.

That is the bar. Not “we have logs somewhere.” Not “the agent usually works.” A real bundle with enough state to reproduce the bug.

Final Takeaway

AI browser agents are only impressive until they fail invisibly.

Deterministic browser replay gives you the missing debugging loop: record the browser, record the agent’s decisions, freeze the starting state, capture the network when needed, and replay actions without asking the model to improvise again.

Start small. Add Playwright tracing today. Wrap clicks and fills tomorrow. Save model decisions after that. Within a few runs, your agent failures will stop looking mystical and start looking like ordinary software bugs.

Ordinary software bugs are annoying. But at least you can fix them.