Durable AI Agent Workflows: Build Jobs That Survive Crashes, Retries, and Human Waits

Your AI agent will fail. Not because the model is dumb. Because the network flakes, the API times out, the human approver goes to lunch, or your server restarts halfway through a 14-step job.

That is why durable AI agent workflows matter. A serious agent workflow cannot live inside one long request, one background worker, or one heroic while loop. It needs checkpoints. It needs retries. It needs safe side effects. It needs a way to pause for hours, days, or weeks without burning compute or forgetting what happened.

This tutorial shows how to design agent jobs that survive real-world chaos.

What You Are Building

You are going to design a durable research-and-approval agent workflow.

The workflow will:

1. Accept a research request.
2. Generate a research plan.
3. Fetch or summarize source material.
4. Draft an answer.
5. Wait for human approval.
6. Publish or revise the result.
7. Recover cleanly if any step crashes.

The code examples use TypeScript-style pseudocode with Inngest-like durable steps because the syntax is easy to read. The same architecture maps to Temporal workflows, LangGraph graphs with persistence, cloud durable functions, or your own queue-plus-database setup.

The point is not the vendor. The point is the shape of the system.

Prerequisites

You do not need to be a distributed systems wizard. You should understand:

• Basic API calls
• JSON inputs and outputs
• Background jobs
• Database records
• Webhooks or events
• The rough idea of an AI agent calling tools

For a production build, you will also need:

• A database such as Postgres
• A durable workflow engine such as Temporal, Inngest, LangGraph with checkpointing, or a comparable system
• An LLM provider
• An event endpoint for approvals, webhooks, and callbacks
• Logging and error tracking

If your current “agent” runs inside a single API route and waits until everything is done, this tutorial is especially for you. That design is a demo. It is not a workflow.

The Core Rule: Split the Agent Into Checkpoints

A durable workflow is not one giant function. It is a chain of named steps where each step can be saved, retried, skipped, inspected, or resumed.

Bad version:

async function runAgent(input) {
  const plan = await makePlan(input);
  const sources = await fetchSources(plan);
  const draft = await writeDraft(sources);
  const approval = await waitForApproval(draft);
  await publish(draft);
}

This looks clean, but it is fragile. If the process dies after writeDraft, your app may rerun the whole thing. That means duplicate API calls, duplicate database writes, duplicate emails, and possibly double publishing.

Durable version:

export const researchWorkflow = inngest.createFunction(
  {
    id: "durable-research-agent",
    retries: 3
  },
  { event: "research.requested" },
  async ({ event, step }) => {
    const plan = await step.run("create-research-plan", async () => {
      return createResearchPlan(event.data.topic);
    });

    const sources = await step.run("collect-sources", async () => {
      return collectSources(plan);
    });

    const draft = await step.run("draft-response", async () => {
      return draftResponse({ plan, sources });
    });

    const approval = await step.waitForEvent("wait-for-editor-approval", {
      event: "research.approved",
      timeout: "7d",
      match: "data.workflowId"
    });

    if (!approval) {
      await step.run("mark-expired", async () => {
        return markWorkflowExpired(event.data.workflowId);
      });
      return;
    }

    await step.run("publish-approved-draft", async () => {
      return publishDraft(draft, approval.data.editorId);
    });
  }
);

Expected result: if collect-sources succeeds and draft-response fails, the retry resumes from draft-response. It does not pay again for work already completed.

That is the entire game.

Step 1: Create a Workflow Record First

Before the agent does anything expensive, create a workflow record in your database. This gives every run a stable identity.

type WorkflowStatus =
  | "requested"
  | "planning"
  | "collecting_sources"
  | "drafting"
  | "waiting_for_approval"
  | "published"
  | "failed"
  | "expired";

async function createWorkflow(input: {
  topic: string;
  requestedBy: string;
}) {
  return db.workflow.create({
    data: {
      id: crypto.randomUUID(),
      topic: input.topic,
      requestedBy: input.requestedBy,
      status: "requested"
    }
  });
}

Expected result: every agent run has a durable ID before the model starts spending tokens.

Why this matters: workflow IDs become your anchor for retries, approvals, logs, billing, and debugging. Without one, you are chasing ghosts through log files.

Step 2: Make Every Step Idempotent

Idempotency means a step can run more than once without causing duplicate damage.

This is not optional. Retries are useless if retries corrupt your system.

Bad:

await db.article.create({
  data: {
    title: draft.title,
    body: draft.body
  }
});

If the database write succeeds but the response times out, the retry may create a duplicate article.

Better:

await db.article.upsert({
  where: {
    workflowId: workflow.id
  },
  create: {
    workflowId: workflow.id,
    title: draft.title,
    body: draft.body
  },
  update: {
    title: draft.title,
    body: draft.body
  }
});

Expected result: the workflow can retry safely. One workflow produces one article draft.

Use these idempotency patterns:

• Use deterministic IDs, not random IDs inside retried steps.
• Prefer upsert over blind create.
• Store external API request IDs where possible.
• Check whether a side effect already happened before doing it again.
• Put unique constraints on business keys like workflowId, invoiceId, or approvalId.

The database should enforce the rule. Your memory of the rule is not enough.

Step 3: Keep LLM Calls Inside Durable Steps

LLM calls are expensive, slow, and nondeterministic. They belong behind checkpoints.

const plan = await step.run("create-research-plan", async () => {
  const response = await callModel({
    model: "your-model",
    input: [
      {
        role: "system",
        content: "Create a concise research plan with search queries and acceptance criteria."
      },
      {
        role: "user",
        content: event.data.topic
      }
    ]
  });

  return parsePlan(response);
});

Expected result: if later steps fail, the saved plan is reused instead of regenerated.

That matters because a regenerated plan may change the workflow path. The first run might choose three sources. The retry might choose five different sources. Now your “same” workflow is no longer the same workflow.

For durable AI agent workflows, treat model output as state. Once generated, save it.

Step 4: Validate Model Output Before Continuing

Agents love producing almost-valid JSON. Almost-valid JSON is a production tax.

Use schemas at every model boundary.

import { z } from "zod";

const ResearchPlanSchema = z.object({
  objective: z.string(),
  queries: z.array(z.string()).min(1).max(8),
  acceptanceCriteria: z.array(z.string()).min(1)
});

async function parsePlan(responseText: string) {
  const json = JSON.parse(responseText);
  return ResearchPlanSchema.parse(json);
}

Expected result: malformed model output fails at the step boundary and can be retried or routed to repair logic.

A better version adds a repair step:

const plan = await step.run("create-and-validate-plan", async () => {
  const raw = await callModelForPlan(event.data.topic);

  try {
    return ResearchPlanSchema.parse(JSON.parse(raw));
  } catch {
    const repaired = await callModel({
      model: "your-model",
      input: `Repair this into valid JSON matching the research plan schema:\n${raw}`
    });

    return ResearchPlanSchema.parse(JSON.parse(repaired));
  }
});

Do not let bad model output leak into tools, payments, publishing, or user-visible actions. Validate early. Fail loudly.

Step 5: Separate Pure Work From Side Effects

Pure work can be retried freely. Side effects need more discipline.

Pure-ish steps:

• Generate a plan
• Classify a document
• Extract fields from text
• Score candidate answers
• Create a draft

Side-effect steps:

• Charge a card
• Send an email
• Publish a page
• Update a CRM
• Delete a file
• Trigger another workflow

Side effects should be isolated in their own named steps.

const draft = await step.run("generate-draft", async () => {
  return generateDraft(sources);
});

await step.run("save-draft", async () => {
  return db.articleDraft.upsert({
    where: { workflowId },
    create: { workflowId, body: draft.body, status: "pending_review" },
    update: { body: draft.body, status: "pending_review" }
  });
});

Expected result: if saving fails, only the save step retries. If publishing fails, the draft is not regenerated.

This gives you boring logs, and boring logs are a luxury.

Step 6: Add Human Waits Without Blocking a Worker

Human-in-the-loop is where fragile agent demos really collapse.

A human might approve in 30 seconds. Or three days. Or never.

Do not keep a Node process alive while you wait. Do not poll every minute forever. Do not fake it with a sleeping worker.

Use a durable wait.

await step.run("set-status-waiting-for-approval", async () => {
  return db.workflow.update({
    where: { id: workflowId },
    data: { status: "waiting_for_approval" }
  });
});

const approval = await step.waitForEvent("editor-approval", {
  event: "article.approved",
  timeout: "7d",
  match: "data.workflowId"
});

Expected result: the workflow pauses without burning compute. When the approval event arrives, it resumes with the event payload.

Your approval endpoint can be simple:

app.post("/api/approve", async (req, res) => {
  const { workflowId, editorId, decision, notes } = req.body;

  await inngest.send({
    name: decision === "approve" ? "article.approved" : "article.rejected",
    data: {
      workflowId,
      editorId,
      notes
    }
  });

  res.json({ ok: true });
});

This is where durable systems shine. The workflow is not “running” while it waits. It is stored.

Step 7: Handle Rejections as a Real Branch

Do not treat human rejection as an error. It is a valid business path.

const review = await step.waitForEvent("editor-review", {
  event: "article.reviewed",
  timeout: "7d",
  match: "data.workflowId"
});

if (!review) {
  await step.run("expire-review", async () => {
    return setWorkflowStatus(workflowId, "expired");
  });
  return;
}

if (review.data.decision === "reject") {
  const revised = await step.run("revise-draft", async () => {
    return reviseDraft({
      draft,
      notes: review.data.notes
    });
  });

  await step.run("save-revision", async () => {
    return saveDraft(workflowId, revised);
  });

  return;
}

Expected result: rejection creates a revision path instead of poisoning your error metrics.

Errors are for broken systems. Rejections are product behavior.

Step 8: Use Retries Carefully

Retries are not magic. They are a power tool.

Good retry candidates:

• Temporary network failures
• Rate limits with backoff
• Database deadlocks
• LLM JSON formatting failures
• Third-party API timeouts

Bad retry candidates:

• Invalid user input
• Missing permissions
• A deleted account
• A payment method that was declined
• A policy violation

Use non-retriable errors for permanent failures.

class NonRetriableError extends Error {}

await step.run("check-user-permission", async () => {
  const allowed = await canUserPublish(userId);

  if (!allowed) {
    throw new NonRetriableError("User cannot publish articles");
  }

  return true;
});

Expected result: the workflow does not hammer a permanent failure three, five, or ten times.

A simple retry policy:

export const workflow = inngest.createFunction(
  {
    id: "agent-publishing-workflow",
    retries: 3
  },
  { event: "article.requested" },
  async ({ step }) => {
    // Durable steps here
  }
);

For production, tune retries by step type. LLM calls may deserve retries. Payment capture may deserve zero automatic retries unless the payment provider gives you a safe idempotency key.

Step 9: Add Observability That Matches the Workflow

You need to see the workflow as a business process, not just a pile of logs.

Track:

• workflowId
• Current status
• Current step
• Attempt count
• Last error
• Token usage
• External API IDs
• Human approver
• Timestamps for each major transition

Example status update helper:

async function markStep(workflowId: string, stepName: string) {
  await db.workflow.update({
    where: { id: workflowId },
    data: {
      currentStep: stepName,
      updatedAt: new Date()
    }
  });
}

Use it inside durable steps:

await step.run("mark-drafting", async () => {
  return markStep(workflowId, "drafting");
});

Expected result: when a user asks “where is my agent job?”, you can answer without spelunking through infrastructure logs.

Your admin UI should show:

• Requested
• Planning
• Collecting sources
• Drafting
• Waiting for approval
• Publishing
• Done or failed

That is the difference between an agent feature and an agent science fair project.

Step 10: Version Long-Running Workflows

Long-running workflows create a nasty problem: what happens when you deploy new code while old runs are still waiting?

Imagine version one waits for article.approved. Version two waits for article.reviewed. Old workflows may never resume if you casually change event names.

Use explicit versioning.

export const researchWorkflowV1 = inngest.createFunction(
  { id: "research-agent-v1" },
  { event: "research.requested.v1" },
  async ({ event, step }) => {
    // Original behavior
  }
);

export const researchWorkflowV2 = inngest.createFunction(
  { id: "research-agent-v2" },
  { event: "research.requested.v2" },
  async ({ event, step }) => {
    // New behavior
  }
);

Expected result: old runs finish on old rules. New runs use new rules.

This is less elegant than pretending every deploy is harmless. It is also less likely to ruin your week.

Common Pitfalls

Pitfall: One Giant Agent Loop

A loop that thinks, acts, observes, and repeats inside one process is easy to build and painful to operate.

Fix: checkpoint each tool call, model call, and side effect. Store the observation after every tool result.

Pitfall: Retrying Non-Idempotent Writes

Retries can duplicate emails, records, payments, and publishes.

Fix: use deterministic IDs, unique constraints, and provider idempotency keys. Make the database reject duplicates.

Pitfall: Letting the Model Choose Unsafe Actions

Do not let the model directly execute destructive tools.

Fix: separate suggestion from execution. The model proposes. Your code validates. Humans approve risky actions.

Pitfall: No Timeout for Human Approval

A workflow waiting forever becomes operational debt.

Fix: set timeouts. Expire or escalate stale workflows.

Pitfall: Treating Durable Execution as a Queue

A queue moves messages. A durable workflow remembers progress.

Fix: use queues for simple one-shot jobs. Use durable workflows for multi-step jobs with retries, waits, branching, and state.

When to Use Temporal, Inngest, or LangGraph

Temporal is the heavyweight option for mission-critical workflows. It is strong when you need deep control, long-running processes, strict reliability, and a mature workflow engine. The tradeoff is more infrastructure and more concepts.

Inngest is pragmatic for product teams that want durable functions without managing a full worker platform. It is especially friendly for event-driven apps, serverless deployments, and TypeScript-heavy teams.

LangGraph is useful when your agent is naturally a graph of states, tools, memory, and human interventions. Its persistence model helps with checkpoints, human-in-the-loop flows, and replaying agent state.

You can also build a smaller version yourself with Postgres, a jobs table, and a worker. That is fine for simple workflows. Just be honest about the missing pieces: replay, retries, cancellation, timeouts, history, and visibility.

A Production Checklist

Before shipping a durable agent workflow, check this list:

• Every workflow has a stable ID.
• Every expensive model call is checkpointed.
• Every side effect is isolated in its own step.
• Retried writes are idempotent.
• Human waits have timeouts.
• Rejections are modeled as normal branches.
• Permanent failures do not retry forever.
• Workflow status is visible in your database or admin UI.
• Long-running workflows are versioned.
• Logs include workflowId and step names.
• Token usage and external API calls are tracked.
• Publishing, payments, deletes, and emails have approval gates where needed.

If you cannot inspect, resume, or explain the workflow, it is not durable. It is just asynchronous.

Final Takeaway

Durable AI agent workflows are not about making the model smarter. They are about making the job survive contact with production.

Break the agent into named steps. Save state after meaningful work. Retry only what is safe to retry. Pause cleanly for humans. Treat side effects like live ammunition.

That is how you turn an impressive demo into a workflow you can trust.