LLM Request Hedging and Failover: Building Reliable Multi-Provider Inference

Your AI app is only as reliable as the slowest model call you refuse to plan for.

LLM request hedging failover sounds like infrastructure nerdery until your shiny chatbot hangs for 28 seconds, your users mash refresh, and your provider starts throwing 429s like confetti. Then it becomes the difference between “minor latency spike” and “support inbox on fire.”

This tutorial shows how to build a practical multi-provider inference layer: timeouts, retries, hedged requests, fallback providers, circuit breakers, and observability. Not theory. The stuff you actually need before sending real traffic through paid model APIs.

What You Are Building

You are going to build a small TypeScript inference router that can:

• Send a request to a primary LLM provider.
• Hedge the request to a backup provider if the primary is slow.
• Fail over when a provider times out, rate limits, or returns a server error.
• Avoid retry storms with budgets and jitter.
• Track which provider won, failed, or got cancelled.
• Return one clean response to the application.

The pattern works whether your providers are OpenAI, Anthropic, Google, Azure-hosted models, self-hosted vLLM, or an internal gateway. The code below uses generic provider adapters so you can plug in whatever stack you already use.

Prerequisites

You should have:

• Node.js 20 or newer.
• Basic TypeScript knowledge.
• API keys for at least two LLM providers, or one real provider plus one mock provider.
• A server-side environment. Do not call paid LLM APIs directly from browser code unless you enjoy leaking keys.
• A rough latency target, such as “95% of requests should finish under 4 seconds.”

Install the tiny dependency set:

npm init -y
npm install typescript tsx zod
npm install -D @types/node
npx tsc --init

Expected result: you have a local TypeScript project that can run files with npx tsx.

The Reliability Model

Before writing code, get the mental model right.

There are four common strategies people mix up:

• Timeouts: stop waiting after a deadline.
• Retries: try the same provider again after a transient failure.
• Failover: switch to another provider after failure.
• Hedging: start a backup request before the first one fails, because “slow” is often “about to be useless.”

Hedging is not the same as blasting every request to three vendors. That is expensive, noisy, and hostile to provider rate limits. A good hedge waits for a short delay, usually around your normal p95 latency, then sends a duplicate to a fallback only if the first request is still pending.

The first valid response wins. The loser gets cancelled where possible.

Step 1: Define A Provider Interface

Create src/types.ts:

export type ChatMessage = {
  role: "system" | "user" | "assistant";
  content: string;
};

export type InferenceRequest = {
  messages: ChatMessage[];
  maxTokens?: number;
  temperature?: number;
  idempotencyKey: string;
};

export type InferenceResponse = {
  text: string;
  provider: string;
  model: string;
  latencyMs: number;
};

export type ProviderErrorCode =
  | "timeout"
  | "rate_limit"
  | "server_error"
  | "auth_error"
  | "bad_request"
  | "cancelled"
  | "unknown";

export class ProviderError extends Error {
  constructor(
    public provider: string,
    public code: ProviderErrorCode,
    message: string,
    public retryAfterMs?: number
  ) {
    super(message);
  }
}

export type LlmProvider = {
  name: string;
  model: string;
  timeoutMs: number;
  generate(
    request: InferenceRequest,
    signal: AbortSignal
  ): Promise<InferenceResponse>;
};

Expected result: every provider adapter has the same shape. Your app no longer cares which vendor SDK is underneath.

The idempotencyKey matters. If you retry or hedge, you need a way to correlate duplicate attempts. For pure text generation, duplicate billing may still happen, but your logs and downstream systems should understand that these attempts belong to one user request.

Step 2: Add Timeout Wrapping

Create src/timeout.ts:

import { ProviderError } from "./types";

export async function withTimeout<T>(
  provider: string,
  timeoutMs: number,
  fn: (signal: AbortSignal) => Promise<T>
): Promise<T> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);

  try {
    return await fn(controller.signal);
  } catch (error) {
    if (controller.signal.aborted) {
      throw new ProviderError(provider, "timeout", `${provider} timed out`);
    }
    throw error;
  } finally {
    clearTimeout(timer);
  }
}

Expected result: a provider call cannot hold your application hostage forever.

Pick timeouts based on user experience, not vendor optimism. If your app is interactive, a 60-second timeout is usually fake reliability. The user has already left, refreshed, or sworn at the screen.

Step 3: Classify Provider Errors

Create src/errors.ts:

import { ProviderError } from "./types";

export function normalizeProviderError(
  provider: string,
  error: unknown
): ProviderError {
  if (error instanceof ProviderError) return error;

  const anyError = error as {
    status?: number;
    code?: string;
    message?: string;
    headers?: Record<string, string>;
  };

  const status = anyError.status;
  const message = anyError.message ?? "Unknown provider error";
  const retryAfter = anyError.headers?.["retry-after"];
  const retryAfterMs = retryAfter ? Number(retryAfter) * 1000 : undefined;

  if (status === 400) return new ProviderError(provider, "bad_request", message);
  if (status === 401 || status === 403) {
    return new ProviderError(provider, "auth_error", message);
  }
  if (status === 429) {
    return new ProviderError(provider, "rate_limit", message, retryAfterMs);
  }
  if (status && status >= 500) {
    return new ProviderError(provider, "server_error", message);
  }

  if (anyError.code === "ABORT_ERR") {
    return new ProviderError(provider, "cancelled", message);
  }

  return new ProviderError(provider, "unknown", message);
}

export function isRetriable(error: ProviderError): boolean {
  return ["timeout", "rate_limit", "server_error", "unknown"].includes(error.code);
}

Expected result: your router can tell the difference between “try somewhere else” and “your request is broken.”

Never retry bad requests. If the prompt is malformed, the schema is impossible, or the auth key is dead, retries just burn money with extra steps.

Step 4: Build A Retry Helper With Jitter

Create src/retry.ts:

import { ProviderError } from "./types";
import { isRetriable } from "./errors";

type RetryOptions = {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
};

function sleep(ms: number, signal: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(new Error("Aborted"));
    });
  });
}

function jitteredDelay(base: number, attempt: number, max: number): number {
  const exponential = Math.min(max, base * 2 ** attempt);
  return Math.floor(Math.random() * exponential);
}

export async function retryWithBackoff<T>(
  operation: (signal: AbortSignal) => Promise<T>,
  signal: AbortSignal,
  options: RetryOptions
): Promise<T> {
  let lastError: ProviderError | undefined;

  for (let attempt = 0; attempt < options.maxAttempts; attempt++) {
    try {
      return await operation(signal);
    } catch (error) {
      const providerError = error as ProviderError;
      lastError = providerError;

      if (!isRetriable(providerError) || attempt === options.maxAttempts - 1) {
        throw providerError;
      }

      const retryAfter = providerError.retryAfterMs;
      const delay =
        retryAfter ?? jitteredDelay(options.baseDelayMs, attempt, options.maxDelayMs);

      await sleep(delay, signal);
    }
  }

  throw lastError;
}

Expected result: transient failures get another chance, but they do not synchronize into a retry stampede.

This is where many LLM apps get dumb. They retry instantly, from every server, at the same time. When the provider is already under stress, that behavior turns your application into part of the outage.

Step 5: Add Provider Adapters

Here is a mock provider so you can test the router without paying for tokens.

Create src/mockProvider.ts:

import {
  InferenceRequest,
  InferenceResponse,
  LlmProvider,
  ProviderError
} from "./types";

type MockProviderOptions = {
  name: string;
  model: string;
  timeoutMs: number;
  latencyMs: number;
  failureRate?: number;
};

export function createMockProvider(options: MockProviderOptions): LlmProvider {
  return {
    name: options.name,
    model: options.model,
    timeoutMs: options.timeoutMs,
    async generate(
      request: InferenceRequest,
      signal: AbortSignal
    ): Promise<InferenceResponse> {
      const startedAt = Date.now();

      await new Promise<void>((resolve, reject) => {
        const timer = setTimeout(resolve, options.latencyMs);
        signal.addEventListener("abort", () => {
          clearTimeout(timer);
          reject(new ProviderError(options.name, "cancelled", "Request cancelled"));
        });
      });

      if (Math.random() < (options.failureRate ?? 0)) {
        throw new ProviderError(
          options.name,
          "server_error",
          `${options.name} simulated failure`
        );
      }

      return {
        text: `Response from ${options.name}: ${request.messages.at(-1)?.content}`,
        provider: options.name,
        model: options.model,
        latencyMs: Date.now() - startedAt
      };
    }
  };
}

Expected result: you can simulate slow providers, broken providers, and fast backups.

A real OpenAI adapter would implement the same generate() function using the official SDK. Keep SDK-specific weirdness inside the adapter. Your router should stay boring.

Step 6: Implement Failover

Create src/failover.ts:

import { normalizeProviderError } from "./errors";
import { retryWithBackoff } from "./retry";
import {
  InferenceRequest,
  InferenceResponse,
  LlmProvider,
  ProviderError
} from "./types";
import { withTimeout } from "./timeout";

export async function failoverGenerate(
  providers: LlmProvider[],
  request: InferenceRequest,
  signal: AbortSignal
): Promise<InferenceResponse> {
  const errors: ProviderError[] = [];

  for (const provider of providers) {
    try {
      return await retryWithBackoff(
        (attemptSignal) =>
          withTimeout(provider.name, provider.timeoutMs, (timeoutSignal) => {
            const combined = AbortSignal.any([signal, attemptSignal, timeoutSignal]);
            return provider.generate(request, combined);
          }),
        signal,
        {
          maxAttempts: 2,
          baseDelayMs: 250,
          maxDelayMs: 1500
        }
      );
    } catch (error) {
      errors.push(normalizeProviderError(provider.name, error));
    }
  }

  throw new AggregateError(errors, "All LLM providers failed");
}

Expected result: if provider A fails, provider B gets a shot.

This is enough for many back-office jobs, summarizers, batch enrichment tasks, and non-interactive workflows. But for user-facing products, failover often starts too late. Waiting for provider A to fully timeout before trying provider B can still create ugly tail latency.

That is where hedging comes in.

Step 7: Implement Hedged Requests

Create src/hedged.ts:

import { normalizeProviderError } from "./errors";
import {
  InferenceRequest,
  InferenceResponse,
  LlmProvider,
  ProviderError
} from "./types";
import { withTimeout } from "./timeout";

type HedgingOptions = {
  hedgeDelayMs: number;
};

function delay(ms: number, signal: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(new Error("Aborted"));
    });
  });
}

export async function hedgedGenerate(
  primary: LlmProvider,
  backup: LlmProvider,
  request: InferenceRequest,
  parentSignal: AbortSignal,
  options: HedgingOptions
): Promise<InferenceResponse> {
  const primaryController = new AbortController();
  const backupController = new AbortController();
  const errors: ProviderError[] = [];

  const primarySignal = AbortSignal.any([
    parentSignal,
    primaryController.signal
  ]);

  const primaryPromise = withTimeout(
    primary.name,
    primary.timeoutMs,
    (timeoutSignal) =>
      primary.generate(request, AbortSignal.any([primarySignal, timeoutSignal]))
  ).catch((error) => {
    const normalized = normalizeProviderError(primary.name, error);
    errors.push(normalized);
    throw normalized;
  });

  const backupPromise = delay(options.hedgeDelayMs, parentSignal)
    .then(() => {
      const backupSignal = AbortSignal.any([
        parentSignal,
        backupController.signal
      ]);

      return withTimeout(backup.name, backup.timeoutMs, (timeoutSignal) =>
        backup.generate(request, AbortSignal.any([backupSignal, timeoutSignal]))
      );
    })
    .catch((error) => {
      const normalized = normalizeProviderError(backup.name, error);
      errors.push(normalized);
      throw normalized;
    });

  try {
    const winner = await Promise.any([primaryPromise, backupPromise]);

    if (winner.provider === primary.name) {
      backupController.abort();
    } else {
      primaryController.abort();
    }

    return winner;
  } catch {
    throw new AggregateError(errors, "Hedged LLM request failed");
  }
}

Expected result: the backup request starts only if the primary has not answered after hedgeDelayMs. The fastest valid response wins.

Use hedging sparingly. It can double your cost for hedged requests, increase provider load, and make rate limits arrive faster. The trick is to hedge only the tail, not the median.

A decent starting point:

• Primary timeout: 6 seconds.
• Backup timeout: 6 seconds.
• Hedge delay: your primary provider’s p95 latency, often 2 to 4 seconds for interactive generation.
• Hedging percentage: 5% to 20% of traffic until you have data.

Step 8: Add A Router Policy

Now combine simple failover and hedging into one router.

Create src/router.ts:

import { failoverGenerate } from "./failover";
import { hedgedGenerate } from "./hedged";
import { InferenceRequest, InferenceResponse, LlmProvider } from "./types";

type RouterOptions = {
  hedgeEnabled: boolean;
  hedgeDelayMs: number;
};

export async function routeInference(
  providers: LlmProvider[],
  request: InferenceRequest,
  options: RouterOptions
): Promise<InferenceResponse> {
  if (providers.length === 0) {
    throw new Error("At least one provider is required");
  }

  const controller = new AbortController();

  if (options.hedgeEnabled && providers.length >= 2) {
    return hedgedGenerate(
      providers[0],
      providers[1],
      request,
      controller.signal,
      { hedgeDelayMs: options.hedgeDelayMs }
    );
  }

  return failoverGenerate(providers, request, controller.signal);
}

Expected result: your application has one entry point for inference. Underneath, policy decides whether to hedge or fail over.

This is the layer where you can later add cost controls, tenant-specific routing, model quality tiers, safety policies, or regional preferences.

Step 9: Run The Demo

Create src/demo.ts:

import { createMockProvider } from "./mockProvider";
import { routeInference } from "./router";

const providers = [
  createMockProvider({
    name: "primary",
    model: "fast-expensive-model",
    timeoutMs: 5000,
    latencyMs: 3500,
    failureRate: 0.1
  }),
  createMockProvider({
    name: "backup",
    model: "steady-backup-model",
    timeoutMs: 5000,
    latencyMs: 900,
    failureRate: 0.02
  })
];

const response = await routeInference(
  providers,
  {
    idempotencyKey: crypto.randomUUID(),
    messages: [
      {
        role: "user",
        content: "Explain request hedging in one sentence."
      }
    ],
    maxTokens: 120
  },
  {
    hedgeEnabled: true,
    hedgeDelayMs: 1200
  }
);

console.log(response);

Run it:

npx tsx src/demo.ts

Expected result: the backup often wins because the primary is intentionally slow. If you increase hedgeDelayMs to 4000, the primary usually wins. That is the knob.

Step 10: Add The Missing Production Pieces

The tutorial code is intentionally small. Production needs a few more guardrails.

Circuit Breakers

If a provider is failing hard, stop sending it traffic for a short cooldown window.

Track recent outcomes per provider:

• Error rate.
• Timeout rate.
• 429 rate.
• p95 and p99 latency.
• Consecutive failures.
• Last successful request time.

A simple rule works:

type ProviderHealth = {
  openUntilMs?: number;
  consecutiveFailures: number;
};

export function shouldSkipProvider(health: ProviderHealth): boolean {
  return Boolean(health.openUntilMs && Date.now() < health.openUntilMs);
}

export function recordFailure(health: ProviderHealth): ProviderHealth {
  const consecutiveFailures = health.consecutiveFailures + 1;

  return {
    consecutiveFailures,
    openUntilMs:
      consecutiveFailures >= 5 ? Date.now() + 30_000 : health.openUntilMs
  };
}

export function recordSuccess(): ProviderHealth {
  return { consecutiveFailures: 0 };
}

Expected result: a provider having a bad minute does not keep poisoning your users’ requests.

Retry Budgets

Do not allow unlimited retries globally. Cap them per process, tenant, or route.

Example policy:

• Maximum 2 attempts per provider.
• Maximum 1 hedge per user request.
• Maximum 10% extra provider calls from retries and hedges combined.
• No retries for validation errors, auth errors, policy blocks, or context length failures.

This matters because retries can amplify outages. Google’s SRE guidance is blunt about this: retry behavior can turn a capacity problem into a cascading failure. LLM APIs are no exception.

Fallback Quality Rules

Failover is not just “provider A broke, call provider B.”

You need compatibility rules:

• Can the backup model follow the same JSON schema?
• Does it support tool calls?
• Does it support your context length?
• Does it support image, audio, or file inputs?
• Does its safety behavior change the user-visible result?
• Is the output quality acceptable for this route?

For example, fallback from a strong reasoning model to a cheap summarizer model may be fine for title generation. It is not fine for legal analysis, medical triage, fraud review, or code changes in production infrastructure.

Your router should know the route intent:

type InferenceRoute = "chat" | "summarize" | "extract_json" | "classify" | "code";

type RoutePolicy = {
  route: InferenceRoute;
  allowHedging: boolean;
  allowedProviders: string[];
  requireJsonSchema: boolean;
};

Expected result: fallback does not silently degrade the product into nonsense.

Common Pitfalls

Pitfall 1: Hedging Every Request

Hedging every request is usually wasteful. You pay more, consume more rate limit, and add load when the ecosystem is already stressed.

Better: hedge only after a delay and only for latency-sensitive routes.

Pitfall 2: Waiting Too Long To Hedge

If your primary timeout is 30 seconds and your hedge delay is 25 seconds, congratulations, you built failover with extra billing.

Better: set hedge delay around the p95 latency of successful primary requests.

Pitfall 3: Retrying 429s Instantly

A 429 means slow down. Some APIs include Retry-After headers. Respect them when present.

Better: use exponential backoff with jitter, cap attempts, and track retry budgets.

Pitfall 4: Ignoring Streaming

Streaming changes the game. Once tokens are flowing to the user, switching providers mid-answer is messy.

Common approach:

• Hedge only before the first token.
• Once streaming starts, commit to that provider.
• If streaming fails midway, show a recoverable error or restart with explicit UX.

Do not splice two model outputs together and pretend nothing happened. Users notice. Logs notice. Your evals will definitely notice.

Pitfall 5: Treating Providers As Equivalent

Models are not interchangeable sockets. They differ in formatting discipline, refusal behavior, tool-call syntax, context limits, latency, and cost.

Better: run provider-specific evals before enabling failover for important workflows.

What To Measure

You cannot tune hedging by vibes. Track these metrics:

• End-to-end latency by route.
• Provider latency by attempt number.
• Hedge start rate.
• Hedge win rate.
• Cancelled request count.
• Retry count by error code.
• Provider error rate.
• 429 rate.
• Cost per successful response.
• User-visible failure rate.
• JSON/schema validity rate.

The key metric is not “did the backup ever win?” The key metric is whether hedging improves tail latency enough to justify the extra calls.

If backup wins 1% of the time and doubles your API bill on 40% of requests, your hedge policy is lazy. Tighten the delay, reduce the eligible routes, or disable it.

A Practical Default Policy

Start conservative:

• Use primary-only routing for non-interactive jobs.
• Use failover for interactive routes where a degraded but correct answer is acceptable.
• Use hedging only for high-value, latency-sensitive routes.
• Set provider timeouts below your user-facing HTTP timeout.
• Retry at most once per provider.
• Use jitter on every retry delay.
• Respect Retry-After.
• Add circuit breakers before serious traffic.
• Keep provider adapters isolated.
• Log every attempt under one request ID.

A clean first production policy might look like this:

const policy = {
  chat: {
    hedgeEnabled: true,
    hedgeDelayMs: 2500,
    maxAttemptsPerProvider: 2
  },
  summarize: {
    hedgeEnabled: false,
    maxAttemptsPerProvider: 2
  },
  extract_json: {
    hedgeEnabled: false,
    maxAttemptsPerProvider: 1
  }
};

Expected result: your most visible workflows get latency protection, while cheaper background tasks avoid unnecessary duplicate calls.

Security Notes

A multi-provider router becomes sensitive infrastructure. Treat it that way.

• Store API keys server-side only.
• Use separate keys per provider and environment.
• Avoid logging raw prompts when they may contain private data.
• Redact secrets from error messages.
• Add per-tenant quotas so one customer cannot drain all provider capacity.
• Validate model output before passing it into tools, databases, or browsers.
• Escape rendered model output to prevent XSS.

Failover does not fix unsafe output handling. If your app blindly trusts model text, a more reliable router just delivers bad behavior faster.

Final Takeaway

LLM reliability is not about finding the one magical provider that never flakes. That provider does not exist.

Build the boring layer: deadlines, retries with jitter, failover, selective hedging, circuit breakers, and metrics. Then make every app call the layer instead of calling model APIs directly.

Start with failover. Add hedging only where tail latency hurts. Measure the cost. Kill the clever bits that do not earn their keep.