LLM Rate Limits and Backpressure: Keep Production AI Responsive Under Load

Your LLM provider’s rate limit is rarely the real outage. The outage begins when your application pretends capacity is infinite.

An effective LLM rate-limit backpressure design decides how much work may enter, where excess work waits, which requests should be rejected, and how recovery happens. Without those decisions, retries multiply traffic, queues grow invisibly, and users wait behind jobs that no longer matter.

Start With a Capacity Model

Providers may limit requests per minute, tokens per minute, concurrent connections, or a combination. Your own budget and latency target create additional limits. Translate them into a small set of runtime numbers:

• maximum active requests per model and tenant;
• maximum queued requests and queue age;
• token budget per rolling window;
• deadline for each request class;
• retry budget per original operation.

Do not use average traffic. A system that averages ten requests per second can still receive one hundred at once. Model input size also matters: one document-analysis job may consume the token capacity of dozens of short chats.

Put Admission Control Before the Provider

Admission control is the front door. It should reject or defer work before expensive preprocessing and provider calls begin.

type Job = {
  tenantId: string;
  estimatedTokens: number;
  deadline: number;
  priority: "interactive" | "batch";
};

function admit(job: Job, state: CapacityState) {
  if (Date.now() > job.deadline) return { ok: false, reason: "expired" };
  if (!state.tenantBucket(job.tenantId).take(job.estimatedTokens)) {
    return { ok: false, reason: "tenant_quota" };
  }
  if (state.queueDepth(job.priority) >= state.maxQueue(job.priority)) {
    return { ok: false, reason: "overloaded" };
  }
  return { ok: true };
}

A bounded queue is essential. An unbounded queue converts a visible overload into a later latency and memory failure. Interactive traffic should usually have a shorter queue and deadline than background summarization.

Control Concurrency and Token Flow

A semaphore limits active requests, while a token bucket controls consumption over time. You often need both. Concurrency alone does not protect a tokens-per-minute limit when prompts vary widely.

Estimate tokens before admission, then reconcile with actual usage after the response. Estimates need not be perfect; they need to prevent gross overcommitment. Reserve capacity for interactive work so a batch import cannot occupy every slot.

Fairness belongs here too. Per-tenant buckets prevent one customer from consuming the shared allowance. Weighted queues can give paid or latency-sensitive traffic more capacity without starving lower-priority jobs.

Retry Less, and Retry Better

A 429 response may include a retry hint. Respect it when present. Otherwise use exponential backoff with jitter, but only for operations that remain valuable and safe to repeat.

async function withRetry<T>(fn: () => Promise<T>, deadline: number) {
  for (let attempt = 0; attempt < 4; attempt++) {
    if (Date.now() >= deadline) throw new Error("deadline_exceeded");
    try {
      return await fn();
    } catch (error) {
      if (!isRetryable(error) || attempt === 3) throw error;
      const cap = Math.min(8_000, 250 * 2 ** attempt);
      const wait = Math.random() * cap;
      if (Date.now() + wait >= deadline) throw error;
      await delay(wait);
    }
  }
  throw new Error("unreachable");
}

Never let every service layer retry independently. If the HTTP client, worker, and orchestration layer each retry three times, one user action can become dozens of calls. Assign retry ownership to one layer and carry an idempotency key for side-effecting workflows.

Degrade Gracefully

Backpressure is a product decision, not only an infrastructure mechanism. When capacity is tight, the system can:

• return a clear “try again” response for nonessential generation;
• offer asynchronous completion with a job ID;
• route eligible work to a smaller model;
• shorten optional context or disable expensive enrichment;
• pause batch traffic while preserving interactive capacity.

Each fallback needs evaluation. A smaller model is not a safe fallback if it violates the task’s accuracy or formatting requirements. Silent quality degradation is just another outage.

Instrument the Queue, Not Just the API

Provider error rate is a lagging signal. Track queue depth, oldest-job age, admission rejections, estimated versus actual tokens, active concurrency, retry amplification, and completion latency by tenant and priority.

Alert on queue age before it exceeds the user deadline. A queue of one hundred short jobs may be healthy; a queue of ten document jobs may not be. Capacity dashboards should show work units that predict service time, not only item counts.

Test the Overload Path

Load tests should include variable prompt sizes, provider latency spikes, partial 429 responses, and a complete provider pause. Verify that queues remain bounded, expired jobs disappear, high-priority traffic retains capacity, and recovery does not release a retry storm.

The success condition is not “no requests rejected.” A reliable overloaded system rejects some work quickly and predictably so the rest can succeed.

The Takeaway

Backpressure makes finite LLM capacity explicit. Bound the queue, control tokens and concurrency, isolate tenants, give retries a budget, and define user-visible degradation before traffic spikes. Your system will still encounter limits—but it will remain understandable and responsive when it does.