Adaptive Concurrency for LLM APIs: Control Backpressure, Latency, and Rate Limits

A fixed pool of 20 LLM workers looks sensible until responses slow from two seconds to twenty, token limits bite, retries multiply, and every worker begins generating more work than it finishes. Adaptive concurrency for LLM APIs solves the problem by changing how many requests may run based on observed capacity.

The goal is not maximum parallelism. It is stable useful throughput with bounded latency and a queue that can recover.

Start With the Right Signals

Track requests and tokens separately because providers often limit both. At minimum, collect in-flight requests, queue age, p50 and p95 latency, success rate, 429 responses, timeouts, input tokens, output tokens, and retry count.

Use end-to-end latency for user experience and provider latency for control decisions. A slow database should not trick your model limiter into reducing capacity, and a slow model should not be hidden inside a healthy overall average.

Expected result: one dashboard can answer whether overload comes from provider limits, long generations, local dependencies, or retry storms.

Build a Simple AIMD Controller

Additive increase, multiplicative decrease is a strong starting point. Increase concurrency slowly while requests are healthy. Cut it quickly when overload signals appear.

let limit = 4;

function onWindow({ p95, errors429, queueAge }) {
  const overloaded = errors429 > 0 || p95 > 12000 || queueAge > 30000;
  if (overloaded) limit = Math.max(1, Math.floor(limit * 0.7));
  else limit = Math.min(64, limit + 1);
}

Run the controller on a window such as 10 to 30 seconds, not after every request. Per-request changes create oscillation. Add a cooldown after a decrease and use exponential backoff with jitter for 429s.

Expected result: concurrency rises during healthy periods and falls before the queue becomes unrecoverable.

Add Token-Aware Admission

Two requests are not equal. A 200-token classification and a 20,000-token report should not consume the same budget. Estimate input size and reserve an output allowance before admitting work.

Maintain both a request semaphore and a rolling token budget. Reject or defer work when either is exhausted. Route unusually large jobs to a separate queue so they cannot block interactive traffic.

interactive queue -> short deadline -> reserved capacity
batch queue       -> long deadline  -> leftover capacity
large-context     -> separate token budget

Expected result: batch summaries cannot starve customer-facing chats.

Apply Backpressure Before Retries

When the queue exceeds a maximum age, stop accepting low-priority work. Return a clear retryable status, degrade to a smaller model, shorten context, or schedule the task. Silent unbounded queues only convert overload into delayed failure.

Use idempotency keys so a timeout does not create duplicate generations or actions. Cap retry attempts and share a retry budget across the service. A provider incident should produce less traffic, not an exponential attack against your own account.

Test the Controller

Replay workloads with variable prompt sizes and injected latency. Force 429 bursts, slow responses, and provider recovery. Watch for oscillation, starvation, and synchronized retries. Compare completed useful requests—not raw attempts—against the fixed-concurrency baseline.

Common failures include increasing the limit too aggressively, averaging away tail latency, ignoring token limits, and using one pool for every priority. Begin conservatively; capacity learned from production can move upward.

The Takeaway

LLM capacity changes by model, prompt, output length, and provider conditions. Adaptive concurrency treats that variability as normal. Measure in-flight work and tokens, increase slowly, decrease decisively, separate priorities, and shed load before the system lies to users with an endlessly growing queue.