LLM Request Coalescing: Stop Paying Twice for the Same Answer

A product launch sends 400 users to the same report. Every browser asks the model to summarize the same public document with the same prompt. The application checks its cache, finds nothing, and launches 400 expensive calls before the first result can be stored.

Request coalescing fixes this cold-cache stampede. One caller becomes the leader, identical callers wait on the same in-flight operation, and all receive the result when it completes. The pattern is simple; defining “identical” safely is not.

Build a key that represents the complete request

A coalescing key must include every input that can change the output: provider, model version, normalized messages, system prompt, tool definitions, response schema, sampling settings, retrieval snapshot, and application prompt version.

Do not include only the visible user text. Two tenants may ask the same question against different private documents. Two deployments may use different safety policies. If either distinction is omitted, coalescing becomes a data-isolation bug.

Hash a canonical serialization rather than concatenating arbitrary strings. Sort object keys, preserve array order, normalize only fields whose whitespace is semantically irrelevant, and version the key format. Store the hash in logs, not raw private prompts.

Apply a hard tenant or authorization boundary before hashing. Public, immutable workloads can sometimes share globally; private workloads should usually coalesce only inside the same access scope.

Coordinate one leader and many followers

Maintain an in-memory map from key to an in-flight promise. The first caller creates the promise and starts the upstream request. Later callers attach to it. Remove the entry in a finally block so success, failure, timeout, and cancellation cannot leave a poisoned permanent lock.

For a single process, this can be a small utility. Across many replicas, use a distributed coordinator only when duplicate-call cost justifies the added failure modes. A short lease in Redis can elect a leader, but followers still need a result channel, timeout, and recovery path if the leader dies.

Keep the coalescing window bounded. A request waiting for an unrelated ten-minute generation is not a cache optimization; it is head-of-line blocking. Separate service classes and cap follower count when large fan-outs could exhaust memory.

Treat streaming and cancellation as product decisions

Streaming complicates fan-out. Late followers have missed earlier tokens, and slow clients can backpressure the leader. The safest first version coalesces only non-streaming jobs or buffers a bounded response before returning it.

If streaming is required, publish chunks through a replayable buffer with explicit memory limits. New followers receive buffered chunks and then join the live stream. Disconnecting one follower must not cancel the shared upstream call while others remain; cancel only when the leader has no interested consumers.

Do not blindly share errors. A deterministic schema rejection may be returned to all followers. A transient timeout may deserve one carefully bounded retry by the leader. If every follower retries independently after a shared failure, the stampede simply moves a few seconds later.

Measure saved work and hidden waiting

Track unique keys, leader calls, follower joins, coalescing ratio, upstream tokens avoided, follower wait time, leader failures, and fallbacks that started a second call. Break metrics down by endpoint and tenant without logging sensitive content.

Load-test synchronized bursts, leader cancellation, process crashes, slow streams, and key-version deployments. Verify that private requests never cross scope and that a stuck leader releases followers by deadline.

Request coalescing works best beside an ordinary result cache. The cache serves completed work; single-flight coordination protects the interval before completion. Together they turn duplicate demand into one paid generation instead of a synchronized bill.