Streaming LLM Responses Without Corrupting Partial JSON

The UI receives a partial object ending after the amount field and the connection closes. A helpful client appends a brace, parses the fragment, and displays a successful approval.

It repaired syntax by inventing meaning.

Streaming model output improves perceived latency, but it introduces a state most APIs try to hide: data that exists without being complete. Reliable systems preserve that distinction from provider event to user interface.

Treat the Stream as Events, Not a Growing Answer

Consume the provider’s typed event protocol. Text deltas, tool-argument deltas, usage, refusals, errors, and completion signals should enter separate handlers. Do not concatenate every payload field into one universal string.

Maintain an operation state machine such as OPEN, COMPLETED, FAILED, CANCELLED, and UNKNOWN. Only an explicit terminal event can move a response to COMPLETED. A socket close, timeout, or browser disconnect is transport information, not proof that generation finished.

Persist the provider response ID and your own request ID before exposing the stream. Every event should carry a monotonically increasing local sequence number so duplicate delivery and missing gaps are detectable.

Send heartbeats separately from content. An idle interval may mean the model is working, the network is buffering, or an intermediary is about to time out. None of those states justifies closing a JSON structure.

Buffer Structured Output Until It Is Valid

For plain prose, the UI can render deltas optimistically. For executable or persisted data, keep an authoritative buffer on the server. Incremental parsing may power a preview, but business logic must wait for terminal completion and full validation.

Tool arguments often arrive as JSON fragments. Append exact bytes in order, then parse once the tool-call item reports completion. Never execute a tool merely because the current prefix happens to form valid JSON; more keys may still arrive.

If the product needs progressive structured updates, stream complete envelopes rather than arbitrary JSON prefixes. JSON Text Sequences, newline-delimited events, or a typed patch protocol can make each unit independently parseable. Give every patch an object version and restrict which paths it may change.

Validate the final object against its schema and deterministic business rules. A closing brace proves grammar, not authorization, totals, citation support, or date logic.

Recover Without Replaying Side Effects

Separate generation recovery from action recovery. If a stream drops before a model finishes, query the provider by response ID when supported. Otherwise mark the generation unknown and retry according to product policy with a new generation attempt linked to the old one.

Do not resend already executed tool calls as part of reconstructing visible text. Tool operations need stable idempotency keys and durable receipts outside the stream. On reconnect, the client should load canonical operation state, not infer it from the last sentence it saw.

For user-facing text, resume delivery from a stored event sequence. If exact continuation is unavailable, replace the partial draft with a clearly identified regenerated answer. Splicing a new completion onto an old prefix can duplicate or contradict content.

Cancellation needs a terminal record too. The user closing a tab may stop rendering without stopping server generation. Decide whether to cancel upstream work, continue for later retrieval, or discard it—and measure the cost of each path.

Test Every Place the Connection Can Break

Cut the connection after headers, halfway through a multibyte character, between tool-argument fields, after the provider commits a tool call, and immediately before the terminal event. Duplicate and reorder events in a test adapter.

Assert that:

• partial structured data never reaches side-effect code
• incomplete UI content is visibly marked
• sequence gaps trigger recovery
• UTF-8 is reconstructed without replacement corruption
• reconnects do not duplicate tool effects
• final persistence requires a terminal event and validation

Track incomplete-stream rate, last event type, buffered bytes, reconnect success, abandoned generations, and tool operations waiting for reconciliation. Segment by provider, model, region, and client version.

Streaming should make the interface feel alive, not make correctness provisional. The design rule is simple: render partial text when useful, but never promote partial state into a completed fact.