Streaming LLM Responses With FastAPI and Server-Sent Events
Token-by-token streaming makes LLM apps feel alive. Here's a minimal, production-ready setup using FastAPI, SSE and a tiny React hook — no websockets, no headaches.
The single biggest perceived-performance upgrade you can give an LLM product is streaming. A response that starts appearing in 200ms feels faster than one that arrives complete in 2 seconds, even though the total time is the same. Users don't measure latency, they measure 'time to first word'.
I default to Server-Sent Events instead of WebSockets for this. SSE is one-way, which is exactly what LLM responses are; it works through almost every proxy without special configuration; and it reconnects automatically. WebSockets are more powerful, but for a 'send prompt, stream tokens back' flow they're overkill.
On the FastAPI side, the whole implementation is a StreamingResponse wrapped around an async generator that yields one SSE frame per chunk from the model SDK: a data line carrying the token, then a blank line to close the frame. Add a heartbeat comment every 15 seconds and most load balancers will stop killing your idle-looking connections.
On the React side, a tiny hook around EventSource is enough. The trickiest part isn't the transport — it's deciding when a stream is 'done', handling cancellation when the user navigates away, and rendering partial markdown without flickering. I batch token updates in a ref and flush on requestAnimationFrame to keep things smooth.
Streaming isn't free. Errors mid-stream are awkward, retries are harder, and observability needs more thought (a 200 response that fails halfway through is still a failure). But for any chat-shaped product, the UX win is so large that I treat streaming as the baseline, not an enhancement.
