Token-by-token output makes AI features feel fast, and every proxy, load balancer and gateway between the model and the browser has a timeout that can cut it off. When to use Server-Sent Events, WebSockets or a background job, with current AWS limits: API Gateway REST streaming up to 15 minutes, WebSocket connections up to 2 hours, Lambda streaming up to 200 MB.
An answer that takes 20 seconds feels slow behind a spinner and fast when words appear after half a second. Streaming changes perceived speed, not total time, and it means holding a response open for as long as generation takes.
For a typical chat feature, SSE is the simplest fit: one request, one streamed response. Many teams read a streamed fetch body rather than using EventSource, which cannot send a POST body or custom headers.
Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
The CDN's origin response timeout: CloudFront waits 30 seconds by default, including between bytes.
The load balancer's idle timeout: 60 seconds by default on an Application Load Balancer.
Your server or function timeout.
Proxy buffering, which holds the whole response before sending it and turns streaming back into waiting. Turn it off on streaming routes.
During long pauses, such as while the model is calling tools, send a heartbeat (an SSE comment line works) every 15 to 30 seconds so idle timeouts never fire.
If a task can run for minutes (research agents, long documents, batch generation) or must finish when the user closes the tab, run it as a background job, save progress as it goes, and stream updates from the saved state. A dropped connection then loses nothing.
Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.
Book a free 30-minute call about it. Bring your setup and your bill; you'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Or keep reading
Rate Limiting: Token Buckets, Per-User Limits and Answering With 429
Rate limits protect your service from abuse, your bill from runaway clients, and your users from each other. The algorithms (fixed window, sliding window, token bucket), where to enforce limits (edge, gateway, application), what to key them on, and how to tell clients to back off properly.
Read
WebSockets Are Easy to Open and Hard to Run: How to Decide if Your Feature Needs One
A WebSocket demo takes an afternoon. Keeping thousands of connections correct through deploys, reconnects, phones changing networks and a second server takes ongoing work. Many "realtime" features are better served by a 15-second poll or a server-sent event stream. A feature-by-feature guide, and the list of what you take on when the answer really is WebSockets.
Read
As of October 26, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.