FastAPI for AI Backends (Streaming, WebSockets)

Why Not Flask or Django?

FastAPI was built for modern async Python. AI backends have unique requirements: streaming responses (LLM tokens arrive one by one), WebSocket connections (real-time chat), concurrent requests (multiple users querying models simultaneously), and long-running tasks (model inference can take seconds). FastAPI handles all of these natively with async/await, while Flask and Django need workarounds.

Analogy - Railway Booking Counter:

Flask is like a single-window booking counter at a small railway station -- one person at a time. Django is like a large station with many counters but lots of paperwork (ORM, admin, templates). FastAPI is like IRCTC online -- handles thousands of simultaneous bookings, gives instant feedback (confirmation SMS), and auto-generates the help documentation. Built for speed and scale from day one.

FastAPI Superpowers for AI:

• StreamingResponse: Send LLM tokens as they are generated, not waiting for the full response
• WebSocket Support: Persistent bidirectional connections for real-time chat
• Async/Await: Handle hundreds of concurrent AI requests without blocking
• Pydantic Validation: Type-safe request/response models, auto-validated
• Auto OpenAPI Docs: Swagger UI generated automatically -- perfect for frontend teams
• Dependency Injection: Cleanly share models, DB connections, rate limiters across routes

The Streaming Problem:

When ChatGPT generates a response, you see tokens appearing one by one (typewriter effect). Without streaming, the user stares at a loading spinner for 5-10 seconds until the full response is ready. This is terrible UX. Server-Sent Events (SSE) solve this by sending each token as soon as the LLM generates it.

How SSE Works in FastAPI:

• StreamingResponse: FastAPI returns a StreamingResponse with media_type="text/event-stream". The response body is a Python async generator that yields data chunks.
• Event Format: Each chunk follows the SSE protocol: "data: {token_text}" followed by two newlines. The client (browser) parses these events automatically.
• Connection: The HTTP connection stays open until the generator is exhausted (response complete) or the client disconnects.

SSE vs WebSocket for Streaming:

When to Use Which:

• SSE: Simple LLM streaming where user sends a prompt and gets a streamed response. One direction (server to client). Most ChatGPT clones use this.
• WebSocket: Real-time bidirectional communication. Live typing indicators, collaborative editing, multi-turn conversations with interruption support.

When You Need WebSockets:

While SSE handles simple streaming, some AI applications need more: real-time typing indicators, user can interrupt mid-generation, server pushes updates (agent status changes), or multi-user collaboration. WebSockets provide a persistent, bidirectional connection where both client and server can send messages at any time.

FastAPI WebSocket Architecture:

• Connection: Client opens a WebSocket connection to /ws endpoint. FastAPI accepts and maintains the connection.
• Message Loop: An infinite loop receives messages from client, processes them (sends to LLM), and streams tokens back through the same connection.
• Connection Manager: A class that tracks all active connections. Handles connect, disconnect, broadcast to multiple users.
• Heartbeat: Periodic ping/pong messages to detect dead connections.

Production Challenges:

• Scaling: WebSocket connections are stateful. If you have 3 server instances behind a load balancer, a user's connection is stuck to one server. You need sticky sessions or Redis pub/sub for cross-server communication.
• Reconnection: Mobile users lose connection frequently (metro, elevator). Implement auto-reconnect with exponential backoff on the client side.
• Memory: Each open connection consumes server memory. 10,000 connections = significant RAM.
• Timeouts: Nginx/ALB default timeouts are 60 seconds. Configure them higher for AI apps where responses can take longer.

Typical Architecture:

• API Layer (FastAPI): Request validation, auth, rate limiting, routing. Handles SSE streaming and WebSocket connections.
• Service Layer: Business logic -- prompt engineering, context management, conversation history, tool orchestration.
• Model Layer: LLM API calls (OpenAI, Anthropic, local models). Handles retries, fallbacks (switch to cheaper model if primary is down), and response parsing.
• Queue Layer (Redis/Celery): For heavy tasks -- document processing, embedding generation, batch inference. Offload from the API process to background workers.
• Storage Layer: PostgreSQL for conversations, Redis for caching, S3 for file uploads, vector DB (Pinecone/ChromaDB) for RAG.

Key Design Patterns:

• Dependency Injection: FastAPI Depends() for sharing LLM clients, DB sessions, auth. Clean and testable.
• Middleware: Request logging, CORS, rate limiting, error handling -- all as middleware layers.
• Background Tasks: FastAPI BackgroundTasks for fire-and-forget operations (logging, analytics, cache warming).
• Health Checks: /health endpoint that checks LLM API connectivity, DB status, queue health.

Indian-Scale Considerations:

• Regional Latency: Deploy in Mumbai region (ap-south-1) for Indian users. LLM APIs are usually in US -- add caching to reduce round trips.
• Cost: GPT-4 API calls are expensive. Use cheaper models (GPT-3.5, Gemini Flash) for simpler queries. Implement a model router.
• Bhashini Integration: Government AI translation API for Indian language support.

Critical Pitfalls:

• Blocking the Event Loop: If you call a synchronous LLM library in an async FastAPI route, it blocks the entire server. Use async LLM clients (httpx, aiohttp) or run sync code in thread pools with asyncio.to_thread().
• No Timeout on LLM Calls: LLM APIs can hang for 30+ seconds. Always set timeouts. If the model is slow, return a "try again" message rather than keeping the connection open forever.
• Storing Chat History in Memory: Server restarts lose everything. Use Redis or PostgreSQL for conversation persistence.
• No Rate Limiting: One user can drain your LLM API budget. Implement per-user rate limits with tokens-per-minute tracking.
• Exposing API Keys: Never send LLM API keys to the frontend. The backend calls the LLM; the frontend talks to your backend.

Best Practices:

• Async Everything: Use async/await for all I/O operations (LLM calls, DB queries, file reads)
• Structured Logging: Log every LLM call with prompt, model, tokens used, latency, cost
• Circuit Breaker: If OpenAI is down, fail fast and switch to fallback model
• Request ID Tracing: Assign unique ID to each request for debugging across services
• Graceful Shutdown: Finish active streaming responses before shutting down the server

Q: Why is FastAPI preferred over Flask for AI backends?

Q: How does Server-Sent Events (SSE) work for LLM streaming?

Q: When would you use WebSockets instead of SSE for AI chat?

Q: What happens when you block the async event loop in FastAPI?

Q: How do you handle WebSocket scaling across multiple server instances?