API Cost & Rate Limit Management

Why AI Costs Spiral Out of Control

AI API costs are deceptive. A prototype with 100 requests/day costs Rs 200/month. Scale to 100K requests/day and suddenly you are paying Rs 5-10 Lakh/month. Unlike traditional APIs (database queries, REST calls), LLM costs scale with both request volume AND the length of each request/response.

Think of it like mobile data plans. A quick WhatsApp message costs almost nothing. But start streaming Netflix (long context + rich output) and your bill explodes. LLM APIs work the same way - every token counts, literally.

How LLM Pricing Works:

• Token-based pricing - You pay per token (roughly 4 characters or 0.75 words). Input tokens and output tokens are priced separately.
• Input tokens are usually cheaper - GPT-4o: input $2.50/1M vs output $10/1M. So sending a long prompt is 4x cheaper than generating a long response.
• Model tier determines cost - GPT-4o-mini is 10-20x cheaper than GPT-4o. Claude Haiku is 12x cheaper than Claude Sonnet.
• Hidden costs - System prompts count as input tokens every request. A 2000-token system prompt at 100K requests/day = 200M tokens/day just in system prompts!

Cost Example - Customer Support Bot:

Now switch to GPT-4o-mini: same workload costs ~Rs 1,500/month. Model choice is the biggest cost lever.

Strategy 1: Optimize System Prompts

Your system prompt is sent with every single request. A 3000-token prompt vs a 1000-token prompt saves 2000 tokens per request. At 100K requests/day, that is 200M tokens saved = Rs 12,500/month on GPT-4o.

• Remove redundant instructions
• Use concise language (the model understands brevity)
• Move rarely-needed instructions to user messages instead
• Use Anthropic prompt caching (90% off cached tokens)

Strategy 2: Conversation History Management

In chat applications, the full conversation history is sent with each request. A 20-message conversation uses 10x more tokens than the first message.

• Sliding Window - Keep only the last N messages. Simple but loses context.
• Summarization - Periodically summarize older messages into a compact summary. Best quality-cost balance.
• RAG for History - Store history in vector DB, retrieve only relevant messages. Most sophisticated.
• Token Budget - Set a max token budget per conversation. Trim oldest messages when exceeded.

Strategy 3: Smart Model Selection

• Classify first, then route - Use a tiny classifier (GPT-4o-mini, ~Rs 0.001/request) to determine query complexity. Route simple queries to cheap models, complex ones to premium.
• Model cascading - Try the cheap model first. If response quality is low (detected by a quality check), retry with a better model.
• Task-specific models - Use embeddings models for search (100x cheaper than chat), specialized models for specific tasks.

Strategy 4: Output Control

• max_tokens - Always set it. Prevent models from generating 4000-token responses when 200 tokens suffice.
• Structured outputs - JSON mode prevents verbose prose. Output only the fields you need.
• Stop sequences - Stop generation at natural boundaries. No wasted tokens after the answer is complete.

How Rate Limits Work

Every AI provider imposes rate limits - maximum requests or tokens per minute/day. These exist to prevent abuse and ensure fair access. When you hit a rate limit, you get a 429 Too Many Requests error and must wait.

Types of Rate Limits:

Rate Limit Management Strategies:

• Exponential Backoff - On 429 error: wait 1s, retry. Still 429? Wait 2s. Then 4s, 8s. Add random jitter to prevent thundering herd.
• Request Queuing - Queue requests and process at a controlled rate below the limit. Use token bucket or leaky bucket algorithms.
• Load Balancing Across Keys - Use multiple API keys (different accounts/orgs) and distribute requests. LiteLLM supports this natively.
• Tier Upgrades - Higher spending = higher limits. OpenAI tiers: Free (3 RPM) -> Tier 1 (500 RPM) -> Tier 5 (10K RPM).
• Provider Failover - Hit OpenAI limit? Route overflow to Anthropic or Google. Each has separate limits.

Per-User Rate Limiting (Your Side):

You should also implement your own rate limits to protect against abuse:

• Free tier users: 20 requests/hour
• Paid users: 200 requests/hour
• Enterprise: custom limits

Why Caching is a Game-Changer

Many AI requests are repetitive. Customer support bots answer the same questions repeatedly. Code assistants see similar patterns. Caching identical or similar requests can reduce API calls by 30-70% depending on your use case.

Types of AI Caching:

• Exact Match Cache - Hash the entire request (prompt + parameters). If identical request seen before, return cached response. Simple, effective for deterministic queries (temperature=0). Use Redis with TTL.
• Semantic Cache - Embed the query, find similar past queries by vector similarity. "How do I return a product?" and "What is the return process?" hit the same cache entry. Tools: GPTCache, Redis Vector Search.
• Prompt Cache (Provider-Side) - Anthropic caches system prompt tokens (90% off). OpenAI caches prefix tokens automatically. You get the discount without building anything.
• Response Template Cache - For predictable queries (FAQ-like), pre-generate and cache responses. Serve from cache with zero API calls.

Cache Architecture for AI:

• Layer 1 (Exact) - Redis exact match. Hash of (model + messages + temperature). 100% hit = zero cost.
• Layer 2 (Semantic) - Vector similarity search on past queries. 80%+ similarity = return cached response.
• Layer 3 (Provider) - Anthropic/OpenAI prompt caching. Automatic for system prompts.
• Layer 4 (CDN) - For static AI-generated content (blog posts, descriptions), cache at CDN level.

Cache Invalidation Considerations:

• Set appropriate TTL based on data freshness needs (1 hour for support, 24 hours for static content)
• Invalidate when underlying data changes (product info update, policy change)
• Do not cache when temperature > 0 and variety is needed
• Monitor cache hit rate - below 20% means caching is not helping much for your use case

Horror Stories

A startup accidentally left a debug loop running that called GPT-4 in a tight loop. Rs 8 Lakh bill in one weekend. A dev testing with production keys sent 50K requests in an hour. An agent stuck in an infinite loop consumed Rs 50,000 in tokens before anyone noticed.

These are real scenarios. Without monitoring and limits, AI costs can spiral in minutes.

Budget Protection Layers:

• Provider Spending Limits - OpenAI, Anthropic let you set monthly spending caps. Set these first! Better a failed request than a surprise bill.
• Per-Request Cost Estimation - Before sending a request, estimate its cost (count input tokens, multiply by rate). Reject requests that would exceed per-request budget.
• Per-User Daily Limits - Each user gets a daily token/cost budget. Prevents one user from consuming all resources.
• Application-Level Budget - Total daily/monthly budget for your app. Circuit breaker when 80% consumed. Hard stop at 100%.
• Alert Thresholds - Slack/email alerts at 50%, 75%, 90% of budget. Real-time dashboards showing burn rate.

Monitoring Dashboard Essentials:

• Real-time spend - Current day/month spend vs budget
• Burn rate - At current pace, when will you hit the limit?
• Per-model breakdown - Which models consume most budget?
• Per-feature breakdown - Which features/endpoints cost most?
• Token efficiency - Average tokens per request. Trending up = problem.
• Cache hit rate - Low hit rate = optimization opportunity

Tools for Monitoring:

• LiteLLM - Built-in spend tracking per key/team/model
• Helicone - AI observability platform with cost tracking
• Langfuse - Open-source LLM monitoring with cost analytics
• Provider Dashboards - OpenAI, Anthropic, Google all have usage dashboards

Q: How would you reduce AI API costs by 80% for a production application?

Q: How do you handle rate limits in a high-traffic AI application?

Q: What is semantic caching and how does it differ from exact match caching?

Q: What monitoring should you have for AI API costs in production?

Q: Why are system prompts a hidden cost driver and how do you optimize them?