LLMOps & Model Lifecycle Management

Real-World Analogy - Running a Zomato Kitchen

Think of LLMOps like managing a large Zomato cloud kitchen. You do not just cook food (build models) - you need to manage recipes (prompts), track ingredient quality (data quality), monitor cooking times (latency), control costs (token usage), handle customer complaints (hallucinations), train new chefs (fine-tuning), and ensure food safety (AI safety). LLMOps manages ALL of these systematically.

MLOps vs LLMOps

The LLMOps Lifecycle

• 1. Development: Prompt engineering, RAG setup, model selection
• 2. Evaluation: Automated testing, human review, benchmarks
• 3. Deployment: A/B testing, canary releases, rollback strategies
• 4. Monitoring: Quality tracking, latency, cost, hallucination detection
• 5. Optimization: Cost reduction, caching, model switching, fine-tuning
• 6. Governance: Audit trails, compliance, access control, data privacy

The Prompt Management Problem

A Flipkart product chatbot might have 20 complex prompts. Each needs to be: version controlled, A/B tested, environment-specific, parameterized, and auditable. Without proper management, one bad prompt change can break everything.

Best Practices

• Separate Prompts from Code: Store in a dedicated system - database, config files, or a prompt management platform.
• Semantic Versioning: Major = behavior change, Minor = improvement, Patch = typo fix.
• Change Tracking: Every change should document what changed, why, and link to evaluation results.
• Rollback Capability: Instantly revert to any previous prompt version.
• Environment Promotion: Dev to staging to production, just like code.

Tools for Prompt Management

• LangSmith: Full prompt versioning, testing, and monitoring by LangChain
• PromptLayer: Dedicated prompt management with analytics
• Humanloop: Prompt management + evaluation + fine-tuning
• Dify Prompt IDE: Built-in A/B testing and version control

The Evaluation Pyramid

• Layer 1 - Automated Metrics (Base): Fast, cheap, run on every change. Format validation, keyword presence, toxicity scores, embedding similarity.
• Layer 2 - LLM-as-Judge (Middle): Use GPT-4 to evaluate your app LLM output. Checks factual accuracy, relevance, coherence.
• Layer 3 - Human Evaluation (Top): Domain experts review samples. Checks nuanced quality, business correctness. Slow but essential.

Building an Evaluation Suite

• Golden Dataset: 100-500 curated QA pairs representing key use cases - your AI unit test suite.
• Edge Cases: Ambiguous questions, adversarial prompts, out-of-scope queries.
• Regression Tests: Previously bad answers. Every bug fix adds a test case.
• A/B Comparisons: Compare new vs production on same inputs.

Key Metrics

• Faithfulness: Does the answer stick to provided context? (Critical for RAG)
• Relevance: Does it address the actual question?
• Hallucination Rate: Percentage of made-up information
• Toxicity Score: Harmful or inappropriate content
• Latency: Time to first token and total response time
• Cost Per Query: Average token usage and cost per interaction

Tools

• RAGAS: Open-source RAG evaluation framework
• DeepEval: LLM unit testing with 14+ metrics
• Promptfoo: Open-source prompt testing and comparison

Four Pillars of LLM Monitoring

• System Metrics: API latency, error rates, throughput, memory. Tells you IF the system is healthy.
• Quality Metrics: Hallucination detection, relevance scoring, format validation. Early warning system.
• Business Metrics: Thumbs up/down, task completion rate, support ticket creation after AI interaction.
• Cost Metrics: Token usage per request, cost per conversation, monthly burn rate by model.

Example: Monitoring a PhonePe AI Support Bot

• Latency Alert: P95 response time exceeds 8 seconds
• Quality Alert: Hallucination rate exceeds 5% in last hour
• Business Alert: Users saying talk to human increased 40% today
• Cost Alert: Daily token spend exceeded budget by 200%

Each requires different response. Latency = scale up. Quality = prompt regression. Business = new query type. Cost = possible attack.

Monitoring Stack

• LangSmith / LangFuse: LLM-specific trace and observability
• Datadog / Grafana: Infrastructure with custom LLM dashboards
• Custom Quality Pipeline: Async quality checks on sampled responses
• User Feedback Loop: Thumbs up/down on every response

Model Selection - Indian Transport Analogy

• Auto-rickshaw (GPT-3.5, Haiku): Simple queries, FAQ answers. Fast, cheap. 60-70% of traffic.
• Sedan (GPT-4o-mini, Sonnet): Moderate complexity, summarization. 20-25% of traffic.
• Mercedes (GPT-4, Opus): Complex reasoning, critical decisions. Expensive. Only 5-10% of traffic.

Cost Optimization Techniques

• Semantic Caching: Cache responses for similar queries. 30-50% cost reduction.
• Prompt Compression: Remove unnecessary context. Every token costs money.
• Request Routing: Classify queries and route to appropriate model tier.
• Batch Processing: Group non-urgent requests. 50% discount on batch APIs.
• Fine-tuning: Fine-tuned GPT-3.5 can match GPT-4 for specific domains at 1/10th cost.

Real Cost Example: 100K queries/day chatbot

• Naive (all GPT-4): ~$6,000/day = $180,000/month
• Optimized: 70K on GPT-3.5 ($70) + 25K on GPT-4o-mini ($125) + 5K on GPT-4 ($300) + 40% cache savings
• Total: ~$295/day = $8,850/month (95% savings!)

LLM Deployment Strategies

• Shadow Mode: Run new version alongside production without serving responses. Compare offline.
• Canary Release: Route 5% traffic to new version. Monitor. Gradually increase.
• A/B Testing: Split users, serve different versions. Measure business metrics.
• Blue-Green: Two identical environments. Switch traffic instantly. Rollback by switching back.

LLM-Specific Incident Scenarios

• Provider Outage: OpenAI goes down. Need automatic failover to Anthropic or local model.
• Hallucination Spike: Model update causes quality drop. Auto-rollback needed.
• Prompt Injection: Users manipulate AI to reveal system prompts. Need input validation.
• Cost Explosion: Bug causes infinite loops. Need rate limits and cost circuit breakers.

Building Resilience

• Multi-Provider: Primary (OpenAI) then Secondary (Anthropic) then Tertiary (Local Llama)
• Circuit Breakers: Auto-switch on slow/erroring providers
• Response Validation: Every response passes format, safety, and business rule checks
• Graceful Degradation: If all AI fails, show cached responses or redirect to humans

Q1: How would you design a monitoring system for an LLM application?

Answer: Four pillars: (1) System Metrics via Datadog/Prometheus for latency and errors. (2) Quality Metrics via LangSmith sampling responses for hallucination and relevance. (3) Business Metrics via user feedback and task completion rates. (4) Cost Metrics tracking token usage and daily spend. Key insight: system health does not equal output quality - you need both.

Q2: How would you reduce LLM costs by 80% without sacrificing quality?

Answer: Multi-layered approach: (1) Request Routing to appropriate model tiers - 60-70% on cheap models. (2) Semantic Caching reducing redundant calls by 30-40%. (3) Prompt Optimization auditing for unnecessary tokens. (4) Batch Processing for non-urgent requests at 50% discount. (5) Fine-tuning smaller models for high-volume domain tasks.

Q3: Your LLM is hallucinating more after a model update. How do you handle it?

Answer: Confirm via quality monitoring metrics. Immediately rollback by pinning to previous model version. Run golden dataset evaluation to quantify degradation. Report to provider with examples. Evaluate fallback provider. Add failing cases to regression tests. Implement automated canary testing that blocks updates if quality drops below thresholds.