CI/CD for AI Applications

Real-World Analogy - Flipkart Product Launch Process

When Flipkart launches a new product feature, they do not just push code and pray. They have automated tests, staging environments, gradual rollouts, and rollback plans. CI/CD for AI is the same discipline applied to AI apps. Every prompt change, every model switch, every RAG pipeline update goes through automated quality gates before reaching users. No more "I changed one word in the prompt and everything broke in production."

Traditional CI/CD vs AI CI/CD

The AI CI/CD Pipeline Stages

• 1. Code Quality: Linting, type checks, formatting
• 2. Unit Tests: Traditional code tests + prompt format validation
• 3. AI Quality Tests: Run prompts against golden dataset, check quality scores
• 4. Cost Check: Estimate token usage and cost impact of changes
• 5. Build: Docker image, push to registry
• 6. Deploy: Staging first, then canary to production
• 7. Post-Deploy: Monitor quality metrics, auto-rollback if degraded

Key GitHub Actions Concepts for AI

• Workflows: YAML files in .github/workflows/ that define your pipeline
• Triggers: Run on push, pull request, schedule, or manual dispatch
• Secrets: Store OPENAI_API_KEY, ANTHROPIC_API_KEY securely
• Caching: Cache pip dependencies and Docker layers for faster builds
• Matrix Builds: Test across multiple Python versions simultaneously
• Environments: Staging and production with approval gates

AI-Specific Pipeline Steps

• Prompt Validation: Check prompt templates parse correctly, no syntax errors
• Golden Dataset Eval: Run 50-100 test queries, compare against expected answers
• Quality Gate: Fail the build if average quality score drops below 0.8
• Cost Estimation: Calculate expected token usage based on prompt length changes
• Regression Check: Compare new outputs against last successful deployment

Pipeline Speed Tips

• Cache Python packages with actions/cache to save 2-5 minutes per run
• Use smaller evaluation datasets for PR checks (20 items), full suite for main branch
• Run AI eval tests in parallel to reduce total pipeline time
• Skip AI tests for documentation-only changes using path filters

Layer 1: Fast Tests (Every PR, Under 2 Minutes)

• Code Linting: Python type checks, formatting (ruff, mypy)
• Prompt Validation: Templates render without errors, required variables present
• Schema Tests: API responses match expected schema
• Mock LLM Tests: Test your code logic with mocked LLM responses
• Config Validation: Model names, temperature values, and token limits are valid

Layer 2: Quality Tests (Main Branch, 5-10 Minutes)

• Golden Dataset: Run 50-100 curated test cases against real LLM
• Quality Scoring: Use LLM-as-judge or RAGAS to score responses
• Regression Detection: Compare scores against last deployment baseline
• Edge Cases: Test adversarial inputs, empty inputs, very long inputs

Layer 3: Integration Tests (Pre-Release, 15-30 Minutes)

• End-to-End: Full user flow from API request to response
• RAG Pipeline: Document ingestion, embedding, retrieval, generation
• Multi-Turn: Conversation flows with context retention
• Cost Estimation: Calculate total cost for test suite, compare to budget

Handling Non-Determinism

LLM outputs are non-deterministic. How do you test something that gives different answers?

• Set temperature to 0 for deterministic tests (not fully deterministic but close)
• Use semantic similarity instead of exact match
• Check for presence of required keywords/concepts
• Use LLM-as-judge for subjective quality scoring
• Run each test 3 times and take the average score

Strategy 1: Blue-Green Deployment

• Two identical environments: Blue (current) and Green (new)
• Deploy to Green, run smoke tests, switch traffic
• Instant rollback by switching back to Blue
• Best for: Major version changes, model switches

Strategy 2: Canary Deployment

• Route 5% traffic to new version, 95% to current
• Monitor quality metrics on the 5% for 1-2 hours
• Gradually increase to 25%, 50%, 100%
• Auto-rollback if quality score drops below threshold
• Best for: Prompt changes, config updates

Strategy 3: Shadow Deployment

• New version runs in parallel but does NOT serve responses to users
• Both versions process the same requests, outputs compared offline
• Zero risk to users. Perfect for evaluating major changes.
• Cost: You pay for two LLM calls per request during shadow period

Rollback Checklist for AI

• Pin to previous Docker image tag (container rollback)
• Revert to previous prompt version (prompt rollback)
• Switch to previous model version if applicable
• Restore previous RAG index if documents changed
• Verify rollback with golden dataset tests

Post-Deploy Monitoring (First 2 Hours)

• Quality Score Watch: Compare real-time quality scores against pre-deploy baseline
• Error Rate: Any increase in API errors or timeout rates
• Latency: P95 latency should stay within 20% of baseline
• Cost Per Request: Token usage should not spike unexpectedly
• User Feedback: Watch thumbs-down rate for immediate quality signals

Auto-Rollback Triggers

• Quality Drop: Average quality score drops more than 10% for 15 minutes
• Error Spike: Error rate exceeds 5% (up from normal 0.5%)
• Latency Spike: P95 exceeds 2x baseline for 10 minutes
• Cost Spike: Per-request cost exceeds 3x normal

When triggered: Automatically revert to last known good deployment. Alert the team. Log the incident.

Post-Deploy Verification Workflow

• Deploy completes successfully
• Run smoke tests (5-10 key queries) against production
• Start canary monitoring dashboard
• Wait 30 minutes, check all metrics
• If all green, increase traffic gradually
• If any red, auto-rollback and investigate

Q1: How does CI/CD for AI applications differ from traditional CI/CD?

Answer: AI CI/CD adds three extra dimensions: (1) Quality testing - running prompts against golden datasets and scoring outputs with LLM-as-judge, not just pass/fail tests. (2) Cost validation - estimating token usage impact of changes and comparing against budget. (3) Non-deterministic testing - using semantic similarity and quality scoring instead of exact output matching. Traditional CI/CD artifacts are code; AI CI/CD also versions prompts, model configs, and RAG indexes.

Q2: How do you test non-deterministic LLM outputs in a CI pipeline?

Answer: Five techniques: (1) Set temperature to 0 for near-deterministic responses. (2) Use semantic similarity (embedding comparison) instead of exact string matching. (3) Check for presence of required concepts/keywords rather than specific wording. (4) Use LLM-as-judge to score quality, relevance, and factual accuracy. (5) Run each test 3 times and take average to smooth out variance. Combine all five for robust testing.

Q3: What deployment strategy would you use for a critical prompt change?

Answer: Shadow deployment first: run new prompt in parallel without serving users, compare outputs offline. If quality is good, proceed to canary: 5% traffic for 2 hours, monitoring quality scores, error rates, and latency. Gradually increase to 25%, 50%, then 100%. Auto-rollback trigger if quality drops more than 10%. Never deploy a major prompt change directly to 100% traffic.

Q4: What should trigger an automatic rollback in an AI deployment?

Answer: Four triggers: (1) Quality score drops more than 10% from baseline for 15 consecutive minutes. (2) Error rate exceeds 5% (up from normal baseline). (3) P95 latency exceeds 2x baseline for 10 minutes. (4) Per-request cost exceeds 3x normal, indicating a prompt bug or infinite loop. Rollback should be automatic to previous known-good version with team notification.