AI System Design Interview Prep

The 5-Step Framework (45-60 minutes total)

• Step 1 - Clarify Requirements (5 min): Ask questions aggressively. What are the users? What scale? What quality bar? What latency is acceptable? What is the budget? Do NOT start designing before you deeply understand the problem.
• Step 2 - High-Level Architecture (10 min): Draw the major components on the whiteboard. User interface, API layer, AI processing pipeline, data stores, external services. Show how data flows through the entire system.
• Step 3 - Deep Dive on AI Components (15 min): This is the core of your interview. Model selection and reasoning, prompt design, RAG pipeline details, evaluation strategy, safety guardrails. Show depth of AI knowledge here.
• Step 4 - Scale & Reliability (10 min): How does this handle 100x traffic? What happens when the LLM provider goes down? Cost optimization strategies. Caching and batching opportunities.
• Step 5 - Monitoring & Evolution (5 min): How do you know if the system is working well? What metrics matter most? How do you improve the system over time? What would version 2 look like?

Common Mistakes That Sink Candidates

• Jumping straight to the solution before understanding requirements thoroughly
• Focusing only on the AI/ML part while completely ignoring infrastructure and ops
• Not discussing trade-offs - every single decision has pros and cons
• Ignoring cost entirely - AI systems are expensive to run at scale
• Forgetting to mention monitoring, evaluation, and feedback loops
• Over-engineering a simple use case that does not need that much complexity

Step 1: Clarify Requirements

• Scale: 50,000 customer queries per day across web and mobile
• Languages: English and Hindi (with Hinglish code-switching)
• Current system: Human agents with 15-minute average response time
• Goal: Resolve 70% of queries automatically, reduce response time to under 30 seconds
• Budget: Must keep cost under Rs 2 per query for it to be viable
• Quality: Must not give wrong information about orders, refunds, or policies

Step 2: High-Level Architecture

• Query Classifier: First, classify the incoming query by intent - is it about orders, returns, payments, general FAQ, or a complaint? A small fine-tuned classifier handles this quickly and cheaply.
• Intent Router: Route to the appropriate handler. Order-related queries go to order service plus AI. General FAQ goes to the RAG system. Complex or emotional issues go directly to human agents.
• RAG Engine: Knowledge base containing product policies, FAQ documents, return policies, and troubleshooting guides. Updated daily by the content team.
• Order Context Engine: Pulls real-time order data from the order management system. The AI can see order status, delivery tracking, payment history, and previous interactions.
• Response Generator: LLM generates a personalized response grounded in both RAG context and real-time order data. Supports English, Hindi, and Hinglish.
• Human Escalation: If AI confidence is low, user explicitly asks for a human, or the issue involves sensitive topics - seamlessly hand off to a human agent with the full conversation context preserved.

Step 3: AI Deep Dive

• Model Selection: GPT-4o-mini for response generation (great quality at low cost). Fine-tuned distilBERT for intent classification (extremely fast, near-zero cost per inference).
• Cost Analysis: Classification: ~Rs 0.01/query. RAG retrieval: ~Rs 0.02. LLM generation: ~Rs 0.80. Total: approximately Rs 1.1 per query - comfortably within the Rs 2 budget.
• Safety Guardrails: Never expose internal system details or error codes. Never make promises the company cannot keep. Always offer human escalation as an option. Automatically flag potentially sensitive queries (legal threats, safety issues) for immediate human review.

Step 4: Scale & Reliability

Cache frequent queries with identical answers (return policy is the same every time). Multi-provider LLM fallback (OpenAI to Anthropic). Per-user rate limiting to prevent abuse. Graceful degradation to pre-written FAQ template answers if the entire AI layer goes down. Queue-based processing during traffic spikes.

Step 1: Requirements

• 10 million posts per day (text, images, short videos)
• Must detect: hate speech, violence, spam, misinformation, CSAM
• Latency: flag content within 60 seconds of posting
• False positive rate must be below 1% (wrongly removing content destroys user trust)
• False negative rate must be below 0.1% for severe violations (CSAM, violence)

Step 2: Architecture - The Tiered Approach

• Tier 1 - Fast Filters (under 100ms): Keyword blocklists, known hash matching (PhotoDNA for CSAM), regex patterns for spam. Catches the most obvious violations instantly. Handles approximately 60% of all violations caught.
• Tier 2 - ML Classifiers (1-5 seconds): Specialized lightweight models for each category: toxicity classifier, spam detector, NSFW image classifier. Fast inference, low cost per item. Catches another 30% of violations.
• Tier 3 - LLM Review (10-30 seconds): For borderline cases that Tier 1 and 2 are uncertain about. The LLM understands nuanced context, sarcasm, cultural references, and subtle violations. More expensive but much more accurate on hard cases.
• Tier 4 - Human Review: For the most ambiguous edge cases. Appeals process for wrongly removed content. Statistical audit sampling of automated decisions to track system accuracy over time.

Step 3: AI Deep Dive - The Key Insight

• Critical Design Decision: Do NOT use LLMs for every single post. At 10M posts per day, LLM cost would be astronomical and completely unaffordable. Use cheap, fast classifiers for 95% of content. Only escalate genuinely uncertain cases to LLMs. This tiered approach keeps costs manageable.
• Multi-Language Challenge: Indian social media has content in 10+ languages plus code-switching. Use multilingual models (XLM-RoBERTa) for classifiers. Hindi abuse detection is particularly challenging due to frequent Hinglish code-switching and transliteration.
• Adversarial Robustness: Users will actively try to bypass filters using creative techniques - special Unicode characters, intentional misspellings, text embedded in images. Need OCR for text in images, audio transcription for video content, and regularly updated adversarial test sets.

Step 4: Monitoring

Track precision and recall per violation category. Monitor false positive rates segmented by user demographics to catch bias. Track latency distribution and escalation rates. Monitor human review queue size and reviewer agreement rate. Alert on emerging violation patterns that existing models miss.

Step 1: Requirements

• 10 million documents spread across Confluence, Google Drive, Slack, Jira, and SharePoint
• 5,000 employees searching daily, approximately 20 searches per person per day
• Must strictly respect access controls - users can only see documents they have permission for
• Support natural language queries like "Who approved the Q3 budget?" or "What is our refund policy for enterprise clients?"
• Search results must appear in under 3 seconds including AI-generated answers

Step 2: Architecture

• Connector Layer: Purpose-built connectors for each data source (Confluence API, Google Drive API, Slack API, etc.). Each connector crawls content, handles incremental syncs, and detects updates and deletions.
• Processing Pipeline: Document parsing to extract text from PDFs, DOCX, slides, and spreadsheets. Intelligent chunking with metadata preservation (source, author, date, section). Embedding generation for all chunks. ACL (Access Control List) tagging on every single chunk.
• Search Engine: Hybrid search combining vector similarity and keyword matching (BM25). ACL filtering at query time - non-negotiable security requirement. Cross-encoder reranking for final relevance ordering.
• AI Answer Engine: For direct factual questions, generate a synthesized answer with source citations from the top search results. For exploratory or broad queries, return ranked documents with highlighted relevant snippets.

Step 3: The Critical Challenge - Access Control

This is the single most important and trickiest part of enterprise search. Every document has access permissions. A search result that accidentally leaks confidential HR documents or salary data to a regular employee is a catastrophic security incident.

• Pre-filtering approach: Tag each chunk with allowed user groups at indexing time. At query time, filter chunks based on the searching user permissions BEFORE vector search. Fast but requires complete re-indexing when permissions change.
• Post-filtering approach: Retrieve top-K results from vector search, then filter by user permissions. Simpler to implement but may return fewer results than expected since many filtered-out results waste retrieval slots.
• Recommended: Hybrid approach. Pre-filter on broad organizational groups (department, team, office), then post-filter on specific document-level permissions. Best balance between search speed and permission accuracy.

Step 4: Scale Considerations

10M documents at approximately 5 chunks each equals 50M vectors. Use a managed vector database (Pinecone or Weaviate) that handles this scale natively. Batch embedding computation during off-peak hours to manage costs. Implement aggressive semantic caching for frequent queries. Different freshness requirements by source: Slack messages need near-real-time indexing (minutes), Confluence pages can use hourly syncs, archived documents need monthly re-crawls at most.

Common AI System Design Trade-offs

• Quality vs Cost: GPT-4 gives noticeably better answers but costs 30x more than GPT-3.5. When is the quality improvement actually worth the cost? It depends entirely on the use case criticality and who the users are.
• Latency vs Quality: Adding a cross-encoder reranking step improves retrieval quality significantly but adds 200ms latency. Is that acceptable for a real-time chat interface? What about a batch analytics report?
• Freshness vs Cost: Syncing documents every minute catches updates almost instantly but is expensive in API calls and compute. Hourly sync is much cheaper but means search results can be up to 60 minutes stale.
• Automation vs Safety: Fully automated AI responses are lightning fast but carry risk of errors. Human-in-the-loop is safer but adds minutes of delay. What is the right balance for medical vs shopping use cases?
• Build vs Buy: Build your own custom RAG pipeline for full control and deep understanding, or use a managed platform like Pinecone or Cohere for dramatically faster time-to-market?

How to Present Trade-offs Like a Senior Engineer

Use this structured format every time: "We could go with approach X or approach Y. X gives us [specific benefit] but costs us [specific drawback]. Y gives us [specific benefit] but costs us [specific drawback]. Given our stated requirements of [reference the specific requirement you clarified earlier], I would choose X because [concrete reason tied to requirements]."

This structured approach demonstrates clear analytical thinking and helps the interviewer follow and evaluate your reasoning.

Universal Design Principles for AI Systems

• Start simple, add complexity only when you have evidence it is needed
• Always have a graceful fallback for when the AI component fails
• Separate AI logic from business logic cleanly - they change at different rates
• Make every AI decision observable, logged, and debuggable
• Design for cost awareness from day one, not as an afterthought when the bill arrives
• Build evaluation and quality metrics into the system architecture, not as a separate afterthought project

Drawing the Architecture Clearly

• Start from the user and work inward (left to right or top to bottom flow)
• Use clear labeled boxes for each component - avoid ambiguous cloud shapes
• Draw directional arrows showing data flow with annotations
• Mark which components are AI-powered with a star or distinct color
• Show external dependencies clearly and separately (LLM APIs, third-party services, databases)
• Add estimated latency at key points in the data flow path

Communication Tips That Win Interviews

• Think aloud constantly: Narrate your thought process. "I am considering between option X and Y because..." Silence during a design interview is your enemy.
• Check in with the interviewer: "Should I go deeper on the RAG pipeline, or would you like me to move to scaling considerations?"
• Be honest about unknowns: "I have not personally used this specific tool, but based on what I know about its capabilities, I would expect..." Honesty builds trust.
• Use concrete numbers: Estimate latency, cost per query, storage requirements. Even rough back-of-envelope numbers show engineering rigor and practical experience.
• Always mention alternatives: "We could also use [alternative] which would give us [benefit], but I chose [current] because [specific reason tied to our requirements]..."

Time Management During the Interview

• Requirements clarification: 5 minutes maximum. Be thorough but do not over-discuss.
• High-level architecture: 10 minutes. Get the full structure on the whiteboard.
• AI deep dive: 15-20 minutes. This is where you shine and demonstrate depth.
• Scale and reliability: 10 minutes. Show you think beyond the happy path.
• Leave 5 minutes for interviewer questions, discussion, and your summary.

Q1: Design a real-time AI translation system for a messaging app like WhatsApp.

Key Design Points: Automatic language detection on every message. Real-time translation under 500ms for short messages. Aggressive caching of common phrases and greetings. Use smaller, specialized models for high-traffic language pairs (Hindi-English is fastest). Fall back to larger general models for rare language pairs. Handle informal language, slang, and Hinglish code-switching gracefully. Display original message with translation below it. Do not translate messages already in the recipient language. Key insight: at massive scale, specialized translation APIs are significantly cheaper than using general LLMs for every message.

Q2: Design an AI-powered code review tool for a large engineering organization.

Key Design Points: Integrate directly into GitHub/GitLab PR workflow. Tier 1: Static analysis plus rule-based checks (instant, near-zero cost). Tier 2: ML-based vulnerability and common bug detection (seconds). Tier 3: LLM-powered logic review, readability suggestions, and architecture feedback (30-60 seconds). Only run expensive LLM review on changed files, not the entire codebase. Cache analysis results for unchanged code across PRs. Allow developers to give thumbs-up/down feedback on every suggestion. Start in non-blocking suggestion mode; graduate to required checks only after accuracy consistently exceeds 90%. Track suggestion acceptance rate as the primary quality metric.

Q3: Design an AI-powered food recommendation system for a Swiggy-like delivery app.

Key Design Points: Combine collaborative filtering (users similar to you ordered X) with content-based filtering (you love biryani, here is a new biryani restaurant). Add an LLM layer for natural language query understanding ("something light for dinner" should understand dietary intent). Real-time contextual signals: time of day, current weather, user location, order history, dietary preferences, budget range. Cold start problem: new users get location-based popular items; new restaurants get a visibility boost initially. A/B test every recommendation algorithm change rigorously. The key success metric is order conversion rate (orders per session), not just click-through rate on recommendations.