AI/LLM Interview Questions & Answers

Typical AI Interview Rounds

• Round 1 - Screening (30-45 min): Resume discussion, basic LLM concepts, why you want to work in AI. They are checking if you actually know the fundamentals or just list buzzwords.
• Round 2 - Technical Deep Dive (60 min): LLM architecture, RAG systems, embeddings, prompt engineering, fine-tuning vs prompting decisions. This is where depth matters.
• Round 3 - System Design (60 min): Design an AI-powered system end-to-end. Architecture, trade-offs, cost estimation, scaling strategy.
• Round 4 - Coding (45-60 min): Build a simple AI feature, implement a RAG pipeline, or debug an existing LLM integration. Practical skills.
• Round 5 - Behavioral (30-45 min): Past experience, teamwork, handling ambiguity, ethical considerations in AI.

What Different Companies Look For

• Startups: Generalists who can build end-to-end. Full-stack plus AI integration skills. Ship fast.
• Big Tech (Google, Microsoft): Specialists in specific areas like NLP, retrieval systems, or agent frameworks.
• Services (TCS, Infosys, Wipro): Applied AI knowledge, ability to implement solutions for clients, communication skills.
• AI-Native (OpenAI, Anthropic, Cohere): Deep ML knowledge, research mindset, ability to push the frontier.

Q: What is the difference between fine-tuning and prompt engineering? When would you use each?

Answer: Prompt engineering modifies the instructions given to a pre-trained model without changing the model itself. It is fast, cheap, and requires no training data. Best for general tasks where the base model already has the required knowledge. Fine-tuning updates the model weights using domain-specific training data. It is slower, more expensive, and requires curated data, but produces better results for highly specialized domains. The rule of thumb: always try prompt engineering first. Only fine-tune when: (1) prompt engineering consistently fails to meet quality bar, (2) you have quality labeled training data, (3) the domain is very specialized like medical or legal, or (4) you need to reduce inference costs by using a smaller fine-tuned model instead of a larger general one.

Q: Explain the transformer attention mechanism in simple terms.

Answer: Imagine reading a book. When you reach the word "it" in a sentence, your brain automatically looks back to figure out what "it" refers to. That is exactly what attention does for language models. For each word (token), the model asks: "Which other words in this entire text are most important for understanding this particular word right now?" It calculates attention scores between every pair of words, then uses those weighted scores to create a context-aware representation of each word. Self-attention allows every word to attend to every other word in the input, which is how transformers capture long-range dependencies that earlier architectures like RNNs and LSTMs struggled with badly.

Q: What are embeddings and why are they critical for AI applications?

Answer: Embeddings are dense numerical vector representations of text (or images or audio) that capture semantic meaning in a mathematical space. Similar concepts end up with similar vectors - for example, "king" and "queen" would be close in embedding space, while "king" and "banana" would be far apart. They are critical for: (1) Semantic search - finding documents similar in meaning, not just matching keywords. (2) RAG systems - matching user queries to relevant document chunks. (3) Clustering - automatically grouping similar content together. (4) Classification - using embeddings as input features for downstream classifiers. Modern embedding models like OpenAI text-embedding-3-small or Cohere embed v3 produce vectors of 256-3072 dimensions.

Q: What is temperature in LLM inference? How does it affect outputs?

Answer: Temperature controls the randomness in model output by scaling the logit probabilities before sampling. At temperature 0, the model always picks the highest probability token (deterministic, consistent, but potentially repetitive). At temperature 1, the model samples proportionally to the probability distribution (creative, varied). At temperature above 1, outputs become increasingly random and potentially incoherent. Practical guidance: Use low temperature (0-0.3) for factual tasks like Q&A and data extraction. Use medium (0.5-0.7) for balanced tasks like summarization. Use higher (0.7-1.0) for creative tasks like brainstorming and story writing.

Q: Explain the RAG architecture. Why is it preferred over fine-tuning for knowledge-intensive tasks?

Answer: RAG (Retrieval Augmented Generation) combines retrieval from a knowledge base with LLM generation. The process: user query is converted to an embedding, similar chunks are retrieved from a vector database, the retrieved context plus the original query are sent to the LLM, and the LLM generates an answer grounded in that context. RAG is preferred over fine-tuning because: (1) Knowledge can be updated instantly by updating the knowledge base - no retraining needed. (2) Sources can be cited, enabling verification and building trust. (3) No training data or expensive GPU resources required. (4) Works with any LLM without model modification. (5) Cleanly separates knowledge from reasoning capabilities.

Q: Your RAG system retrieves irrelevant documents. How do you systematically fix it?

Answer: Systematic debugging approach with seven strategies: (1) Check embedding quality - are your embeddings capturing semantic meaning? Try a better embedding model like Cohere embed v3. (2) Review chunking strategy - are chunks too large (diluting relevance) or too small (losing context)? Try 500-1000 tokens with 100 token overlap. (3) Implement hybrid search combining vector similarity with BM25 keyword search for better coverage. (4) Add a reranking step using a cross-encoder model to reorder retrieved results by true relevance. (5) Use metadata filtering to narrow the search space before vector search. (6) Implement query expansion - rewrite the user query into multiple search queries to improve recall. (7) Build a golden evaluation dataset and measure improvement objectively with metrics.

Q: What is the difference between vector search, keyword search, and hybrid search?

Answer: Vector search (semantic) converts text to embedding vectors and finds similar vectors using cosine similarity. It understands meaning - searching for "automobile" will find "car" results. But it can miss exact keyword matches and struggles with rare technical terms. Keyword search (BM25/TF-IDF) matches exact words and their frequencies. It excels at finding specific terms, product names, and error codes, but completely misses synonyms. Hybrid search combines both approaches - typically running both searches in parallel and merging results using Reciprocal Rank Fusion (RRF). This gives you the best of both worlds: semantic understanding plus exact keyword matching. Most production RAG systems use hybrid search.

Q: How do you evaluate a RAG system comprehensively?

Answer: Use the RAGAS framework with these metrics: (1) Faithfulness - does the answer only use information from the retrieved context? This measures hallucination. Target above 95%. (2) Answer Relevance - does the answer actually address the question asked? Target above 85%. (3) Context Precision - of the retrieved chunks, how many were actually relevant? Target above 80%. (4) Context Recall - did we retrieve all the relevant information that exists? Target above 70%. Additionally track operational metrics: latency (time to answer), cost per query (tokens consumed), and user satisfaction (thumbs up/down). Build a golden test set of 50-100 curated QA pairs for automated regression testing on every change.

Q: What are AI agents and how do they differ from simple LLM chains?

Answer: A simple LLM chain follows a fixed sequence of steps: prompt goes in, model generates, output comes out. It is like a train on tracks - the path is predetermined. An agent, on the other hand, can make dynamic decisions about which tools to use and in what order based on the current situation. It is like a taxi driver who adapts the route based on traffic conditions. Agents have four key capabilities: (1) Reasoning - they can plan multi-step approaches to solve problems. (2) Tool usage - they can call external APIs, search the web, execute code, read files. (3) Memory - they remember previous interactions and use that context for future decisions. (4) Autonomy - they decide their own action sequence rather than following a hardcoded pipeline.

Q: What are the main challenges with deploying AI agents in production?

Answer: Five major challenges that every production team faces: (1) Reliability - agents can take unexpected paths, use wrong tools, or get stuck in infinite loops. You need maximum step limits, timeout guards, and robust fallback handling. (2) Cost - agents make multiple LLM calls per user request. A single task might trigger 5-15 API calls, making cost unpredictable. You need per-request budgets and cost monitoring. (3) Latency - sequential tool calls add up fast. A research agent might take 30-60 seconds. Users need progress indicators and possibly async processing. (4) Security - agents with tool access can potentially execute harmful actions. You need strict sandboxing, permission systems, and tool allowlists. (5) Observability - debugging why an agent made a wrong decision requires detailed logging of every reasoning step, tool call, and observation.

Q: Explain the ReAct (Reasoning + Acting) pattern for agents.

Answer: ReAct is a prompting pattern where the agent alternates between thinking and acting in a structured cycle. The cycle goes: Thought (the agent reasons about what to do next), Action (the agent uses a tool or takes an action), Observation (the agent observes the result of that action), then the cycle repeats until the task is complete. For example: Thought: "I need to find the current weather in Mumbai to answer this question." Action: "Call the weather API for Mumbai." Observation: "28 degrees Celsius, partly cloudy." Thought: "Now I have the information needed to answer the user." This pattern makes agent reasoning completely transparent and debuggable, which is critical for production systems where you need to understand and fix agent mistakes.

Q: How would you prevent hallucinations in a production LLM application?

Answer: Defense in depth with multiple layers: (1) RAG - ground the model in retrieved facts rather than relying on parametric knowledge alone. (2) Prompt engineering - explicitly instruct the model to say "I do not know" rather than guessing, and use low temperature for factual tasks. (3) Output validation - use an NLI (Natural Language Inference) model to verify the answer is actually supported by the provided context. (4) Citation requirements - force the model to cite specific sources for every claim, making unsupported statements immediately visible. (5) Confidence scoring - use retrieval similarity scores as a proxy for answer confidence and flag low-confidence answers. (6) Human-in-the-loop - for high-stakes applications like medical or legal, route low-confidence answers to human reviewers before showing to users.

Q: How do you defend against prompt injection attacks?

Answer: Multi-layer defense strategy since no single defense is bulletproof: (1) Input filtering - detect and block known injection patterns before they reach the model. (2) System prompt hardening - use XML delimiters or special tokens to clearly separate system instructions from user input. (3) Output monitoring - detect when the model starts revealing system prompt content or attempting unauthorized actions. (4) Principle of least privilege - limit the tools and data accessible to the AI to only what is strictly necessary. (5) Rate limiting - prevent brute-force injection attempts by limiting requests per user. (6) Regular red teaming - proactively test for new injection techniques as they are discovered. This is an active arms race, so defense must be continuously updated.

Q: Your LLM application cost suddenly tripled overnight. How do you investigate?

Answer: Systematic investigation checklist: (1) Check monitoring dashboards for traffic spikes - did user count increase or is a single user responsible? (2) Check average token usage per request - did request sizes suddenly increase? Could indicate a prompt regression or data pipeline issue. (3) Check for abuse - look for bot traffic or prompt injection attempts designed to generate massive responses. (4) Check model routing - did the router accidentally start sending all traffic to an expensive model instead of the cheap one? (5) Check semantic caching - did the cache break, causing every single request to hit the LLM instead of serving cached responses? (6) Implement immediate cost caps while investigating to stop the bleeding. Long-term: set up anomaly detection alerts on daily cost metrics with automatic notifications.

Q: You are asked to add an AI chatbot to a healthcare app. What concerns would you raise?

Answer: I would raise several critical concerns before writing a single line of code: (1) Hallucination risk - medical misinformation can cause real physical harm. The AI must never present uncertain information as medical fact. (2) Regulatory compliance - healthcare AI may need FDA approval in the US or CDSCO approval in India depending on its claims. (3) Liability - who is legally responsible if the AI gives wrong medical advice? Clear disclaimers are mandatory. (4) Data privacy - medical data (PHI) has strict regulations (HIPAA in US, DPDP Act in India). All data handling must be compliant. (5) Bias - AI trained primarily on certain demographics may give worse advice to underrepresented groups. (6) Scope limitations - the AI should explicitly refuse to diagnose conditions and always recommend consulting an actual doctor. I would advocate for starting with low-risk use cases like appointment scheduling before even considering medical Q&A.

Q: Your team wants to use AI to automate code reviews. How would you approach this?

Answer: Careful phased approach: (1) Start by using AI as an assistant, never a replacement - it suggests issues, humans make final decisions. (2) Define clear scope - start with objective checks like style violations, security vulnerabilities, and missing documentation. Avoid subjective "code quality" judgments initially. (3) Evaluate on historical PRs where we already know the review outcome and measure accuracy. (4) Shadow mode for 2 weeks - run AI reviews alongside human reviews without acting on them and compare results. (5) Gradual rollout - begin with non-blocking suggestions, then promote to required checks only as accuracy consistently exceeds 90%. (6) Feedback loop - developers can thumbs-up or thumbs-down AI suggestions to continuously improve the system over time.

Q: How do you stay updated with the rapidly changing AI landscape?

Answer: A structured multi-channel approach: (1) Follow key researchers and practitioners on Twitter/X - people like Andrej Karpathy, Simon Willison, Lilian Weng, and Harrison Chase. (2) Read applied papers on Arxiv from OpenAI, Google DeepMind, Anthropic, and Meta - focus on papers with real benchmarks and code. (3) Build small projects to test new tools and frameworks hands-on. Reading alone is not enough. (4) Participate in communities like Discord servers, Reddit r/LocalLLaMA, and Hacker News AI discussions. (5) Write about what I learn - teaching forces much deeper understanding than passive consumption. (6) Focus on understanding principles over memorizing tools - knowing why things work helps me adapt quickly when tools change.

Rapid-Fire Round - Concise Answers Expected

• Q: Context window vs max tokens?
  A: Context window is the total capacity (input + output) the model can handle - it is a hard model constraint. Max tokens is the limit you configure on output length - it is your configuration choice. Example: GPT-4 has 128K context window, but you might set max_tokens to 2000 for a concise answer.
• Q: Top-p vs temperature?
  A: Both control output randomness but differently. Temperature scales the entire probability distribution. Top-p (nucleus sampling) only considers tokens within the top cumulative probability p, cutting off the long tail. Generally use one or the other, not both simultaneously.
• Q: When streaming vs non-streaming?
  A: Streaming for user-facing chat interfaces (shows real-time progress, feels faster). Non-streaming for batch processing, function calling, and when you need the complete response before processing (like JSON parsing or structured output validation).
• Q: Vector DB vs traditional DB for AI apps?
  A: Vector DBs are optimized for similarity search on high-dimensional embeddings (Pinecone, Weaviate, ChromaDB). Traditional DBs handle structured data, user accounts, and transactions (PostgreSQL). Most production AI apps need both - vector DB for semantic retrieval, traditional DB for everything else.
• Q: System prompt vs user prompt?
  A: System prompt sets AI behavior, personality, constraints, and rules - configured by the developer. User prompt is the actual end-user input. Think of it this way: system prompt is the employee training manual, user prompt is a customer walking in with a request.