AI App Architecture

Levels of AI Integration:

Not every "AI app" is the same. There is a spectrum from light AI touch to fully autonomous AI-driven products:

• Level 1 - AI Feature: An existing app adds an AI feature. Example: Notion AI adds text generation to an existing note-taking app. The app works fine without AI.
• Level 2 - AI-Assisted: AI is central to the workflow but humans drive decisions. Example: GitHub Copilot suggests code but the developer accepts or rejects.
• Level 3 - AI-Native: The product would not exist without AI. Example: ChatGPT, Midjourney, Cursor. The AI IS the product.
• Level 4 - Autonomous AI: AI operates independently with minimal human oversight. Example: Self-driving cars, autonomous trading bots.

Analogy - Adding Spice to Your Daal:

Level 1 is like adding tadka to existing daal -- enhances the dish but daal is still daal. Level 2 is like having a chef suggest recipes while you cook -- they advise, you decide. Level 3 is like a fully automated kitchen -- the cooking IS the technology. Most companies should start at Level 1-2 before jumping to Level 3.

The #1 Mistake:

Companies build AI for the sake of AI. The right approach: Start with a user problem, then evaluate if AI is the best solution. AI is a tool, not a strategy. If a simple rule-based system solves the problem with 95% accuracy, you do not need an LLM that costs 100x more and adds latency.

Pattern 1: Direct LLM Call (Simplest):

User input goes directly to LLM API, response comes back. Good for: text generation, summarization, translation. Challenge: no context beyond the prompt, hallucination risk, no real-time data.

Pattern 2: RAG (Retrieval-Augmented Generation):

Before calling the LLM, retrieve relevant documents/data and include them in the prompt. This grounds the AI in YOUR data, reducing hallucinations. Architecture: User question -> Vector search in your knowledge base -> Top K results added to prompt -> LLM generates answer using those results.

Example: A banking chatbot retrieves your account rules and policies before answering, instead of hallucinating wrong interest rates.

Pattern 3: AI Agent (Autonomous):

The AI plans, uses tools, and acts autonomously. It can search the web, call APIs, query databases, and make decisions. Architecture: User goal -> Agent plans steps -> Executes tools -> Observes results -> Iterates until goal is achieved.

Example: "Book me the cheapest flight to Bangalore next Friday" -- agent searches multiple airlines, compares prices, applies your credit card offers, and books.

Pattern 4: Human-in-the-Loop:

AI does the heavy lifting, but critical decisions require human approval. Architecture: AI generates draft -> Human reviews and approves -> System executes. Used in medical diagnosis (AI suggests, doctor confirms), financial trading (AI recommends, trader approves), and content moderation (AI flags, human decides).

System Prompts as Application Logic:

In traditional apps, your business logic lives in code (if-else, algorithms). In AI apps, much of the logic lives in the system prompt. The system prompt defines: who the AI is (persona), what it can and cannot do (constraints), how it should respond (format), what tools it has access to, and safety guardrails. A bad system prompt = a bad product.

Prompt Engineering Best Practices:

• Be Specific: "Respond in 2-3 sentences" not "Be concise"
• Give Examples: Show the AI 2-3 examples of desired output (few-shot)
• Define Boundaries: "If you do not know, say so. Do not make up information."
• Output Format: Specify JSON schema, markdown format, or structured response
• Persona: "You are a helpful banking assistant for HDFC Bank customers"
• Guardrails: "Never share customer data from one user with another"

Prompt Versioning and Testing:

Treat prompts like code: version them, test them, review changes. A small prompt change can completely alter app behavior. Best practice: store prompts in a config file or database (not hardcoded), A/B test prompt changes, and have a prompt evaluation suite that tests against known inputs/outputs before deploying.

Phase 1: Validate the Idea (1-2 days):

• Try the use case manually in ChatGPT/Claude playground
• Test with 10 real user queries. Does the AI give good answers?
• Identify failure modes -- what questions does it get wrong?
• If AI fails 30%+ of the time, reconsider or add RAG

Phase 2: Build the Prototype (1 week):

• Use Streamlit or Gradio for the UI (fastest path)
• Direct LLM API calls with a well-crafted system prompt
• Add RAG if the AI needs domain-specific knowledge
• Test with 50 users, collect feedback

Phase 3: Production Build (2-4 weeks):

• Proper frontend (Next.js/React) with streaming
• Backend API (FastAPI) with auth, rate limiting, logging
• Evaluation pipeline -- automated tests for AI quality
• Cost monitoring -- track spend per user, per feature
• Feedback loop -- thumbs up/down to track quality over time

Indian AI App Ideas That Work:

• GST Filing Assistant: Upload invoices, AI categorizes and prepares GST returns
• Prescription Reader: Photograph doctor handwriting, AI reads and lists medicines with dosage
• Kisan Mitra: Farmer describes crop issue in Hindi voice, AI diagnoses and suggests treatment
• Interview Prep Bot: AI conducts mock technical interviews, scores answers, gives improvement tips

Cost Traps:

• Token Costs Compound: Every message includes the full conversation history. A 20-message conversation with GPT-4 can cost Rs 5-10 per conversation. Multiply by thousands of users.
• RAG Overhead: Embedding generation for documents, vector DB storage, and retrieval adds 20-40% to your LLM costs.
• Retry Costs: When AI gives bad output and you retry, you pay double. Prompt engineering that reduces retries saves money.

Cost Optimization Strategies:

• Model Routing: Use GPT-3.5/Gemini Flash for simple queries, GPT-4/Claude for complex ones
• Context Window Management: Summarize old messages instead of sending full history
• Caching: Cache responses for common queries (FAQ answers)
• Prompt Optimization: Shorter prompts = fewer tokens = less cost. Remove redundant instructions
• Usage Limits: Free tier with message limits, paid tier for heavy users

Evaluation and Quality:

• LLM-as-Judge: Use a powerful model to evaluate outputs of a cheaper model
• Golden Dataset: 100+ input/output pairs that represent expected behavior. Run after every prompt change.
• User Feedback: Thumbs up/down on every response. Track satisfaction rate over time.
• A/B Testing: Test prompt changes on 10% of traffic before full rollout.

Q: What is RAG and when should you use it?

Q: How do you manage LLM API costs in a production app?

Q: How do you evaluate AI output quality in production?

Q: What is the difference between AI Feature, AI-Assisted, and AI-Native products?

Q: Why should you validate an AI app idea in ChatGPT before writing code?