Vision AI

The Big Picture:

Vision AI refers to AI models that can understand and analyze images - describing what is in a photo, reading text from documents, identifying objects, comparing images, and answering questions about visual content. Modern multimodal LLMs like GPT-4V, Claude 3.5, and Gemini Pro Vision can do all of this.

Think of it like this: traditional LLMs can only read text. Vision AI gives them eyes - they can now see photographs, screenshots, documents, diagrams, charts, and anything visual.

Real-World Analogy - Flipkart Visual Search:

You see a nice kurta someone is wearing. Instead of typing "blue embroidered kurta with golden border", you take a photo and search. Flipkart's Vision AI analyzes the image - identifies the garment type, color, pattern, embroidery style - and finds similar products. This is Vision AI in action.

Types of Vision AI Tasks:

What are Multimodal LLMs?

Multimodal LLMs are language models that can process both text and images as input. Instead of just receiving text prompts, you can send them images (photos, screenshots, documents, diagrams) along with text questions, and they respond intelligently about the visual content.

How They Work (Simplified):

1. Image Encoding: The image is broken into small patches (like puzzle pieces) and converted into numerical vectors (embeddings) by a vision encoder.
2. Alignment: These image embeddings are projected into the same space as text token embeddings, so the LLM can process them together.
3. Processing: The LLM processes the combined text + image tokens using the same transformer architecture, generating a text response.

The result: you can ask "What is in this image?" and get a detailed text description, or ask "How many people are in this photo?" and get an accurate count.

Key Multimodal Models:

• GPT-4V / GPT-4o (OpenAI): Excellent general vision. Great at reading text, understanding charts, and complex reasoning about images.
• Claude 3.5 Sonnet (Anthropic): Strong vision capabilities. Particularly good at document understanding and detailed image analysis.
• Gemini Pro Vision (Google): Native multimodal - trained on images from the start. Good at spatial reasoning and multi-image comparison.
• LLaVA / Qwen-VL (Open Source): Open-source alternatives. Good for self-hosted deployments and privacy-sensitive use cases.

Cost Considerations:

Images consume tokens too! A typical image uses 500-2000 tokens depending on resolution and detail level. Processing 1000 images with GPT-4V can cost $5-20. Optimize by resizing images and using low-detail mode when full resolution is not needed.

What is OCR?

OCR (Optical Character Recognition) is the technology that converts images of text into actual text data. From reading printed documents to extracting data from handwritten forms - OCR digitizes the physical world.

OCR in Indian Context - Real Applications:

• Aadhaar/PAN Card Processing: Extract name, number, DOB from ID card photos. Banks and fintech apps use this for KYC verification.
• Invoice Processing: Read vendor name, items, amounts from scanned invoices. Automates accounting data entry.
• Prescription Reading: Digitize doctor prescriptions (handwritten!) for pharmacy apps. One of the hardest OCR challenges.
• Meter Reading: Electricity boards use OCR to read meter photos instead of manual entry.
• Cheque Processing: Banks use MICR + OCR to read cheque details for clearance.

OCR Approaches:

For simple clean text, traditional OCR is fastest and cheapest. For complex documents where you need to understand context (not just read text), multimodal LLMs are superior.

1. Document Processing Pipeline:

Use Case: Auto-process expense reports

[Employee uploads receipt photo]
        |
        v
[Image Preprocessing]
  - Auto-rotate if tilted
  - Enhance contrast for faded receipts
  - Crop to receipt boundary
        |
        v
[Multimodal LLM Analysis]
  Prompt: "Extract from this receipt:
    - Vendor name
    - Date
    - Items with prices
    - Total amount
    - Payment method
    Return as structured JSON"
        |
        v
[Validation]
  - Total matches sum of items?
  - Date is reasonable?
  - Amount within policy limits?
        |
        v
[Auto-fill expense report in ERP system]

2. Visual Quality Inspection:

Manufacturing companies use Vision AI to inspect products on assembly lines - detecting scratches, dents, color variations, or assembly errors that human eyes might miss.

3. Accessibility:

Vision AI can describe images for visually impaired users, read signboards, identify objects in surroundings - making the visual world accessible through text and speech.

Best Practices:

• Image Quality: Better input = better output. Preprocess images before sending to vision AI.
• Prompt Engineering: Be specific about what you want. "Describe this image" gives generic results. "List all food items with estimated calories" gives useful data.
• Cost Optimization: Resize images before sending. 512x512 is often sufficient. Use low-detail mode for simple tasks.
• Fallback Strategy: Vision AI can fail on blurry, dark, or unusual images. Always have a human review fallback.

Current Limitations:

• Counting: Vision AI struggles with accurately counting large numbers of objects ("How many people in this stadium photo?")
• Spatial Reasoning: Struggles with precise spatial relationships ("Is the red car to the left or right of the blue car?")
• Small Text: Tiny text in large images may not be readable. Cropping helps.
• Hallucination: Can describe objects that are not actually in the image, or misread text
• Multi-language OCR: Performs best with English text. Hindi, Tamil, and other Indian scripts have lower accuracy
• Speed: Vision API calls are slower than text-only calls (1-5 seconds typical)

Privacy and Ethical Considerations:

• Facial Recognition: Using Vision AI for identifying individuals raises serious privacy and legal concerns
• Bias: Models may perform differently across skin tones, cultures, and geographies
• Consent: Processing images of people without their consent can violate privacy laws
• Data Retention: Images sent to cloud APIs may be stored. Use on-premise solutions for sensitive data.

Q: How do multimodal LLMs process images?

Q: When would you use multimodal LLMs for OCR vs traditional OCR?

Q: What are the limitations of current Vision AI?

Q: How would you build a document processing pipeline with Vision AI?