Multimodal RAG (Images, Tables, PDFs)

The Problem with Text-Only RAG:

Standard RAG only works with plain text. But real-world documents contain charts, tables, diagrams, images, and complex layouts. A financial report has pie charts. A medical report has X-ray images. A research paper has data tables. Text-only RAG misses all of this.

Multimodal RAG extends the pipeline to ingest, embed, retrieve, and reason over multiple data modalities - text, images, tables, and structured documents.

Real-World Analogy - CA Office:

Imagine a Chartered Accountant analyzing a company balance sheet PDF:

• Text-only RAG: Can read paragraphs of text but completely ignores the financial tables, charts, and footnote images. Like a CA who can only read words but not numbers!
• Multimodal RAG: Reads the text, extracts data from tables, understands the pie chart showing revenue distribution, and even reads scanned signatures. A proper CA who sees the full picture.

Three Approaches to Multimodal RAG:

1. Extract and Convert: Convert images/tables to text descriptions, then use standard text RAG. Simplest but loses visual nuance.
2. Multimodal Embeddings: Use models like CLIP or Jina-CLIP to create joint embeddings for text and images in the same vector space.
3. Vision-Language Models: Pass raw images + text to models like GPT-4V or Claude Vision at generation time. Most powerful but expensive.

Use Cases:

• Healthcare: RAG over medical reports with X-rays and lab results
• Finance: Analyzing annual reports with charts, tables, and graphs
• E-commerce: Product search combining descriptions and product images
• Legal: Contract analysis with scanned documents, stamps, signatures
• Education: Textbook Q&A that understands diagrams and formulas

The Parsing Challenge:

PDFs are designed for display, not data extraction. A "simple" PDF might have text in random order, tables as positioned rectangles, images as embedded objects, and no semantic structure at all. Garbage in, garbage out - your RAG is only as good as your parser.

Parsing Approaches:

• Rule-Based (PyPDF, pdfplumber): Extract raw text + coordinates. Fast and cheap. Fails on complex layouts, scanned docs, and merged table cells.
• OCR-Based (Tesseract, Azure Document Intelligence): Converts images/scans to text. Necessary for scanned documents. Quality depends on scan quality.
• Layout-Aware (Unstructured.io, DocTR): Understands page layout - identifies headers, paragraphs, tables, images as separate elements. Best for structured extraction.
• Vision-Based (GPT-4V, Claude Vision): Send page screenshots to vision models. Most accurate for complex layouts. Expensive but handles anything.

Table Extraction Deep Dive:

Tables are the hardest part of document parsing. Key tools:

• Camelot/Tabula: Rule-based table detection from PDFs. Works well for simple tables with clear borders.
• Table Transformer (Microsoft): ML model that detects and extracts tables from document images. Handles complex layouts.
• Unstructured.io: Combines multiple strategies. Identifies table regions, extracts to HTML/markdown.

Best practice: Convert extracted tables to markdown format - LLMs understand markdown tables much better than raw CSV or HTML.

Indian Document Challenges:

• Multi-language: Hindi + English mixed documents (Aadhaar, PAN forms)
• Scanned quality: Government documents often poorly scanned
• Stamps and watermarks: Interfere with OCR accuracy
• Non-standard layouts: Each state has different certificate formats

Two Ways to Handle Images in RAG:

You can either convert images to text and use standard text retrieval, or embed images directly into a shared vector space with text. Each has trade-offs.

Approach 1: Image-to-Text (Captioning)

• Use vision models (GPT-4V, Claude Vision, LLaVA) to generate detailed text descriptions of images
• Store the text descriptions alongside original images
• Retrieve using standard text search on the descriptions
• Pros: Simple, works with existing text RAG pipeline
• Cons: Descriptions may miss details, lossy conversion

Approach 2: Multimodal Embeddings

• Use models like CLIP, Jina-CLIP, or ColPali to embed images and text into the same vector space
• A text query like "bar chart showing revenue growth" retrieves the actual chart image
• Pros: No information loss, true cross-modal retrieval
• Cons: Needs specialized embedding models, larger index

ColPali - The Game Changer:

ColPali (2024) is a vision-language model that directly embeds document page images for retrieval. Instead of parsing text from PDFs, you just screenshot each page and embed the image. It understands text, tables, charts, and layout all from the visual representation.

This eliminates the entire parsing pipeline - no OCR, no table extraction, no layout analysis. Just image in, embeddings out. Early benchmarks show it matches or beats traditional parsing + text embedding approaches.

Step-by-Step Pipeline:

1. Ingest: Accept PDFs, images, presentations. Use Unstructured.io or LlamaParse to extract text, tables, and images separately.
2. Process Tables: Convert to markdown format. Optionally generate text summaries of complex tables using an LLM.
3. Process Images: Generate detailed captions using GPT-4V. Store both the caption and a reference to the original image.
4. Chunk: Chunk text normally. Keep tables as single chunks (do not split a table). Store image captions as individual chunks.
5. Embed: Use text embeddings for text + table markdown + image captions. Optionally use CLIP for cross-modal image embeddings.
6. Store: Vector DB with metadata (source_type: text/table/image, page_number, document_id).
7. Retrieve: Hybrid search. Filter by source type if needed.
8. Generate: Pass retrieved text to LLM. For image chunks, include original image in the prompt (for vision-capable models).

Tool Stack Recommendations:

Production Tips:

• Always keep original images/tables linked to their text chunks - you may need to show them in the UI
• Use page-level metadata so users can verify sources
• Tables should be chunked as whole units - splitting a table destroys meaning
• For scanned docs, always run OCR quality checks before indexing

Challenge 1: Table Accuracy

Even the best parsers struggle with merged cells, multi-row headers, nested tables, and tables that span multiple pages. Always validate extracted tables against the original document. For critical applications, human review of extracted tables is still necessary.

Challenge 2: Cost

Vision model calls are expensive. GPT-4V costs roughly 10x more than GPT-4 for the same content. If you have 10,000 pages with 3 images each, captioning alone can cost hundreds of dollars. Budget carefully and cache aggressively.

Challenge 3: Latency

Multimodal pipelines are slower - parsing a PDF takes seconds, vision model calls take 2-5 seconds per image. For real-time use cases, pre-process everything at ingest time, not at query time.

Challenge 4: Evaluation

How do you evaluate if an image was correctly retrieved? Standard text metrics do not apply. You need specialized evaluation frameworks that can handle multimodal ground truth data. This is an active research area.

Q: What are the three main approaches to handling images in a RAG pipeline?

Q: How would you handle tables in a multimodal RAG pipeline?

Q: What is ColPali and how does it change the multimodal RAG paradigm?