Katkat
Back

Design a Multimodal Retrieval-Augmented Generation System

J.P. MorganSoftware Engineer II · Onsite
MediumSoftware Engineer IIOnsiteML System Design

Last updated

This problem tests the ability to design an end-to-end RAG system capable of processing both text and image inputs while ensuring high retrieval precision, security, and traceability.

Problem: Design a Multimodal RAG System

Design a Retrieval-Augmented Generation (RAG) system for a financial research platform that allows analysts to query both internal PDF documents (containing text and charts) and raw market data images.

Requirements:

  1. Ingestion & Provenance: Documents are processed and stored such that every retrieved snippet or image can be traced back to its specific source, page, and version.
  2. Multimodal Retrieval: The system must support queries across text-heavy documents and visual data (charts/graphs).
  3. Hybrid Search: Implement a retrieval strategy that combines vector search (semantic similarity) with keyword-based search (BM25) to improve precision for specific financial terminology or ticker symbols.
  4. Reranking: Incorporate a cross-encoder reranking step to refine the retrieval results before passing them to the LLM.
  5. Security & Abstention: Ensure the system does not leak PII, and implement an 'abstention' mechanism where the system returns 'I don't know' if the retrieval confidence score falls below a predefined threshold.

Deliverables:

  • Architecture Diagram: Describe the flow from document ingestion to the final generated response.
  • Data Model: Define the schema for storing indexed multimodal chunks.
  • Evaluation Strategy: Explain how you would separately evaluate the quality of the retrieval phase (Recall@K) and the final answer generation (Faithfulness/Hallucination).

Hints

Hint 1

Consider using separate embedding models for text (e.g., CLIP or BERT-based) and visual data, then mapping them into a shared embedding space.

Hint 2

For provenance, attach metadata (source_id, timestamp, page_num) to every embedding vector in your vector database.

Hint 3

Use a 'confidence threshold' logic at the reranking stage: if the top reranked document has a score below a certain value, trigger the abstention protocol.

Follow-up questions

  • How would you handle document updates to ensure that the retrieved content is always the most current version?
  • What specific metrics would you monitor in production to detect model drift or degradation in retrieval accuracy?
  • If the latency is too high, what specific components of this pipeline would you optimize first and how?