Katkat
Back

Design a Multi-Agent Document Search and Visualization Product

HebbiaMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteML System Design

Last updated

This problem tests your ability to design a complex multi-agent orchestration system that integrates large-scale document retrieval with reliable citation tracking and a user-facing visualization interface.

System Design: Multi-Agent Document Search & Visualization

Design a system that allows users to query a large corpus of unstructured documents (PDFs, reports, spreadsheets) to generate a high-level summary supported by interactive data visualizations (e.g., charts, trend lines, or relationship graphs) derived directly from the source evidence.

Requirements

Functional:

  1. Query Planning: Decompose natural language prompts into sub-tasks (e.g., "find revenue trends," "compare competitor pricing").
  2. Tool Orchestration: Enable agents to call search engines, extract tables, and perform code-based analysis (e.g., pandas/Python) for visualization.
  3. Citations: Every claim in the final output must be linked back to a specific document section or a specific chart data point.
  4. Visualization: Produce valid JSON or specifications for a frontend component to render charts.

Non-Functional:

  1. Latency: Initial answers should be streamed to the user within seconds.
  2. Accuracy: Minimize hallucinations and ensure citations are verifiable.
  3. Scalability: The system should handle thousands of concurrent queries across millions of documents.

Deliverables

  1. High-Level Architecture: Describe the interaction between a 'Planner' agent, 'Worker' agents, and the tool-use execution environment.
  2. Citation Mechanism: Explain how you maintain grounding throughout the multi-step transformation process (from raw text to a visualized chart).
  3. Failure Handling: How does the system detect and recover from tool execution errors (e.g., a failed Python snippet) or agent circular reasoning?
  4. Evaluation: Define metrics to measure the quality of both the textual answer and the accuracy of the visualizations provided.

Hints

Hint 1

Consider implementing a 'ReAct' or 'Plan-and-Solve' framework to help agents break down complex user requests.

Hint 2

Use a structured logging or memory buffer approach to track provenance metadata at every step of the agent pipeline.

Hint 3

Think about the trade-offs between using a single large LLM for everything versus using smaller, fine-tuned models for specific tasks like data extraction or code generation.

Follow-up questions

  • How would you handle a scenario where the user's request requires information that is currently missing from the document corpus?
  • If the visualization component produces an incorrect chart, how can we leverage the LLM to perform self-correction or verification?
  • How would you evolve the system if the documents were dynamic, such as live RSS feeds or streaming financial tickers?