Katkat
Back

Design Hebbia Chat for SEC Filings

HebbiaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

This problem tests your ability to design a retrieval-augmented generation (RAG) system specialized for financial documents, focusing on multi-agent orchestration, citations, and latency management.

Design an Intelligent SEC Filing Assistant

Hebbia enables users to query massive document sets (e.g., 10-K filings, 10-Qs) to generate complex, cited answers. Design a system that receives a natural language query and returns a response grounded in specific SEC filing sections, while maintaining high accuracy in citations.

Requirements

Functional Requirements:

  • Multi-Document Reasoning: The system must synthesize information across multiple filings (e.g., "Compare the risk factors of Company A and Company B for FY2023").
  • Citations: Every claim in the generated response must link back to a specific page or paragraph in the source filing.
  • Agentic Flow: Use an agent to decide if the query requires a multi-step search or can be answered directly.

Non-Functional Requirements:

  • Scale: Ability to index and search millions of pages of SEC filings.
  • Latency: Provide streaming responses to the user to minimize perceived wait time.
  • Reliability: Prevent hallucination by enforcing a strict context-only constraint.

Deliverables

  1. High-level Architecture: Describe the ingestion pipeline, the retrieval mechanism, and the agent orchestration flow.
  2. Data Model: Define the structure for storing parsed SEC filings and their vector embeddings (including metadata for citations).
  3. Latency Mitigation: Propose how to handle long-running document analysis tasks without timing out the user-facing request.
  4. Trade-offs: Discuss how you would balance retrieval precision versus recall when dealing with financial jargon.

Hints

Hint 1

Consider using a hierarchical retrieval approach: retrieve relevant documents first, then chunk and retrieve specific paragraphs, then re-rank.

Hint 2

Think about how to store and retrieve metadata (like document ID and page number) to ensure the LLM can generate exact citations.

Hint 3

Consider a 'Router' agent that decides whether a query needs a simple search or an analytical 'Map-Reduce' style document comparison.

Follow-up questions

  • How would you handle a scenario where a user query is extremely broad, requiring the retrieval of thousands of document chunks?
  • If the model continues to hallucinate citations, what specific changes would you make to the retrieval or prompting strategy?
  • How would you measure the performance of this system beyond standard token-based metrics (e.g., RAGAS, citation accuracy)?