Design Retrieval-Augmented Answers from a Document Vault

Product Scenario

Enterprise organizations store sensitive legal agreements, corporate filings, and internal memos inside a document vault. Users need a natural language question-answering service that produces accurate, fully grounded answers backed by precise citations (document, section, and page coordinates). Because the vault contains confidential and privileged records, the system must enforce strict role-based access control (RBAC), prevent data leakage between tenants and unauthorized internal users, handle document versioning (e.g., amendments overriding master agreements), and safely refuse to answer when evidence is missing or ambiguous.

Requirements

Functional Requirements

1. Versioned Ingestion & Parsing: Ingest multi-format documents (PDF, DOCX, scans), extract layout structure, and maintain parent-child chunk hierarchies tied to document versions.
2. Grounded QA with Exact Provenance: Produce answers with verified citations mapping directly to exact text snippets, page numbers, and bounding boxes.
3. Granular Access Control: Filter results according to tenant isolation boundaries and document-level access control lists (ACLs).
4. Document Freshness & Deletion: Support soft/hard deletions (GDPR/compliance) and near real-time updates without stale retrieval.
5. Contradiction & Abstention: Detect superseded clauses or conflicting documents, presenting timelines or abstaining when retrieval yields insufficient evidence.
6. Safety & Security: Mitigate indirect prompt injection attacks embedded within vault documents.

Non-Functional Requirements & Scale

• Scale: 500 million chunks across 1,000 tenants; largest tenant has 50 million chunks.
• Latency: End-to-end P95 query latency under 2.5 seconds (retrieval <= 300 ms, generation streaming starts <= 800 ms).
• Freshness: Ingested updates or ACL updates must reflect in retrieval within 60 seconds.
• Reliability & Accuracy: Strict zero-tolerance for cross-tenant leakage; hallucination rate below 1% on grounded benchmarks.

What Your Design Should Cover

• End-to-End Architecture: Ingestion pipeline, indexing layer, retrieval/reranking service, and generation service.
• Data Model & Search Strategy: Choice between dense, sparse, and hybrid retrieval; handling chunking with structural context; ACL enforcement strategy (pre-filtering vs. post-filtering).
• Provenance & Grounding: How the model outputs citations and how those citations are verified against source spans.
• Conflict Resolution: How to resolve temporal conflicts (e.g., an amendment dated 2024 modifying a clause from 2021).
• Evaluation: Offline and online evaluation frameworks to measure retrieval recall, citation precision, faithfulness, and injection resistance.