Katkat
Back

Design an Agent for Internal Document Search

GoogleSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

Design an enterprise search agent that provides grounded, cited answers over internal knowledge repositories while strictly enforcing access control lists (ACLs) and handling prompt injection.

Scenario

Large enterprises maintain millions of internal documents across wikis, cloud drives, ticket trackers, and code repositories. Employees struggle to find authoritative information spread across disjointed tools.

Design an intelligent, internal document-search agent that answers natural language questions by synthesizing content across internal documents. The system must provide inline citations, gracefully abstain when documents lack sufficient evidence, and strictly enforce document access controls.

Requirements

Functional Requirements

  • Permission-Aware Search: Users must only receive answers and citations derived from documents they are explicitly authorized to read. Revoked permissions must take effect immediately or near-instantaneously.
  • Grounded Synthesis & Citations: Generated answers must be strictly grounded in retrieved documents, featuring exact inline citations linked to source passages.
  • Calibrated Abstention: If no relevant documents exist or the retrieved context is conflicting/insufficient, the agent must decline to answer rather than hallucinate.
  • Adversarial Robustness: Defend against indirect prompt injections embedded in internal documents designed to exfiltrate restricted data or hijack instructions.

Non-Functional Requirements

  • Latency: Time-to-first-token under 800 ms; full response delivered within 2.5 seconds (p95).
  • Freshness: Newly indexed or modified documents must be searchable within 5 minutes.
  • Availability: 99.9% uptime during standard enterprise working hours.

Scale & Constraints

  • Corpus Size: 50 million documents (~500 GB to 1 TB text), growing by 100,000 documents/day.
  • User Base: 100,000 active employees generating peak query loads of 100 QPS.
  • Access Control Complexity: Documents inherit permissions from dynamic hierarchical groups and individual user ACLs.

Key Design Areas to Cover

  1. Data Ingestion & Permission Indexing: Chunking strategy, hybrid retrieval (dense embeddings + BM25), and whether to store ACLs in the vector index or enforce them via post-retrieval filters.
  2. Retrieval & Reranking Architecture: Multi-stage retrieval, cross-encoder reranking, and dynamic context-window assembly.
  3. Grounded Generation & Abstention: LLM orchestration, citation attribution mechanisms, and confidence scoring to trigger abstention.
  4. Security & Governance: Defenses against indirect prompt injections and data leaks across user privilege tiers.
  5. Evaluation & Observability: Offline benchmarking datasets, metrics for faithfulness and retrieval recall, and production monitoring.

Hints

Hint 1

Evaluate the trade-offs between pre-filtering (filtering by user ACLs directly during vector search) versus post-filtering retrieved candidates against an enterprise authorization service.

Hint 2

For verifiable citations, consider a two-step generation process: generating an initial synthesis followed by an attribution verification pass that aligns each statement to a retrieved passage.

Hint 3

Address indirect prompt injection by isolating untrusted document content in structured data fields and using a separate validation model before final rendering.

Follow-up questions

  • How would you handle document permission changes (e.g., removing a user group from a folder) without triggering expensive vector re-indexing for millions of chunks?
  • How do you design a semantic caching layer that avoids leaking confidential information across users with different access levels?
  • What metrics and automated test suites would you deploy to continuously measure hallucination, citation precision, and retrieval recall?