Katkat
Back

Design an ML search system with RAG

OpenAIMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenML System Design

Last updated

Design an ML search system with RAG evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design an ML-powered enterprise search system using Retrieval-Augmented Generation (RAG). Context and constraints:

  • Corpus: 5M documents (avg 2 KB) from PDFs/web pages/tickets; updates must be searchable within 5 minutes.
  • Traffic: 300 QPS; multi-tenant with per-document ACLs.
  • SLOs: p95 latency ≤ 1.2 s end-to-end; budget ≤ $0.002 per query. Sub-questions: (a) Ingestion and chunking: parsing, deduplication, metadata extraction, embedding generation, chunk-size strategy, versioning, and incremental updates. (b) Indexing and retrieval: hybrid sparse+vector (BM25 + ANN), metadata filters, tenant isolation, query understanding/reformulation, top-k selection, and cross-encoder reranking. (c) Generation: prompt design, grounding with citations, constrained decoding, tool usage, streaming responses, and multilingual handling. (d) Guardrails and safety: hallucination reduction, citation enforcement, out-of-policy refusal, PII/security controls, and ACL-aware retrieval. (e) Evaluation and monitoring: offline metrics (NDCG@k, recall@k, answer faithfulness), online A/B tests, user feedback loops, drift/latency/cost monitoring. (f) Architecture and scaling: service decomposition, model hosting/batching, caching, vector store selection, backpressure, failover, and disaster recovery. (g) Cost and latency calculations: derive per-stage latency/cost, capacity plan for embeddings, ANN index size, and compute requirements. Justify model choices under constraints.