Katkat
Back

Evaluate a Retrieval-Augmented Generation System

C3 AIData Scientist · Technical Screen
HardData ScientistTechnical ScreenML System Design

Last updated

Design an end-to-end evaluation framework for an enterprise RAG system that assesses retrieval quality, generative fidelity, claim-level attribution, security controls, and production telemetry.

Scenario

An enterprise AI platform is deploying a conversational assistant that answers employee queries across proprietary data stores (technical specifications, financial reports, HR policies, and customer contracts). Because business decisions rely on these answers, hallucinations, stale data, and access violations carry severe consequences.

You are tasked with designing a comprehensive evaluation framework to benchmark, monitor, and continuously improve this Retrieval-Augmented Generation (RAG) system.

Scale and Assumptions

  • Corpus: 10 million documents across multiple access tiers, with daily incremental updates.
  • Workload: 50 queries per second (QPS) at peak; target p95 end-to-end latency < 2.0 seconds.
  • Cost ceiling: Average inference cost < $0.02 per query.
  • Ground truth: 5,000 curated gold-standard query-context-answer triples available for regression testing, with the ability to generate synthetic evaluation sets.

Requirements

Your design should address:

  1. Component-Level Evaluation:

    • Retrieval: Metrics to evaluate dense and hybrid retrievers (e.g., Recall@K, MRR, NDCG, contextual relevance) independent of generation.
    • Generation: Metrics for groundedness (faithfulness to retrieved context), answer relevance, and completeness.
    • Claim-Level Attribution: Verifying that inline citations accurately support the specific factual claims made.

  2. Enterprise Constraints & Safety:

    • Abstention & Negative Testing: Evaluating when the model should refuse to answer (e.g., out-of-domain queries, insufficient evidence, conflicting sources).
    • Access Control & Permissions: Testing that users cannot extract unauthorized context (e.g., RBAC/ABAC leakage).
    • Freshness & Version Drift: Detecting degradation when document versions change or contradict older snapshots.

  3. Online Telemetry & Production Monitoring:

    • Implicit signals (clicks on sources, follow-up query sentiment, dwell time) and explicit signals (thumbs up/down, user corrections).
    • Production guardrails: Latency tracking, token budget monitoring, and automated drift/hallucination sampling.

What to Deliver

  • A structured architectural diagram or flow of the offline evaluation harness versus online production telemetry.
  • Methodology for creating and maintaining the evaluation dataset (synthetic vs. human-annotated).
  • Metric definitions, judge models (LLM-as-a-judge calibration), and deterministic validation checks.
  • Trade-offs between evaluation depth, latency, and cost in CI/CD pipelines.

Hints

Hint 1

Decouple retrieval quality from generation quality so you can isolate whether an incorrect answer stemmed from missing context or an LLM hallucination.

Hint 2

For claim-level citation validation, consider breaking generated answers into atomic factual claims and evaluating NLI (Natural Language Inference) entailment against cited spans.

Hint 3

Incorporate adversarial negative test sets to systematically measure precision of abstention and access-control boundaries.

Follow-up questions

  • How do you quantify and mitigate judge-model biases (such as position bias, verbosity bias, and self-preference) when using LLM-as-a-judge for groundedness?
  • How would you evaluate the system's ability to handle temporal reasoning and conflicting information across two document versions published in different years?
  • If your CI/CD test suite costs $500 per run using full LLM judge evaluations, how would you design a tiered evaluation strategy to reduce costs while maintaining high regression coverage?