Katkat
Back

Evaluate RAG System Accuracy and Cost Control Strategies

AmazonData Scientist · Technical Screen
HardData ScientistTechnical ScreenMachine Learning

Last updated

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate RAG System Accuracy and Cost Control Strategies states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Deep-learning discussion on LLM pipelines, knowledge-graph integration and retrieval-augmented generation.

Question

How would you control the cost of maintaining a knowledge graph used by an LLM? How do you measure the accuracy of LLM outputs, both offline and online? Compare Transformer, RNN and LSTM. Why are Transformers preferred for modern LLMs? Derive the scaled dot-product attention formula and explain each term. Explain the end-to-end workflow of Retrieval-Augmented Generation (RAG). What is a reranker model and where does it sit in the RAG stack? How does embedding vector dimensionality influence retrieval quality? What is LoRA, how does it work and why is it parameter-efficient? How would you evaluate the accuracy of a RAG system?

Hints

Relate theory to production: costs, equations, eval metrics (BLEU, EM, precision@k), trade-offs.