Katkat
Back

Build a Small Agent or RAG Tool with the Mistral API

Mistral AIMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

Design an end-to-end Retrieval-Augmented Generation (RAG) agent using the Mistral API to answer technical queries over internal developer documentation. The question evaluates chunking strategies, dense retrieval, prompt orchestration, latency optimizations, and evaluation methodologies.

Scenario

Your engineering organization wants an internal technical assistant that answers developer questions by querying a continuously updated knowledge base of API specs, architectural decision records (ADRs), and internal runbooks. The system must use Mistral's model suite (e.g., mistral-embed for vector representations and mistral-small or mistral-large for generation/orchestration).

Design an end-to-end RAG-based agent that retrieves relevant documentation, decides whether to invoke additional internal lookup tools, and generates accurate, grounded responses with inline citations.

Requirements

Functional Requirements

  • Ingest internal Markdown/HTML documentation, extract metadata (author, service, update timestamp), and maintain an up-to-date vector index.
  • Support single-turn and multi-turn developer queries with source attribution.
  • Implement agentic tool use: the agent must query documentation first, but can call a live service health check API if the user asks about current production status.
  • Reject out-of-domain questions and acknowledge when internal documents do not contain the answer (hallucination mitigation).

Non-Functional Requirements & Scale

  • Document corpus: 500,000 documents, averaging 1,500 tokens per document (~750M tokens total), with ~5,000 document updates per day.
  • Query traffic: 20 queries per second (QPS) peak during developer working hours.
  • Latency: End-to-end P95 latency under 1.5 seconds for standard RAG queries.
  • Availability: 99.9% uptime.

What Your Design Should Cover

  1. Data Ingestion & Chunking: Chunking strategy (fixed vs. semantic vs. hierarchy-aware), handling code snippets alongside text, and incremental index updates.
  2. Retrieval & Reranking Architecture: Dense search using mistral-embed, hybrid search (sparse BM25 + dense), and multi-stage reranking.
  3. Agent & Orchestration Flow: Routing logic (when to search docs vs. call live status API vs. direct answer), prompt template structure, context window budgeting, and streaming output.
  4. Latency & Cost Optimization: Caching strategies (semantic cache, prompt caching), choice of generator model (mistral-small vs. mistral-large), and concurrency.
  5. Evaluation & Guardrails: Offline evaluation metrics (RAG triad: context relevance, groundedness, answer relevance) and guardrails against prompt injection from indexed documents.

Hints

Hint 1

Consider how code blocks and markdown headings impact chunk boundaries to avoid severing code context from its explanatory text.

Hint 2

A hybrid retrieval approach combining dense vectors from mistral-embed with sparse lexical search (BM25) helps capture exact error codes and service names that embeddings often blur.

Hint 3

Evaluate using a smaller, faster model (e.g., mistral-small) for tool routing and verification checks, reserving mistral-large for final synthesis only when high reasoning capacity is required.

Follow-up questions

  • How would you handle document updates and deletions in the vector store to prevent stale or hallucinated answers without re-indexing the entire corpus?
  • If users complain that the assistant retrieves outdated runbooks instead of newly published ones, how would you incorporate temporal decay or versioning into the retrieval score?
  • How would you protect the system against indirect prompt injection if an internal user embeds malicious instructions into a public README?