Design a RAG-Based Agent System
ByteDanceSoftware Engineer · Technical ScreenLast updated
This question tests the ability to architect a scalable Retrieval-Augmented Generation (RAG) system that handles document indexing, retrieval, and multi-step reasoning while maintaining low latency and data freshness.
Scenario
You are tasked with designing a RAG-based support agent for ByteDance's internal developer platform. The system must answer technical questions by querying thousands of internal documentation pages, JIRA tickets, and Slack archives.
Requirements
-
Functional:
- Users ask natural language questions.
- The agent should perform multi-step reasoning (e.g., retrieving data from multiple sources before synthesizing an answer).
- The system must respect document access permissions based on user groups.
-
Non-Functional:
- Latency: The time-to-first-token should be under 1 second.
- Freshness: Documentation updates must be searchable within 5 minutes.
- Scale: The system must handle 500 requests per second (RPS) with varying document lengths.
Your Design Should Cover:
- Data Pipeline: How documents are ingested, chunked, and embedded.
- Retrieval Strategy: How you handle hybrid search (vector similarity + keyword search) and manage access control lists (ACLs).
- Agent Orchestration: How the system determines when to fetch new data vs. when to synthesize an answer.
- Evaluation: How you measure the quality of retrieval and the accuracy of the generated responses.
Hints
Hint 1
Consider how to integrate ACL metadata into the vector search process, either through pre-filtering or post-filtering.
Hint 2
For freshness, think about using a streaming pipeline (e.g., Kafka) to trigger incremental updates to your vector store.
Hint 3
To achieve low latency, evaluate the trade-offs between using a single large prompt versus a chain-of-thought agentic approach.
Follow-up questions
- How would you handle cases where the retrieved context contains contradictory information?
- What specific metrics would you monitor in production to detect model drift or degradation in retrieval quality?
- How would you optimize the system if your primary vector database became a bottleneck under high concurrent load?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.