Katkat
Back

Design a scalable chatbot platform

AtlassianMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteSystem Design

Last updated

This problem tests your ability to architect a scalable, stateful chatbot system that integrates RAG (Retrieval-Augmented Generation) with low-latency conversational requirements.

Problem: Designing an Enterprise RAG-Enabled Chatbot

Design a scalable chatbot platform for internal use that allows employees to query internal documentation (e.g., Confluence, Jira tickets). The system must support multi-turn conversations, retrieve relevant context from unstructured data, and handle sudden spikes in traffic during company-wide events.

Functional Requirements

  1. Users can ask natural language questions and receive context-aware responses.
  2. The system must maintain conversation state (context) for at least 10 minutes.
  3. The platform must support a 'human handoff' feature if the ML model's confidence is below a specific threshold.
  4. Responses must be grounded in verified internal documentation.

Non-Functional Requirements

  1. Latency: Response time should be under 2 seconds for a retrieval-augmented query.
  2. Scale: Support 10,000 concurrent users across different time zones.
  3. Accuracy: Minimize hallucinations by ensuring the model only answers based on provided documents.
  4. Security: Access control is mandatory; users should only receive results from documents they are authorized to view.

Your Task

Provide a high-level system architecture design. Include:

  • A component diagram covering ingestion (ETL), retrieval, and inference.
  • The schema or strategy for storing conversation context and document embeddings.
  • A strategy for how you would handle access control within the retrieval pipeline.
  • A plan for how to monitor the system for 'drift' or poor performance after deployment.

Hints

Hint 1

Consider using a vector database for document embeddings and a fast key-value store (like Redis) for managing conversation session state.

Hint 2

Address the security requirement by pre-filtering retrieval results based on user metadata or implementing a 'Permissions-aware' vector search.

Hint 3

Think about how the 'human handoff' logic integrates into the request-response lifecycle—do you route to a queue or a separate live-chat service?

Follow-up questions

  • How would you implement incremental updates to the knowledge base so that newly published documents are searchable within seconds?
  • What specific metrics would you track in a dashboard to detect if the LLM is providing irrelevant or hallucinated answers?
  • If the inference latency is too high, what caching strategies would you implement for frequently asked questions or common query vectors?