Design a chatbot fallback for unknown questions
OpenAIMachine Learning Engineer · Onsite
Last updated
This question evaluates a candidate's competency in ML system design, focusing on handling model uncertainty, retrieval-augmented components, tool integration, decision policies, and operational evaluation within the ML System Design domain.
Scenario
You run a ChatGPT-like assistant. Users sometimes ask questions the model cannot answer reliably (unknown/uncertain/needs up-to-date facts).
Task
Design a system that improves user experience and answer quality when the assistant "doesn't know".
Requirements
- The assistant should avoid hallucinations and be transparent about uncertainty.
- It should still help the user make progress (clarify, retrieve sources, route to tools, etc.).
- Support both factual queries (need citations) and personal/workflow queries (need clarification).
- Low latency for common queries; allow slower tool usage when needed.
Deliverables
- High-level architecture (LLM, retrieval, tools, policies).
- Decision policy for: answer directly vs. ask clarifying questions vs. retrieve vs. refuse.
- How you evaluate success (offline and online metrics) and handle failure modes.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.