Detect Harmful Content in LLM Prompts and Responses
DatabricksMachine Learning Engineer · Onsite
Last updated
Design a machine learning system that detects harmful content in prompts sent to, and responses produced by, a large language model. Tests policy-driven framing, labeling and adversarial data, layered classifiers under latency limits, per-category evaluation, streaming checks and monitoring.
Design a machine learning system that detects harmful content submitted to, or produced by, a large language model (LLM) product.
The report records only this prompt from an ML design round. The scope, the harm categories and the product constraints are yours to clarify and state.
Clarifying Questions
- Should the system screen user prompts, model responses, or both?
- Which harm categories are in scope, and is there a written policy defining each one?
- What happens on a detection: block, warn, rewrite the response, or route to human review?
- How much latency can each request absorb, and are responses streamed token by token?
- Which languages and modalities (text only, code, images) must be covered?
Hints
Hint 1
Where does the check sit?: List the points in a chat request's lifetime where content could be checked, and what each point can and cannot see.
Hint 2
Not all mistakes cost the same: Think separately about the cost of a missed harmful response and the cost of refusing a benign one, and how each should shape the per-category thresholds.
Follow-up questions
- How would you detect a harmful request split across many innocuous-looking turns of a conversation?
- How would you keep the classifier robust to jailbreak prompts that change quickly?
- How would you measure recall in production, when you cannot see most of the harmful content you missed?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.