Katkat
Back

Reduce LLM hallucination and handle class imbalance

CVS HealthData Scientist · Technical Screen
EasyData ScientistTechnical ScreenMachine Learning

Last updated

This question tests your ability to design a robust pipeline for detecting clinical inaccuracies in LLM outputs while managing highly imbalanced datasets where missing a critical diagnosis is costlier than a false alarm.

Scenario

You are building a clinical summarization tool for CVS Health that processes patient notes using an LLM. You face two major challenges:

  1. Hallucination: The model occasionally invents symptoms or medications not present in the source notes.
  2. Class Imbalance: You are training a binary classifier to flag summaries as 'Potentially Hallucinated' or 'Safe'. However, hallucinations occur in less than 1% of generated summaries.

The Task

  1. Mitigation Strategy: Propose a workflow to minimize hallucinations using a Retrieval-Augmented Generation (RAG) framework. How would you structure the prompt and verification step to ensure citations to source documents?
  2. Modeling Approach: Given the 1% hallucination rate, explain your strategy for training a classifier to detect these errors. Specifically, address how you would handle the extreme class imbalance and which evaluation metrics you would prioritize (Precision vs. Recall) and why.
  3. Trade-off Analysis: Discuss how you would balance the 'token cost' of verifying every summary against the 'safety cost' of letting a potential hallucination pass through to a clinician.

Hints

Hint 1

Consider implementing a 'Self-Correction' loop where the LLM is asked to verify its own output against a provided context window before final presentation.

Hint 2

For the imbalanced classifier, think about techniques like SMOTE (Synthetic Minority Over-sampling Technique), adjusting loss function weights, or using cost-sensitive learning to penalize false negatives more heavily.

Hint 3

Prioritize Recall over Precision because in a clinical setting, missing a 'hallucinated' diagnosis carries significantly higher risk than a false flag that requires a manual human check.

Follow-up questions

  • How would you measure the 'ground truth' for hallucinations if you do not have a pre-existing labeled dataset?
  • If your model detects a high probability of hallucination, what is the best UX design: blocking the output, highlighting the risky text, or forcing a human-in-the-loop review?
  • How does the choice of embedding model or retrieval technique (e.g., semantic search vs. keyword search) impact the hallucination rate?