Katkat
Back

Design an LLM-based binary classifier

AnthropicSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

This question evaluates knowledge of LLM-based binary classification, probabilistic scoring and numerical stability (including log-prob handling and log-sum-exp), prompt construction, batching and sequence-level aggregation, calibration and evaluation metrics, plus analysis of failure modes and inference-time complexity.

You are given a helper function score_batch(prompt, inputs) that, for each input string, returns per-token probabilities (possibly as log-probabilities) from a large language model under the provided system prompt. Design a binary text classifier using only this helper: (

  1. Construct system prompts for classes A and B. (
  2. For each input, query the helper with both prompts and produce a continuous score s in [0,1] estimating P(class=A|x), not just a hard label. (
  3. If the helper returns log-probabilities, show how to compute a numerically stable score (e.g., via log-sum-exp and log-prob normalization). (
  4. Define batching and how token-level probabilities are aggregated into a single sequence-level score. (
  5. Describe threshold selection and evaluation metrics (ROC-AUC, precision/recall, F 1), including calibration. (
  6. Propose performance improvements: prompt engineering, repeated sampling, temperature control, prompt ensembling, score calibration, and handling class imbalance. (
  7. Discuss failure modes and how to validate/iterate offline without fine-tuning. Provide pseudocode and the inference-time complexity.