Katkat
Back

Interpret Language-Model Perplexity

GoogleMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Learn the exact likelihood basis of perplexity, correct evaluation details, tokenizer caveats, and limits as an LLM quality metric.

Interpret Language-Model Perplexity

Define perplexity for an autoregressive language model, derive its relationship to average negative log-likelihood, and explain what it does and does not reveal about model quality.

Constraints & Assumptions

  • Evaluate on a fixed tokenized dataset with a stated logarithm base.
  • Mask padding and any tokens that are not prediction targets.
  • Perplexities from different tokenizers are not directly comparable.

Clarifying Questions to Ask

  • Is loss averaged per token, sequence, or byte?
  • Was the evaluation set seen during training?
  • How are long contexts and truncation handled?

Hints

Hint 1

Start from likelihood: Express perplexity as the exponential of mean token cross-entropy before discussing intuition.

Follow-up questions

  • Why can lower perplexity fail to improve a user's task?
  • How would byte-level normalization help compare different tokenizers?