Katkat
Back

Explain surprisal and its units

AmazonApplied Scientist · Technical Screen
MediumApplied ScientistTechnical ScreenMachine Learning

Last updated

This question evaluates understanding of probabilistic information measures in language modeling, specifically the concept of surprisal, its units, and its relationship to evaluation metrics such as cross-entropy and perplexity within the Machine Learning/NLP domain.

You are discussing a language-modeling / NLP project. The interviewer asks about surprisal.

  1. Define surprisal for an event/token with probability (p).
  2. What are the possible units of surprisal (analogous to meters for distance), and what determines the unit?
  3. How is surprisal related to cross-entropy and perplexity in language modeling?
  4. What common pitfalls or misunderstandings arise when interpreting surprisal values?