Katkat

Implement Stable Softmax

XPengSoftware Engineer · Technical Screen

Last updated

This question tests the ability to implement a numerically stable softmax function by handling potential floating-point overflows or underflows common in high-dimensional vector exponentiation.

Problem Statement

The softmax function is a critical component in machine learning, mapping a vector of real numbers to a probability distribution. Given a vector logits of size N, the softmax function is defined as P(i) = exp(logits[i]) / sum(exp(logits[j])) for all j in 0 to N-1.

A naive implementation of softmax is prone to numerical instability. Specifically, exp(x) grows extremely rapidly, and for large values of logits[i], the calculation can easily result in an arithmetic overflow (inf), leading to NaN outputs.

Implement the stable_softmax function. You must ensure the result remains accurate even when input values are very large.

Function Signature

def stable_softmax(logits: list[float]) -> list[float]:
    # Implementation here

Constraints

  • 1 <= len(logits) <= 10^5
  • -10^4 <= logits[i] <= 10^4

Example 1

Input: [1.0, 2.0, 3.0] Output: [0.090, 0.245, 0.665]

Example 2

Input: [1000.0, 1000.0] Output: [0.5, 0.5]

Note: A naive implementation using exp(1000.0) will return NaN due to overflow, but the stable version should correctly return [0.5, 0.5].

Hints

Hint 1

Consider how the softmax result changes if you subtract a constant `C` from every element in the input vector.

Hint 2

Mathematical identity: `exp(x - C) / sum(exp(x_i - C))` is equivalent to `exp(x) / sum(exp(x_i))`.

Hint 3

If you choose `C` to be the maximum value in the input vector, what is the largest possible value passed to the `exp()` function?

Follow-up questions

  • How would you modify your implementation to handle a 2D matrix (batch of vectors) instead of a 1D list?
  • What are the computational complexity implications of your chosen approach?
  • How does floating-point precision (e.g., float32 vs float64) impact the accuracy of the softmax calculation?