Implement Stable Softmax
XPengSoftware Engineer · Technical ScreenLast updated
This question tests the ability to implement a numerically stable softmax function by handling potential floating-point overflows or underflows common in high-dimensional vector exponentiation.
Problem Statement
The softmax function is a critical component in machine learning, mapping a vector of real numbers to a probability distribution. Given a vector logits of size N, the softmax function is defined as P(i) = exp(logits[i]) / sum(exp(logits[j])) for all j in 0 to N-1.
A naive implementation of softmax is prone to numerical instability. Specifically, exp(x) grows extremely rapidly, and for large values of logits[i], the calculation can easily result in an arithmetic overflow (inf), leading to NaN outputs.
Implement the stable_softmax function. You must ensure the result remains accurate even when input values are very large.
Function Signature
def stable_softmax(logits: list[float]) -> list[float]:
# Implementation here
Constraints
1 <= len(logits) <= 10^5-10^4 <= logits[i] <= 10^4
Example 1
Input: [1.0, 2.0, 3.0]
Output: [0.090, 0.245, 0.665]
Example 2
Input: [1000.0, 1000.0]
Output: [0.5, 0.5]
Note: A naive implementation using exp(1000.0) will return NaN due to overflow, but the stable version should correctly return [0.5, 0.5].
Hints
Hint 1
Consider how the softmax result changes if you subtract a constant `C` from every element in the input vector.
Hint 2
Mathematical identity: `exp(x - C) / sum(exp(x_i - C))` is equivalent to `exp(x) / sum(exp(x_i))`.
Hint 3
If you choose `C` to be the maximum value in the input vector, what is the largest possible value passed to the `exp()` function?
Follow-up questions
- How would you modify your implementation to handle a 2D matrix (batch of vectors) instead of a 1D list?
- What are the computational complexity implications of your chosen approach?
- How does floating-point precision (e.g., float32 vs float64) impact the accuracy of the softmax calculation?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.