Katkat
Back

Explain the Bellman Equation

VoleonMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Understand Bellman expectation and optimality equations through one-step return decomposition, fixed points, and reinforcement-learning algorithms.

Explain the Bellman Equation

State and explain the Bellman expectation and optimality equations. Show how they decompose value into immediate reward and discounted future value, and connect them to policy evaluation and control.

Constraints & Assumptions

  • Use a finite or discounted Markov decision process.
  • State whether reward is attached to the transition or current state-action pair.
  • Distinguish value under a fixed policy from optimal value.

Clarifying Questions to Ask

  • Are you writing a state-value or action-value equation?
  • Is the policy fixed?
  • Are transition and reward expectations known or sampled?

Hints

Hint 1

Condition on one step: Expand return into the next reward plus the discounted remaining return, then take the appropriate expectation or maximum.

Follow-up questions

  • Why does replacing expectation with a maximum change the problem?
  • What breaks when undiscounted continuing rewards are unbounded?