Which method updates estimates step-by-step while the episode is in progress, using bootstrap estimates?

Which method updates estimates step-by-step while the episode is in progress, using bootstrap estimates?

• A. Monte Carlo method
• B. Temporal Difference (TD) method
• C. Q-Learning
• D. Deep Reinforcement Learning

Answer: (B)

The key idea is learning while you act by using bootstrapped targets. In Temporal Difference learning, after each step you adjust the current state's value toward the reward you just received plus the discounted value of the next state. This uses the current estimate of future value (a bootstrap) rather than waiting for the episode to end to know the actual return. Because of this, updates happen step by step during the episode, enabling online learning. Concretely, you update V(S_t) toward R_{t+1} + gamma * V(S_{t+1}). This approach blends immediate reward with an estimate of future value, allowing continuous refinement as the agent progresses through the episode. This is in contrast to Monte Carlo methods, which wait until episode completion to compute the actual return G_t and then update. Other methods like Q-Learning also use bootstrapping and update online, but the general concept described here—updating estimates step-by-step during the episode using bootstrap targets—fits Temporal Difference learning best. Deep Reinforcement Learning extends these ideas with function approximation, but the core online, bootstrapped update characterizes Temporal Difference methods.

The key idea is learning while you act by using bootstrapped targets. In Temporal Difference learning, after each step you adjust the current state's value toward the reward you just received plus the discounted value of the next state. This uses the current estimate of future value (a bootstrap) rather than waiting for the episode to end to know the actual return. Because of this, updates happen step by step during the episode, enabling online learning.

Concretely, you update V(S_t) toward R_{t+1} + gamma * V(S_{t+1}). This approach blends immediate reward with an estimate of future value, allowing continuous refinement as the agent progresses through the episode. This is in contrast to Monte Carlo methods, which wait until episode completion to compute the actual return G_t and then update.

Other methods like Q-Learning also use bootstrapping and update online, but the general concept described here—updating estimates step-by-step during the episode using bootstrap targets—fits Temporal Difference learning best. Deep Reinforcement Learning extends these ideas with function approximation, but the core online, bootstrapped update characterizes Temporal Difference methods.