Which method is a value-based reinforcement learning method built on Temporal Difference learning, learning the value of taking an action in a state (Q-value) rather than just the state's value?

Which method is a value-based reinforcement learning method built on Temporal Difference learning, learning the value of taking an action in a state (Q-value) rather than just the state's value?

• A. Temporal Difference (TD) method
• B. Q-Learning
• C. Deep Reinforcement Learning
• D. Monte Carlo method

Answer: (B)

Learning an action-value function with temporal-difference updates is what Q-learning does. It estimates Q(s, a), the expected return of taking action a in state s and then following the best policy thereafter. The update blends what was observed (the reward) with an estimate of future value, a bootstrapped term: Q(s, a) becomes Q(s, a) plus a learning rate times [r + gamma times the maximum Q(s', a') over all possible next actions a' minus the current Q(s, a)]. This focuses on the value of taking specific actions, not just the value of being in a state. Because the update uses the best possible next-action value regardless of which action was actually taken, Q-learning is off-policy and can learn the optimal policy while exploring. It’s the classic value-based, action-value TD method. Monte Carlo would learn from complete episodes without bootstrapping, Deep Reinforcement Learning refers to using function approximators (like neural nets) for broader methods (including Q-learning variants), and a general TD method refers to the broader family rather than this specific action-value approach.

Learning an action-value function with temporal-difference updates is what Q-learning does. It estimates Q(s, a), the expected return of taking action a in state s and then following the best policy thereafter. The update blends what was observed (the reward) with an estimate of future value, a bootstrapped term: Q(s, a) becomes Q(s, a) plus a learning rate times [r + gamma times the maximum Q(s', a') over all possible next actions a' minus the current Q(s, a)]. This focuses on the value of taking specific actions, not just the value of being in a state. Because the update uses the best possible next-action value regardless of which action was actually taken, Q-learning is off-policy and can learn the optimal policy while exploring. It’s the classic value-based, action-value TD method. Monte Carlo would learn from complete episodes without bootstrapping, Deep Reinforcement Learning refers to using function approximators (like neural nets) for broader methods (including Q-learning variants), and a general TD method refers to the broader family rather than this specific action-value approach.