Scribd
Upload
1 views64 pages

Module3 RL

Q-Learning is a model-free, off-policy reinforcement learning algorithm that learns state-action values through trial and error, converging to an optimal policy with sufficient exploration. Double Q-Learning enhances Q-Learning by using two Q-tables to reduce overestimation bias and improve performance. SARSA, an on-policy algorithm, learns the value of the policy currently being followed, resulting in safer policies compared to Q-Learning's more optimistic approach.

Uploaded by

mecartinone
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
1 views64 pages

Module3 RL

Q-Learning is a model-free, off-policy reinforcement learning algorithm that learns state-action values through trial and error, converging to an optimal policy with sufficient exploration. Double Q-Learning enhances Q-Learning by using two Q-tables to reduce overestimation bias and improve performance. SARSA, an on-policy algorithm, learns the value of the policy currently being followed, resulting in safer policies compared to Q-Learning's more optimistic approach.

Uploaded by

mecartinone
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Q-Learning

•Q-Learning is a model-free reinforcement learning algorithm.


•It is an off-policy learning method.
•It learns state-action values (Q-values).
•It updates Q-values using the Bellman equation.
•It selects actions using exploration and exploitation.
•It always considers the maximum future reward.
•It learns through trial and error.
•It converges to the optimal policy with sufficient exploration.
•It is suitable for discrete state and action spaces.
•It is widely used in robotics, autonomous driving, game AI,
recommendation systems, and network routing.
Double Q-Learning
[Link] two Q-tables: Maintains Q₁ and Q₂ instead of a single Q-table to
estimate action values.
[Link] overestimation bias: By separating action selection and action
evaluation, it avoids the overly optimistic value estimates common in
standard Q-Learning.
[Link] updates: In each step, only one Q-table is updated, while the
other is used to evaluate the selected action.
[Link]-policy algorithm: Like Q-Learning, Double Q-Learning learns the
optimal policy independently of the agent's exploratory behavior.
[Link] performance: It provides more accurate Q-value estimates,
leading to more stable learning and improved performance in environments
where overestimation is a problem.
SARSA (State–Action–Reward–State–
Action)
SARSA is an on-policy reinforcement learning algorithm. It learns the
value of the same policy that the agent is currently following.
The name SARSA comes from the sequence:
S → A → R → S' → A'
•S = Current State
•A = Current Action
•R = Reward received
•S' = Next State
•A' = Next Action actually chosen by the current policy
Why does SARSA learn safer policies than
Q-Learning?

Answer:
SARSA updates its Q-values using the actual next action
selected by the current policy, including exploratory actions.
Because it accounts for the risks introduced by exploration, it
tends to learn safer and more conservative policies. Q-
Learning, on the other hand, assumes the best possible next
action during updates, making it more optimistic and often
more aggressive.

You might also like