Scribd
Upload
13 views5 pages

Reinforcement Learning

The document discusses various reinforcement learning problems, including the Grid World, Tic-Tac-Toe, and Maze problems, highlighting their definitions, components, objectives, and applications. It also covers policy gradient methods, their advantages and disadvantages, and the Bellman expectation equations and optimality, which form the foundation for key reinforcement learning algorithms. The content emphasizes the importance of learning optimal policies and maximizing expected rewards in different environments.

Uploaded by

sharmavardhan703
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
13 views5 pages

Reinforcement Learning

The document discusses various reinforcement learning problems, including the Grid World, Tic-Tac-Toe, and Maze problems, highlighting their definitions, components, objectives, and applications. It also covers policy gradient methods, their advantages and disadvantages, and the Bellman expectation equations and optimality, which form the foundation for key reinforcement learning algorithms. The content emphasizes the importance of learning optimal policies and maximizing expected rewards in different environments.

Uploaded by

sharmavardhan703
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

1.

Study of Various Reinforcement Learning Problems


A. Grid World Problem

 Definition
o Grid World is a simple environment represented as a 2D grid.
o An agent moves from one cell (state) to another to reach a goal.

 Components
o States: Each grid cell.
o Actions: Up, Down, Left, Right.
o Rewards:
 Positive reward for goal
 Negative or zero reward for other moves
o Policy: Mapping from states to actions.

 Objective
o Learn an optimal policy to reach the goal with maximum
cumulative reward.

 Applications
o Robot navigation
o Path planning
o Understanding value iteration & policy iteration

B. Tic-Tac-Toe Problem

 Definition
o A two-player, turn-based reinforcement learning problem.
o Agent learns to play optimally through repeated games.

 Components
o States: Board configurations.
o Actions: Placing X or O in empty cells.
o Rewards:
 Win → +1
 Loss → −1
 Draw → 0

 Learning Process
o Uses Q-learning or Monte Carlo methods.
o Learns optimal moves by self-play.

 Key Insight
o Demonstrates adversarial learning and exploration vs
exploitation.

C. Maze Problem

 Definition
o Agent navigates through a maze to reach a target.

 Components
o States: Positions in the maze.
o Actions: Move in directions.
o Rewards:
 Goal → positive reward
 Wall hit → penalty
 Step cost → small negative reward

 Challenges
o Sparse rewards
o Dead ends
o Long-term planning

 Algorithms Used
o Q-learning
o SARSA
o Deep Q-Networks (DQN)

2. Policy Gradient Methods


A. Definition

 Policy Gradient methods directly optimize the policy instead of


value functions.
 Policy is represented by parameters θ.
B. Core Idea

 Adjust policy parameters to maximize expected reward.


 Uses gradient ascent on expected return.

C. Mathematical Objective

 Maximize:
o

 Update rule:
o

D. Common Policy Gradient Algorithms

 REINFORCE
 Actor-Critic
 Proximal Policy Optimization (PPO)
 Trust Region Policy Optimization (TRPO)

E. Advantages

 Works well with continuous action spaces


 Learns stochastic policies
 More stable in complex environments

F. Disadvantages

 High variance
 Slower convergence
 Requires many samples
3. Bellman Expectation Equations & Optimality
A. Bellman Expectation Equation (State Value Function)

 Defines value of a state under a policy π:

V^\pi(s) = \sum_a \pi(a|s) \sum_{s'} P(s'|s,a)


[ R(s,a,s') + \gamma V^\pi(s') ]

 Meaning:
o Current value = immediate reward + discounted future value

B. Bellman Expectation for Action-Value Function

Q^\pi(s,a) = \sum_{s'} P(s'|s,a)


[ R(s,a,s') + \gamma \sum_{a'} \pi(a'|s') Q^\pi(s',a') ]

C. Optimal Value Functions

 Optimal state value function:

V^*(s) = \max_\pi V^\pi(s)

 Optimal action value function:

Q^*(s,a) = \max_\pi Q^\pi(s,a)

D. Bellman Optimality Equation

V^*(s) = \max_a \sum_{s'} P(s'|s,a)


[ R(s,a,s') + \gamma V^*(s') ]

E. Optimal Policy

 A policy π* is optimal if:

\pi^*(s) = \arg\max_a Q^*(s,a)


 Guarantees maximum expected return from every state.

F. Importance

 Foundation of:
o Value Iteration
o Policy Iteration
o Q-learning

You might also like