Scribd
Upload
2 views9 pages

Module2 A

The document discusses concepts related to Reinforcement Learning (RL), including the structure of trajectories in finite Markov Decision Processes (MDPs), the definition of rewards and policies, and the goal of maximizing returns. It emphasizes the importance of the value function, which estimates expected returns under a given policy, and introduces optimal values and policies. The document also highlights the Markov property, indicating that the future state depends only on the current state and action, not on past states or actions.

Uploaded by

Gnana Sai Siddartha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views9 pages

Module2 A

The document discusses concepts related to Reinforcement Learning (RL), including the structure of trajectories in finite Markov Decision Processes (MDPs), the definition of rewards and policies, and the goal of maximizing returns. It emphasizes the importance of the value function, which estimates expected returns under a given policy, and introduces optimal values and policies. The document also highlights the Markov property, indicating that the future state depends only on the current state and action, not on past states or actions.

Uploaded by

Gnana Sai Siddartha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Full RL

• Here
• The trajectory followed is S0 A0 R1 S1 A1
R2 S2 A2………..
• In finite MDP, S, A, R all are finite
meaning they have finite number of
elements
• Rt and St are random variables and
have a well defined probability
distribution.
Reward and policy
• Reward is a scalar function
• Agent has no prior knowledge on how to get rewards – out of direct
control of the agent
• Bounded and frequent
Policy

Policy defines the probability that an action a is selected in state s at


time t
Normally we deal with stationary policies
Returns (deals with long term)
• General notation
Gt = Rt+1 + Rt+2 + Rt+3 ………+ RT (Sum of all the rewards we receive in future)
T – end of episode (eg. End of chess game) – random variable
Goal: To maximize returns
• Notion 2
Gt = Rt+1 + γRt+2 + γ2Rt+3 …… (0 ≤ γ < 1) γ – discount factor
The reward received now is more valuable than the reward I may get in
future
Notion 3
Rt+1 + Rt+2 + - for finite N
Rt+1 + Rt+2 + for infinite N
→
Full RL
• For particular values of random variables, s’ S and r R, there
is a probability of those values occurring at time t, given
particular values of the preceding state and action:

Satisfies
Full RL
• In a Markov decision process, the probabilities given by p
completely characterize the environment’s dynamics.
• The probability of each possible value for St and Rt depends only
on the immediately preceding state and action, St−1 and At−1,
and, given them, not at all on earlier states and actions
• The state must include information about all aspects of the past
agent–environment interaction that make a difference for the
future – Markov property
Value function
• The value function of a state s under a policy п, denoted vп(s), is the
expected return when starting in s and following п thereafter.
• state-value function for policy п at time t is given as

• the value of taking action a in state s under a policy п , denoted as qп(s, a), as
the expected return starting from s, taking the action a, and thereafter
following policy п
Value function – recursive relationship
Optimal values and optimal policies
• Optimal state value function

• Optimal action value function

• Optimal action value function as a function of optimal state


value function
Optimal values

You might also like