One of the core concepts of reinforcement learning is
the Bellman Equation and Value Functions.
A clear intuition of these concepts will allow you to solve
most Reinforcement Learning problems if you have an
environment to solve them.
The Reward Signal
As we all know Reinforcement Learning is a reward-
based learning system. The agent learns from past
experiences of actions and their subsequent rewards.
A reward signal(R) is scalar feedback from the
environment on how well the agent is performing in an
environment for a given state.
An episode can be defined as an agent-environment
interaction. We seek to maximize the total reward called
the Expected Return.
(1)
The expected Return is represented by Gt
You can assume (1) is the simplest case representing
the total reward as given but this can be suitable for
episodic tasks that end at the terminal state.
There are situations where there is no terminal state
also called continual tasks. You can imagine for such
tasks the reward keeps accumulating and as time
increases all rewards earned in states would move
towards infinity.
Thus we introduce a concept of discounting also called
the discount factor denoted by gamma. The discount
factor helps you figure out just how valuable the
rewards are to the future states.
(2)
The Expected Return Gt is represented along with the discount factor in eq. 2.
The reward earned k steps in the future is not as
valuable as the same reward earned in an earlier state.
The discount factor has a range of (0, 1).
Setting the discount factor to 0 makes the agent
“myopic” where the agent learns to maximize the
reward only on the first time step.
The closer the discount factor is to 1 the agent is
“farsighted”, the agent takes the importance of more
future rewards.
The Value Function and Bellman Equation
Value Function is one of the most important
components of an RL agent. The Value Function as a
performance metric to evaluate how well an agent’s
policy performs in an environment and the potential
rewards expected to be earned.
Each state in an environment has a Value Function. The
Value Function of a state is the expected future
reward to be earned from the current state the agent is
in until it reaches the terminal state. It can be
formulated using the Bellman Equation.
(3)
The Value Function is the expected return Gt (sum of future rewards) of a state s.
This equation is broken down into two parts:
1. The reward earned by the agent taking an action in
the current state.
2. The Value Function of the next state the agent
transitions to.
(4)
Bellman Equation of the Value Function
The Value Function (4) is represented as the sum of the
reward earned after taking an action in state s and the
Value Function of the next state. (Note: The reward for
state s at time step t is represented as R with a time
step t+1 because the agent earns the reward only after
it has taken an action in state s and transitioned to the
next state.)
(5)
Intuition in using the Value Function of the next state to calculate the value function of
the current state.
The reason behind using the Value Function of the next
state is simple, it represents the sum of returns Gt
earned by the agent being in state s at time
step t+1 until the terminal state. It can be represented
as given in (4).
The above Value Function provides us an estimate of the
rewards earned in a state and an agent’s performance
but no action-specific information. This is where
the Action-Value Function of a state comes into play.
(6)
Bellman Equation of the Action-Value Function
Instead of having a single Value Function for each state,
with the concept of Action-Value Function, each state
has a separate Value Function for every possible action
in a given state.
An Action-Value Function represents the expected
future reward that can be earned given a certain
action the agent takes in a state. The Action-Value
Function of a state generally the best action possible in
a state will be the one with the highest Action-Value
Function.
Conclusion:
1. The Reward Signal and how it can be calculated.
2. The Bellman Equation and how it presents us with
the Value Function.
3. The Action-Value Function that can be derived
from the Value Function and how it is different from
the Value Function.