[REINFORCEMENT LEARNING] UNIT 5

▪ Introduction:
Reinforcement learning is a type of machine learning technique in which an
agent(hardware/software) interacts with the environment and learns to act according to it.
The reinforcement learning is the core part of artificial intelligence. For example, a robotic arm
learns the movement from its actions. The word reinforcement means to make something
stronger or to increase behavior.

Figure 1.1 : Reinforcement Learning.

▪ Basic concepts of reinforcement learning:
a) Agent: The agent is a software or hardware or a computer program which interacts with
the environment act receives rewards or penalty. For example, AI robot, robotic car, etc.
b) Environment: The situation of the agent. For example, Highway, Road, etc.
c) Action: The movements of agent inside the environment is called action. For example,
Move right, left, etc.
d) State: The situation returned by the environment after each action is called the state.
e) Reward: A positive feedback returned to the agent from the environment.
f) Penalty: A negative feedback returned to the agent from the environment.
g) Policy: Policy is a strategy of agent for next action. It is a method to map agent’s state to
the actions.
h) Policy map: The agent’s action selection is called policy map.
▪ Steps of reinforcement learning:
There are mainly three steps in reinforcement learning:
1. Take action.
2. Get feedback.
3. Remain in same state or change the state.
The primary objective in reinforcement learning is to improve the performance by
getting maximum rewards. A hardware or software agent takes actions. If action is
good, then agent gets positive feedback calls rewards. If the action is bad, then agent
gets negative feedback called penalty.

CST(6TH SEM, 3RD YEAR) Page 1

 [REINFORCEMENT LEARNING] UNIT 5

Figure 1.2 : Reinforcement Learning.

▪ Approaches to implement reinforcement learning:
There are three ways to implement reinforcement learning:
1. Value based: The value based approach is about to find the optimal value function.
2. Policy based: The policy based approach is to find the optimal policy for maximum
future rewards. These are two types:
i. Deterministic: Same action is repeated at any state.
ii. Stochastic(Random): Action is determined by probability randomly.
3. Model based: In this approach, a virtual model is created for the environment. The
agent explores the environment to learn it.

▪ Elements of reinforcement learning:

There are five elements of reinforcement learning:
1. Policy(𝜋): A policy can be defined as a way how an agent behaves at a given time. The
policy is the core element of reinforcement learning. T defines the behavior of an agent.
Policy can be a function or a lookup table.
2. Reward signal: After each state, the environment sends a signal to the learning agent.
This signal is called a reward signal. The rewards can be positive or negative. If the
action is good, then a positive reward signal is given to the agent. If the action is bad,
then a negative reward signal is given to the agent.
The reward signal can change the policy. If the rewards received by the agent are low
then agent changes its policy. Thus, rewards are the feedback from the environment.
3. Value function(V or V(s)): The value function specifies the good state and action for the
future. The value function determines the future rewards that an agent will receive in
the future. The value of a state is denoted by V(s).
4. Model based learning: In model-based learning, the model mimics the behavior of the
environment. If a state(s) and an action(a) are given, then the model can predict the
next state(S𝑖 ) and reward(𝑟𝑖 ). The approach for solving reinforcement learning problem
with the help of model is called model-based leaning. Thus, model is the environment in
which agent works.

CST(6TH SEM, 3RD YEAR) Page 2

 [REINFORCEMENT LEARNING] UNIT 5

5. State(S): It is the current situation of the agent.

▪ Bellman Equation:
In 1953, Mr. Richard Bellman introduced an equation for solving decision problems. This
equation is called Bellman Equation.
In reinforcement learning bellman equation is used to calculate the value of decision problems.
The main parameters used in bellman equation are as follows:
a) Agent action(a).
b) Gent state(s).
c) Reward point(R).
d) Discount factor gamma().

The simplified form of bellman equation is as follows:

V(s) = Max[R(s,a) + .V(s’)]

Where, V(s) = Value of present state(s).
Max() = Maximum value of the function.
R(s,a) = Reward at a state(s) after doing action(a).
 = Discount factor gamma.
V(s’) = Value at previous state(s).
In Bellman equation, we take maximum value because the agent tries to find optimal solution.

▪ Types of reinforcement learning:

There are two types of reinforcement learning:
1. Positive Reinforcement: If the action(a) is good, then a positive reward is achieved. This
is called positive reinforcement. It increases the same behaviour of agent.
2. Negative Reinforcement: If the action is poor, then a negative reward is achieved. This is
called negative reinforcement. It decreases the same behaviour of agent.

CST(6TH SEM, 3RD YEAR) Page 3

 [REINFORCEMENT LEARNING] UNIT 5

▪ The Learning Task:

A learning task is the type of prediction or decision being made, based on the problem that is
being asked and the available data.
One of the main method of learning task process is:
❖ Markov Decision Process(MDP):
The Markov Decision Process(MDP) is used to formulate and represent the
reinforcement learning problems. The dynamic states(S) of an environment can be
modeled as a Markov Process.
• Markov State: The Markov state contains all the required information of an
agent movements. A state(St) is called Markov state if it follows the following
condition:
P[S(t+1)/S(t)] = P[S(t+1)/S1,S2,……St]

Markov state analysis is used to forecast the value of a variable. This value is affected
by the current state. The reinforcement learning works on fully observable
environments. The agent can observe the environment and act for a new state. This
process is called Markov Decision Process(MDP). In Markov Decision Process, the agent
interacts with environment and perform actions. At each action, the environment
responds and provides a new state.

• The MDP contains four elements as:

i. A set of finite states(S).
ii. A set of finite actions(A).
iii. Rewards received®.
iv. Probability(P).
• Markov property:
The Markov property states that:
If an agent present in current state(St), performs an action(A1) and moves to
state(St). Then the state transition depends only on current state. It does not
depend on past actions, states and rewards. Hence MDP is useful to model
reinforcement learning problems. For example, chess game, football game. In
these games, only focus on the present state and no need to remember previous
state.
• Finite MDP:
In reinforcement learning, we consider only finite MDP. A finite MDP is a process
in which all states actions and rewards are finite.

CST(6TH SEM, 3RD YEAR) Page 4

 [REINFORCEMENT LEARNING] UNIT 5

• Markov Process(Markov Chain): A Markov Process is a memoryless process with

states S1, S2,…..,St. It is also called Markov Chain.
▪ Reinforcement Learning algorithms:
❖ Q-Learning Algorithm:
Q-learning is a model free reinforcement learning algorithm. It learns the value of an
action in a particular state. It does not require any model of environment. Hence, it is
called mode free learning. The Q in Q-Learning stands for Quality of actions. The quality
represents the usefulness of a given action.
Steps of Q-Learning algorithm:
i. In state(s) perform an action.
ii. Receive a reward(r) and move to resultant state(s’).
iii. Calculate the value of resultant state V(s’).
iv. Update the Q-value in the Q-table using equation:
W’(a,s) = W(a,s) V(s’)
Where, W’(a,s) = New weight of action-state pair.
W(a,s) = Previous weight of action-state pair.
V(s’) = Value of next state received.

Flow chart of Q-Learning algorithm:

Initialize Q-table

Choose an action

A number of Perform an action

iteration result a
good Q-table
Measure reward

Update Q-table

CST(6TH SEM, 3RD YEAR) Page 5

 [REINFORCEMENT LEARNING] UNIT 5

Bellman equation in Q-Learning:

The Q-function is based upon Bellman equation. The Bellman equation is a value
iteration update method. It uses weighted average of old value Q(s,a) and new value Qnew(s,a).
The Bellman equation is given as follows:

Qnew(𝑠𝑡 + 𝑎𝑡 ) = Q(𝑠𝑡 + 𝑎𝑡 ) + α(rt + .max.Q(st+1,a) – Q(st, at))

Where, Qnew(𝑠𝑡 + 𝑎𝑡 ) = New value of state action pair.

Q(𝑠𝑡 + 𝑎𝑡 ) = old value of state action pair.
α = learning rate.
rt = Reward.
 = discount factor.
max.Q(st+1,a) = Estimate of optimal future value rt + . max.Q(st+1,a)
rt + .max.Q(st+1,a) – Q(st, at) = temporal difference.

❖ Temporal Difference Learning:

The word Temporal means time. The Temporal Difference Learning is a type of learning
in which an agent learns from the environment through episodes(time steps). The agent
does not have prior knowledge of environment.
The temporal difference learning is an unsupervised technique which is most commonly
used in reinforcement learning. It is used predicting the total rewards expected in the
future. The temporal difference learning focuses on predicting the future value in a
sequence of state.

Parameters in Temporal Difference Learning:

1. Learning Rate(α): It shows the adjustment of estimates based on errors. It range
from 0 to 1. It is denoted by alpha(α).
2. Discount Rate(): It shows how much future rewards are valued. It range from 0
to 1. It is denoted by gamma()

Mathematical equation of Temporal Difference Learning:

V(s) = V(s) + α(r + .V(s’) – V(s))

Where, V(s) = New value prediction.

V(s) = Original value.

CST(6TH SEM, 3RD YEAR) Page 6

 [REINFORCEMENT LEARNING] UNIT 5

α = learning rate.
r = reward.
 = discount rate.
V(s’) = Old value function.
V(s) = Original value.
TD target = r + .S(s)
TD error = 𝑅𝑡+1 + .V(𝑠𝑡+1 ) – V(𝑠𝑡 )

▪ Application of Reinforcement Learning:

i. Robotics: Robots with pre-programmed behavior are useful in structured
environments, such as the assembly line of an automobile manufacturing plant, where
the task is repetitive in nature.
ii. A master chess player makes a move. The choice is informed both by planning,
anticipating possible replies and counter replies.
iii. Marketing and advertising.
iv. Trading and finance: An RL agent can decide on such a task; whether to hold, buy, or
sell. The RL model is evaluated using market benchmark standards in order to ensure
that it’s performing optimally.
v. News recommendation.
vi. Natural language processing: In NLP, RL can be used in text summarization, question
answering
vii. Self-diving cars: Some of the autonomous driving tasks where reinforcement learning
could be applied include trajectory optimization, motion planning, dynamic pathing,
controller optimization, and scenario-based learning policies for highways.

▪ Advantages of Reinforcement Learning:

i. Reinforcement learning can be used to solve very complex problems that cannot be
solved by conventional techniques.
ii. Reinforcement learning doesn’t require large labeled datasets. It’s a massive advantage
because as the amount of data in the world grows it becomes more and more costly to
label it for all required applications.
iii. One of the benefits of reinforcement learning is Adaptable, unlike supervised
learning algorithms, reinforcement learning doesn’t require retraining because it adapts
to new environments automatically on the fly.
iv. In RL, training data is obtained via the direct interaction of the agent with the
environment
v. Reinforcement learning can be used to solve a wide range of problems, including
those that involve decision making, control, and optimization.

CST(6TH SEM, 3RD YEAR) Page 7

 [REINFORCEMENT LEARNING] UNIT 5

▪ Disadvantages of Reinforcement Learning:

i. Reinforcement learning is not preferable to use for solving simple problems.
ii. Reinforcement learning requires an excessive amount of data and computation
iii. Reinforcement learning is highly dependent on the quality of the reward function. So,
when the reward function is designed poorly, it becomes difficult for the agent to learn
the behavior
iv. Reinforcement learning can be difficult to debug and interpret.

CST(6TH SEM, 3RD YEAR) Page 8

 [REINFORCEMENT LEARNING] UNIT 5

QUESTIONS:
1. What is reinforcement learning?
2. State some applications of RL.
3. Define Bellman Equation.
4. Write down steps of Q-Learning algorithm.
5. Explain temporal difference learning.
6. Advantages and disadvantages of RL.

CST(6TH SEM, 3RD YEAR) Page 9