Supervised Machine Learning

Supervised learning is the types of machine learning in which machines are

trained using well "labelled" training data, and on basis of that data, machines
predict the output. The labelled data means some input data is already tagged
with the correct output.

In supervised learning, the training data provided to the machines work as

the supervisor that teaches the machines to predict the output correctly. It
applies the same concept as a student learns in the supervision of the
teacher.

Supervised learning is a process of providing input data as well as correct

output data to the machine learning model. The aim of a supervised learning
algorithm is to find a mapping function to map the input variable(x) with the
output variable(y).

In the real-world, supervised learning can be used for Risk Assessment, Image
classification, Fraud Detection, spam filtering, etc.

How Supervised Learning Works?

In supervised learning, models are trained using labelled dataset, where the
model learns about each type of data. Once the training process is completed,
the model is tested on the basis of test data (a subset of the training set),
and then it predicts the output.

The working of Supervised learning can be easily understood by the below

example and diagram:
Suppose we have a dataset of different types of shapes which includes
square, rectangle, triangle, and Polygon. Now the first step is that we need
to train the model for each shape.

o If the given shape has four sides, and all the sides are equal, then it will be
labelled as a Square.

o If the given shape has three sides, then it will be labelled as a triangle.

o If the given shape has six equal sides then it will be labelled as hexagon.

Now, after training, we test our model using the test set, and the task of the
model is to identify the shape.

The machine is already trained on all types of shapes, and when it finds a
new shape, it classifies the shape on the bases of a number of sides, and
predicts the output.

Steps Involved in Supervised Learning:

o First Determine the type of training dataset
 o Collect/Gather the labelled training data.

o Split the training dataset into training dataset, test dataset, and validation
dataset.

o Determine the input features of the training dataset, which should have
enough knowledge so that the model can accurately predict the output.

o Determine the suitable algorithm for the model, such as support vector
machine, decision tree, etc.

o Execute the algorithm on the training dataset. Sometimes we need validation

sets as the control parameters, which are the subset of training datasets.

o Evaluate the accuracy of the model by providing the test set. If the model
predicts the correct output, which means our model is accurate.

Types of supervised Machine learning Algorithms:

Supervised learning can be further divided into two types of problems:

1. Regression

Regression algorithms are used if there is a relationship between the input

variable and the output variable. It is used for the prediction of continuous
variables, such as Weather forecasting, Market Trends, etc. Below are some
popular Regression algorithms which come under supervised learning:

o Linear Regression

o Regression Trees

o Non-Linear Regression

o Bayesian Linear Regression

o Polynomial Regression

2. Classification

Classification algorithms are used when the output variable is categorical,

which means there are two classes such as Yes-No, Male-Female, True-false,
etc.

Spam Filtering,

o Random Forest

o Decision Trees

o Logistic Regression

o Support vector Machines

Note: We will discuss these algorithms in detail in later chapters.

Advantages of Supervised learning:
o With the help of supervised learning, the model can predict the output on the
basis of prior experiences.

o In supervised learning, we can have an exact idea about the classes of

objects.
 o Supervised learning model helps us to solve various real-world problems such
as fraud detection, spam filtering, etc.

Disadvantages of supervised learning:

o Supervised learning models are not suitable for handling the complex tasks.

o Supervised learning cannot predict the correct output if the test data is
different from the training dataset.

o Training required lots of computation times.

o In supervised learning, we need enough knowledge about the classes of

object.

← PrevNext →
Unsupervised Machine Learning
In the previous topic, we learned supervised machine learning in which
models are trained using labeled data under the supervision of training data.
But there may be many cases in which we do not have labeled data and need
to find the hidden patterns from the given dataset. So, to solve such types of
cases in machine learning, we need unsupervised learning techniques.

What is Unsupervised Learning?

As the name suggests, unsupervised learning is a machine learning technique
in which models are not supervised using training dataset. Instead, models
itself find the hidden patterns and insights from the given data. It can be
compared to learning which takes place in the human brain while learning
new things. It can be defined as:

Unsupervised learning is a type of machine learning in which models are trained using unlabeled
dataset and are allowed to act on that data without any supervision.

Unsupervised learning cannot be directly applied to a regression or

classification problem because unlike supervised learning, we have the input
data but no corresponding output data. The goal of unsupervised learning is
to find the underlying structure of dataset, group that data according to
similarities, and represent that dataset in a compressed format.

Example: Suppose the unsupervised learning algorithm is given an input

dataset containing images of different types of cats and dogs. The algorithm
is never trained upon the given dataset, which means it does not have any
idea about the features of the dataset. The task of the unsupervised learning
algorithm is to identify the image features on their own. Unsupervised
learning algorithm will perform this task by clustering the image dataset into
the groups according to similarities between images.

Why use Unsupervised Learning?

Below are some main reasons which describe the importance of Unsupervised
Learning:

o Unsupervised learning is helpful for finding useful insights from the data.

o Unsupervised learning is much similar as a human learns to think by their own

experiences, which makes it closer to the real AI.

o Unsupervised learning works on unlabeled and uncategorized data which

make unsupervised learning more important.

o In real-world, we do not always have input data with the corresponding output
so to solve such cases, we need unsupervised learning.

Working of Unsupervised Learning

Working of unsupervised learning can be understood by the below diagram:
Here, we have taken an unlabeled input data, which means it is not
categorized and corresponding outputs are also not given. Now, this
unlabeled input data is fed to the machine learning model in order to train it.
Firstly, it will interpret the raw data to find the hidden patterns from the data
and then will apply suitable algorithms such as k-means clustering, Decision
tree, etc.

Once it applies the suitable algorithm, the algorithm divides the data objects
into groups according to the similarities and difference between the objects.

Types of Unsupervised Learning Algorithm:

The unsupervised learning algorithm can be further categorized into two
types of problems:
 o Clustering: Clustering is a method of grouping the objects into clusters such

that objects with most similarities remains into a group and has less or no
similarities with the objects of another group. Cluster analysis finds the
commonalities between the data objects and categorizes them as per the
presence and absence of those commonalities.

o Association: An association rule is an unsupervised learning method which is

used for finding the relationships between variables in the large database. It
determines the set of items that occurs together in the dataset. Association
rule makes marketing strategy more effective. Such as people who buy X item
(suppose a bread) are also tend to purchase Y (Butter/Jam) item. A typical
example of Association rule is Market Basket Analysis.

Note: We will learn these algorithms in later chapters.

Unsupervised Learning algorithms:
Below is the list of some popular unsupervised learning algorithms:

o K-means clustering

o KNN (k-nearest neighbors)

 o Hierarchal clustering

o Anomaly detection

o Neural Networks

o Principle Component Analysis

o Independent Component Analysis

o Apriori algorithm

o Singular value decomposition

Advantages of Unsupervised Learning

o Unsupervised learning is used for more complex tasks as compared to
supervised learning because, in unsupervised learning, we don't have labeled
input data.

o Unsupervised learning is preferable as it is easy to get unlabeled data in

comparison to labeled data.

Disadvantages of Unsupervised Learning

o Unsupervised learning is intrinsically more difficult than supervised learning
as it does not have corresponding output.

o The result of the unsupervised learning algorithm might be less accurate as

input data is not labeled, and algorithms do not know the exact output in
advance.

← PrevNext →
Introduction to Semi-Supervised Learning
Semi-Supervised learning is a type of Machine Learning algorithm that represents
the intermediate ground between Supervised and Unsupervised learning
algorithms. It uses the combination of labeled and unlabeled datasets during the
training period.

Before understanding the Semi-Supervised learning, you should know the

main categories of Machine Learning algorithms. Machine Learning consists
of three main categories: Supervised Learning, Unsupervised Learning,
and Reinforcement Learning. Further, the basic difference between Supervised
and unsupervised learning is that supervised learning datasets consist of an
output label training data associated with each tuple, and unsupervised
datasets do not consist the same. Semi-supervised learning is an important
category that lies between the Supervised and Unsupervised machine
learning. Although Semi-supervised learning is the middle ground between
supervised and unsupervised learning and operates on the data that consists
of a few labels, it mostly consists of unlabeled data. As labels are costly, but
for the corporate purpose, it may have few labels.

The basic disadvantage of supervised learning is that it requires hand-labeling

by ML specialists or data scientists, and it also requires a high cost to process.
Further unsupervised learning also has a limited spectrum for its
applications. To overcome these drawbacks of supervised learning and
unsupervised learning algorithms, the concept of Semi-supervised learning is
introduced. In this algorithm, training data is a combination of both labeled
and unlabeled data. However, labeled data exists with a very small amount
while it consists of a huge amount of unlabeled data. Initially, similar data is
clustered along with an unsupervised learning algorithm, and further, it helps
to label the unlabeled data into labeled data. It is why label data is a
comparatively, more expensive acquisition than unlabeled data.

We can imagine these algorithms with an example. Supervised learning is

where a student is under the supervision of an instructor at home and college.
Further, if that student is self-analyzing the same concept without any help
from the instructor, it comes under unsupervised learning. Under semi-
supervised learning, the student has to revise itself after analyzing the same
concept under the guidance of an instructor at college.

Assumptions followed by Semi-Supervised

Learning
To work with the unlabeled dataset, there must be a relationship between the
objects. To understand this, semi-supervised learning uses any of the
following assumptions:
 o Continuity Assumption:

As per the continuity assumption, the objects near each other tend to share
the same group or label. This assumption is also used in supervised learning,
and the datasets are separated by the decision boundaries. But in semi-
supervised, the decision boundaries are added with the smoothness
assumption in low-density boundaries.

o Cluster assumptions- In this assumption, data are divided into different discrete

clusters. Further, the points in the same cluster share the output label.

o Manifold assumptions- This assumption helps to use distances and densities,

and this data lie on a manifold of fewer dimensions than input space.

o The dimensional data are created by a process that has less degree of freedom
and may be hard to model directly. (This assumption becomes practical if high).

Working of Semi-Supervised Learning

Semi-supervised learning uses pseudo labeling to train the model with less
labeled training data than supervised learning. The process can combine
various neural network models and training ways. The whole working of semi-
supervised learning is explained in the below points:

o Firstly, it trains the model with less amount of training data similar to the
supervised learning models. The training continues until the model gives
accurate results.

o The algorithms use the unlabeled dataset with pseudo labels in the next step,
and now the result may not be accurate.

o Now, the labels from labeled training data and pseudo labels data are linked
together.

o The input data in labeled training data and unlabeled training data are also
linked.
 o In the end, again train the model with the new combined input as did in the
first step. It will reduce errors and improve the accuracy of the model.

Difference between Semi-supervised and Reinforcement

Learning.

Reinforcement learning is different from semi-supervised learning, as it works

with rewards and feedback. Reinforcement learning aims to maximize the
rewards by their hit and trial actions, whereas in semi-supervised learning, we train
the model with a less labeled dataset.

Real-world applications of Semi-supervised

Learning-
Semi-supervised learning models are becoming more popular in the
industries. Some of the main applications are as follows.

o Speech Analysis- It is the most classic example of semi-supervised learning

applications. Since, labeling the audio data is the most impassable task that
requires many human resources, this problem can be naturally overcome with
the help of applying SSL in a Semi-supervised learning model.

o Web content classification- However, this is very critical and impossible to label

each page on the internet because it needs mode human intervention. Still,
this problem can be reduced through Semi-Supervised learning algorithms.
Further, Google also uses semi-supervised learning algorithms to rank a
webpage for a given query.

o Protein sequence classification- DNA strands are larger, they require active

human intervention. So, the rise of the Semi-supervised model has been
proximate in this field.
 o Text document classifier- As we know, it would be very unfeasible to find a large

amount of labeled text data, so semi-supervised learning is an ideal model to

overcome this.

← PrevNext →
Reinforcement Learning Tutorial

Our Reinforcement learning tutorial will give you a complete overview of

reinforcement learning, including MDP and Q-learning. In RL tutorial, you will
learn the below topics:

o What is Reinforcement Learning?

o Terms used in Reinforcement Learning.

o Key features of Reinforcement Learning.

o Elements of Reinforcement Learning.

o Approaches to implementing Reinforcement Learning.

o How does Reinforcement Learning Work?

o The Bellman Equation.

o Types of Reinforcement Learning.

o Reinforcement Learning Algorithm.

o Markov Decision Process.

o What is Q-Learning?

o Difference between Supervised Learning and Reinforcement Learning.

o Applications of Reinforcement Learning.

o Conclusion.
What is Reinforcement Learning?
o Reinforcement Learning is a feedback-based Machine learning technique in
which an agent learns to behave in an environment by performing the actions
and seeing the results of actions. For each good action, the agent gets positive
feedback, and for each bad action, the agent gets negative feedback or
penalty.

o In Reinforcement Learning, the agent learns automatically using feedbacks

without any labeled data, unlike supervised learning.

o Since there is no labeled data, so the agent is bound to learn by its experience
only.

o RL solves a specific type of problem where decision making is sequential, and

the goal is long-term, such as game-playing, robotics, etc.

o The agent interacts with the environment and explores it by itself. The primary
goal of an agent in reinforcement learning is to improve the performance by
getting the maximum positive rewards.

o The agent learns with the process of hit and trial, and based on the
experience, it learns to perform the task in a better way. Hence, we can say
that "Reinforcement learning is a type of machine learning method where an
intelligent agent (computer program) interacts with the environment and learns to
act within that." How a Robotic dog learns the movement of his arms is an

example of Reinforcement learning.

o It is a core part of Artificial intelligence, and all AI agent works on the concept
of reinforcement learning. Here we do not need to pre-program the agent, as
it learns from its own experience without any human intervention.

o Example: Suppose there is an AI agent present within a maze environment,

and his goal is to find the diamond. The agent interacts with the environment
 by performing some actions, and based on those actions, the state of the
agent gets changed, and it also receives a reward or penalty as feedback.

o The agent continues doing these three things (take action, change state/remain
in the same state, and get feedback), and by doing these actions, he learns and

explores the environment.

o The agent learns that what actions lead to positive feedback or rewards and
what actions lead to negative feedback penalty. As a positive reward, the
agent gets a positive point, and as a penalty, it gets a negative point.

Terms used in Reinforcement Learning

o Agent(): An entity that can perceive/explore the environment and act upon it.
 o Environment(): A situation in which an agent is present or surrounded by. In

RL, we assume the stochastic environment, which means it is random in

nature.

o Action(): Actions are the moves taken by an agent within the environment.

o State(): State is a situation returned by the environment after each action

taken by the agent.

o Reward(): A feedback returned to the agent from the environment to evaluate

the action of the agent.

o Policy(): Policy is a strategy applied by the agent for the next action based on

the current state.

o Value(): It is expected long-term retuned with the discount factor and opposite

to the short-term reward.

o Q-value(): It is mostly similar to the value, but it takes one additional

parameter as a current action (a).

Key Features of Reinforcement Learning

o In RL, the agent is not instructed about the environment and what actions
need to be taken.

o It is based on the hit and trial process.

o The agent takes the next action and changes states according to the feedback
of the previous action.

o The agent may get a delayed reward.

o The environment is stochastic, and the agent needs to explore it to reach to

get the maximum positive rewards.
Approaches to implement Reinforcement Learning
There are mainly three ways to implement reinforcement-learning in ML,
which are:

1. Value-based:

The value-based approach is about to find the optimal value function, which
is the maximum value at a state under any policy. Therefore, the agent
expects the long-term return at any state(s) under policy π.

2. Policy-based:

Policy-based approach is to find the optimal policy for the maximum future
rewards without using the value function. In this approach, the agent tries to
apply such a policy that the action performed in each step helps to maximize
the future reward.
The policy-based approach has mainly two types of policy:

o Deterministic: The same action is produced by the policy (π) at any

state.

o Stochastic: In this policy, probability determines the produced action.

3. Model-based: In the model-based approach, a virtual model is created for the

environment, and the agent explores that environment to learn it. There is no
particular solution or algorithm for this approach because the model
representation is different for each environment.

Elements of Reinforcement Learning

There are four main elements of Reinforcement Learning, which are given
below:

1. Policy
 2. Reward Signal

3. Value Function

4. Model of the environment

1) Policy: A policy can be defined as a way how an agent behaves at a given

time. It maps the perceived states of the environment to the actions taken
on those states. A policy is the core element of the RL as it alone can define
the behavior of the agent. In some cases, it may be a simple function or a
lookup table, whereas, for other cases, it may involve general computation
as a search process. It could be deterministic or a stochastic policy:

For deterministic policy: a = π(s)

For stochastic policy: π(a | s) = P[At =a | St = s]

2) Reward Signal: The goal of reinforcement learning is defined by the reward

signal. At each state, the environment sends an immediate signal to the
learning agent, and this signal is known as a reward signal. These rewards are
given according to the good and bad actions taken by the agent. The agent's
main objective is to maximize the total number of rewards for good actions.
The reward signal can change the policy, such as if an action selected by the
agent leads to low reward, then the policy may change to select other actions
in the future.

3) Value Function: The value function gives information about how good the
situation and action are and how much reward an agent can expect. A reward
indicates the immediate signal for each good and bad action, whereas a value
function specifies the good state and action for the future. The value function
depends on the reward as, without reward, there could be no value. The goal
of estimating values is to achieve more rewards.
4) Model: The last element of reinforcement learning is the model, which
mimics the behavior of the environment. With the help of the model, one can
make inferences about how the environment will behave. Such as, if a state
and an action are given, then a model can predict the next state and reward.

The model is used for planning, which means it provides a way to take a
course of action by considering all future situations before actually
experiencing those situations. The approaches for solving the RL
problems with the help of the model are termed as the model-based approach.
Comparatively, an approach without using a model is called a model-free
approach.

How does Reinforcement Learning Work?

To understand the working process of the RL, we need to consider two main
things:

o Environment: It can be anything such as a room, maze, football ground, etc.

o Agent: An intelligent agent such as AI robot.

Let's take an example of a maze environment that the agent needs to explore.
Consider the below image:
In the above image, the agent is at the very first block of the maze. The maze
is consisting of an S 6 block, which is a wall, S8 a fire pit, and S4 a diamond
block.

The agent cannot cross the S6 block, as it is a solid wall. If the agent reaches
the S4 block, then get the +1 reward; if it reaches the fire pit, then gets -1
reward point. It can take four actions: move up, move down, move left, and move
right.

The agent can take any path to reach to the final point, but he needs to make
it in possible fewer steps. Suppose the agent considers the path S9-S5-S1-S2-
S3, so he will get the +1-reward point.
The agent will try to remember the preceding steps that it has taken to reach
the final step. To memorize the steps, it assigns 1 value to each previous
step. Consider the below step:

Now, the agent has successfully stored the previous steps assigning the 1
value to each previous block. But what will the agent do if he starts moving
from the block, which has 1 value block on both sides? Consider the below
diagram:
It will be a difficult condition for the agent whether he should go up or down
as each block has the same value. So, the above approach is not suitable for
the agent to reach the destination. Hence to solve the problem, we will use
the Bellman equation, which is the main concept behind reinforcement
learning.

The Bellman Equation

The Bellman equation was introduced by the Mathematician Richard Ernest
Bellman in the year 1953, and hence it is called as a Bellman equation. It is
associated with dynamic programming and used to calculate the values of a
decision problem at a certain point by including the values of previous states.
It is a way of calculating the value functions in dynamic programming or
environment that leads to modern reinforcement learning.

The key-elements used in Bellman equations are:

o Action performed by the agent is referred to as "a"

o State occurred by performing the action is "s."

o The reward/feedback obtained for each good and bad action is "R."

o A discount factor is Gamma "γ."

The Bellman equation can be written as:

1. V(s) = max [R(s,a) + γV(s`)]

Where,

V(s)= value calculated at a particular point.

R(s,a) = Reward at a particular state s by performing an action.

γ = Discount factor

V(s`) = The value at the previous state.

In the above equation, we are taking the max of the complete values because
the agent tries to find the optimal solution always.

So now, using the Bellman equation, we will find value at each state of the
given environment. We will start from the block, which is next to the target
block.

For 1st block:

V(s3) = max [R(s,a) + γV(s`)], here V(s')= 0 because there is no further
state to move.

V(s3)= max[R(s,a)]=> V(s3)= max[1]=> V(s3)= 1.

For 2nd block:

V(s2) = max [R(s,a) + γV(s`)], here γ= 0.9(lets), V(s')= 1, and R(s, a)= 0,
because there is no reward at this state.

V(s2)= max[0.9(1)]=> V(s)= max[0.9]=> V(s2) =0.9

For 3rd block:

V(s1) = max [R(s,a) + γV(s`)], here γ= 0.9(lets), V(s')= 0.9, and R(s, a)=
0, because there is no reward at this state also.

V(s1)= max[0.9(0.9)]=> V(s3)= max[0.81]=> V(s1) =0.81

For 4th block:

V(s5) = max [R(s,a) + γV(s`)], here γ= 0.9(lets), V(s')= 0.81, and R(s, a)=
0, because there is no reward at this state also.

V(s5)= max[0.9(0.81)]=> V(s5)= max[0.81]=> V(s5) =0.73

For 5th block:

V(s9) = max [R(s,a) + γV(s`)], here γ= 0.9(lets), V(s')= 0.73, and R(s, a)=
0, because there is no reward at this state also.

V(s9)= max[0.9(0.73)]=> V(s4)= max[0.81]=> V(s4) =0.66

Consider the below image:

Now, we will move further to the 6 th block, and here agent may change the
route because it always tries to find the optimal path. So now, let's consider
from the block next to the fire pit.
Now, the agent has three options to move; if he moves to the blue box, then
he will feel a bump if he moves to the fire pit, then he will get the -1 reward.
But here we are taking only positive rewards, so for this, he will move to
upwards only. The complete block values will be calculated using this formula.
Consider the below image:
Types of Reinforcement learning
There are mainly two types of reinforcement learning, which are:

o Positive Reinforcement

o Negative Reinforcement

Positive Reinforcement:

The positive reinforcement learning means adding something to increase the

tendency that expected behavior would occur again. It impacts positively on
the behavior of the agent and increases the strength of the behavior.
This type of reinforcement can sustain the changes for a long time, but too
much positive reinforcement may lead to an overload of states that can
reduce the consequences.

Negative Reinforcement:

The negative reinforcement learning is opposite to the positive reinforcement

as it increases the tendency that the specific behavior will occur again by
avoiding the negative condition.

It can be more effective than the positive reinforcement depending on

situation and behavior, but it provides reinforcement only to meet minimum
behavior.

How to represent the agent state?

We can represent the agent state using the Markov State that contains all the
required information from the history. The State St is Markov state if it follows
the given condition:

P[St+1 | St ] = P[St +1 | S1,......, St]

The Markov state follows the Markov property, which says that the future is
independent of the past and can only be defined with the present. The RL
works on fully observable environments, where the agent can observe the
environment and act for the new state. The complete process is known as
Markov Decision process, which is explained below:
Markov Decision Process
Markov Decision Process or MDP, is used to formalize the reinforcement
learning problems. If the environment is completely observable, then its
dynamic can be modeled as a Markov Process. In MDP, the agent constantly
interacts with the environment and performs actions; at each action, the
environment responds and generates a new state.

MDP is used to describe the environment for the RL, and almost all the RL
problem can be formalized using MDP.

MDP contains a tuple of four elements (S, A, Pa, Ra):

o A set of finite States S

o A set of finite Actions A

 o Rewards received after transitioning from state S to state S', due to action a.

o Probability Pa.

MDP uses Markov property, and to better understand the MDP, we need to
learn about it.

Markov Property:

It says that "If the agent is present in the current state S1, performs an action a1
and move to the state s2, then the state transition from s1 to s2 only depends on
the current state and future action and states do not depend on past actions,
rewards, or states."

Or, in other words, as per Markov Property, the current state transition does
not depend on any past action or state. Hence, MDP is an RL problem that
satisfies the Markov property. Such as in a Chess game, the players only focus
on the current state and do not need to remember past actions or states.

Finite MDP:

A finite MDP is when there are finite states, finite rewards, and finite actions.
In RL, we consider only the finite MDP.

Markov Process:

Markov Process is a memoryless process with a sequence of random states

S1, S2, ....., St that uses the Markov Property. Markov process is also known
as Markov chain, which is a tuple (S, P) on state S and transition function P.
These two components (S and P) can define the dynamics of the system.
Reinforcement Learning Algorithms
Reinforcement learning algorithms are mainly used in AI applications and
gaming applications. The main used algorithms are:

o Q-Learning:

o Q-learning is an Off policy RL algorithm, which is used for the temporal

difference Learning. The temporal difference learning methods are the

way of comparing temporally successive predictions.

o It learns the value function Q (S, a), which means how good to take
action "a" at a particular state "s."

o The below flowchart explains the working of Q- learning:

o State Action Reward State action (SARSA):

 o SARSA stands for State Action Reward State action, which is an on-
policy temporal difference learning method. The on-policy control

method selects the action for each state while learning using a specific
policy.

o The goal of SARSA is to calculate the Q π (s, a) for the selected current
policy π and all pairs of (s-a).

o The main difference between Q-learning and SARSA algorithms is

that unlike Q-learning, the maximum reward for the next state is not
required for updating the Q-value in the table.

o In SARSA, new action and reward are selected using the same policy,
which has determined the original action.

o The SARSA is named because it uses the quintuple Q(s, a, r, s',

a'). Where,
s: original state
a: Original action
r: reward observed while following the states
s' and a': New state, action pair.

o Deep Q Neural Network (DQN):

o As the name suggests, DQN is a Q-learning using Neural networks.

o For a big state space environment, it will be a challenging and complex

task to define and update a Q-table.

o To solve such an issue, we can use a DQN algorithm. Where, instead of

defining a Q-table, neural network approximates the Q-values for each
action and state.

Now, we will expand the Q-learning.

Q-Learning Explanation:
o Q-learning is a popular model-free reinforcement learning algorithm based on
the Bellman equation.

o The main objective of Q-learning is to learn the policy which can inform the agent
that what actions should be taken for maximizing the reward under what
circumstances.

o It is an off-policy RL that attempts to find the best action to take at a current

state.

o The goal of the agent in Q-learning is to maximize the value of Q.

o The value of Q-learning can be derived from the Bellman equation. Consider
the Bellman equation given below:

In the equation, we have various components, including reward, discount

factor (γ), probability, and end states s'. But there is no any Q-value is given
so first consider the below image:
In the above image, we can see there is an agent who has three values
options, V(s1), V(s2), V(s3). As this is MDP, so agent only cares for the current
state and the future state. The agent can go to any direction (Up, Left, or
Right), so he needs to decide where to go for the optimal path. Here agent
will take a move as per probability bases and changes the state. But if we
want some exact moves, so for this, we need to make some changes in terms
of Q-value. Consider the below image:

Q- represents the quality of the actions at each state. So instead of using a

value at each state, we will use a pair of state and action, i.e., Q(s, a). Q-
value specifies that which action is more lubricative than others, and
according to the best Q-value, the agent takes his next move. The Bellman
equation can be used for deriving the Q-value.

To perform any action, the agent will get a reward R(s, a), and also he will
end up on a certain state, so the Q -value equation will be:
Hence, we can say that, V(s) = max [Q(s, a)]

The above formula is used to estimate the Q-values in Q-Learning.

What is 'Q' in Q-learning?

The Q stands for quality in Q-learning, which means it specifies the quality of
an action taken by the agent.

Q-table:

A Q-table or matrix is created while performing the Q-learning. The table

follows the state and action pair, i.e., [s, a], and initializes the values to zero.
After each action, the table is updated, and the q-values are stored within
the table.

The RL agent uses this Q-table as a reference table to select the best action
based on the q-values.

Difference between Reinforcement Learning and

Supervised Learning
The Reinforcement Learning and Supervised Learning both are the part of
machine learning, but both types of learnings are far opposite to each other.
The RL agents interact with the environment, explore it, take action, and get
rewarded. Whereas supervised learning algorithms learn from the labeled
dataset and, on the basis of the training, predict the output.

The difference table between RL and Supervised learning is given below:

Reinforcement Supervised
Learning Learning

RL works by Supervised
interacting with the learning works on
environment. the existing
dataset.

The RL algorithm Supervised

works like the human Learning works as
brain works when when a human
making some learns things in the
decisions. supervision of a
guide.

There is no labeled The labeled dataset

dataset is present is present.

No previous training Training is provided

is provided to the to the algorithm so
learning agent. that it can predict
the output.
RL helps to take In Supervised
decisions learning, decisions
sequentially. are made when
input is given.

Reinforcement Learning Applications

1. Robotics:

a. RL is used in Robot navigation, Robo-soccer, walking, juggling, etc.

2. Control:
 a. RL can be used for adaptive control such as Factory processes,

admission control in telecommunication, and Helicopter pilot is an

example of reinforcement learning.

3. Game Playing:

a. RL can be used in Game playing such as tic-tac-toe, chess, etc.

4. Chemistry:

a. RL can be used for optimizing the chemical reactions.

5. Business:

a. RL is now used for business strategy planning.

6. Manufacturing:

a. In various automobile manufacturing companies, the robots use deep

reinforcement learning to pick goods and put them in some containers.

7. Finance Sector:

a. The RL is currently used in the finance sector for evaluating trading

strategies.

Conclusion:
From the above discussion, we can say that Reinforcement Learning is one of
the most interesting and useful parts of Machine learning. In RL, the agent
explores the environment by exploring it without any human intervention. It
is the main learning algorithm that is used in Artificial Intelligence. But there
are some cases where it should not be used, such as if you have enough data
to solve the problem, then other ML algorithms can be used more efficiently.
The main issue with the RL algorithm is that some of the parameters may
affect the speed of the learning, such as delayed feedback.