AL3451- MACHINE LEARNING

UNIT-1 INTRODUCTION TO MACHINE LEARNING

Review of Linear Algebra for machine learning; Introduction and

motivation for machine learning; Examples of machine learning
applications, Vapnik-Chervonenkis (VC) dimension, Probably
Approximately Correct (PAC) learning, Hypothesis spaces,
Inductive bias, Generalization, Bias variance trade-off.
1. LINEAR ALGEBRA FOR MACHINE LEARNING

 Machine learning has a strong connection with mathematics. Each machine learning
algorithm is based on the concepts of mathematics & also with the help of mathematics,
one can choose the correct algorithm by considering training time, complexity, number of
features, etc.
 Linear Algebra is an essential field of mathematics, which defines the study of vectors,
matrices, planes, mapping, and lines required for linear transformation.

 Linear algebra plays a vital role and key foundation in machine learning, and it enables
ML algorithms to run on a huge number of datasets.
 The concepts of linear algebra are widely used in developing algorithms in machine
learning. Although it is used almost in each concept of Machine learning, specifically, it
can perform the following task:

o Optimization of data.
o Applicable in loss functions, regularisation, covariance matrices, Singular Value
Decomposition (SVD), Matrix Operations, and support vector machine classification.
o Implementation of Linear Regression in Machine Learning.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Besides the above uses, linear algebra is also used in neural networks and the data science field.

 Basic mathematics principles and concepts like Linear algebra are the foundation of
Machine Learning and Deep Learning systems.
 To learn and understand Machine Learning or Data Science, one needs to be familiar with
linear algebra and optimization theory.

1.1 Why learn Linear Algebra before learning Machine Learning?

Linear Algebra is just similar to the flour of bakery in Machine Learning. As the cake is based on
flour similarly, every Machine Learning Model is also based on Linear Algebra. Further, the
cake also needs more ingredients like egg, sugar, cream, soda. Similarly, Machine Learning also
requires more concepts as vector calculus, probability, and optimization theory. So, we can say
that Machine Learning creates a useful model with the help of the above-mentioned
mathematical concepts.

Below are some benefits of learning Linear Algebra before Machine learning:

o Better Graphic experience

o Improved Statistics
o Creating better Machine Learning algorithms
o Estimating the forecast of Machine Learning
o Easy to Learn

Better Graphics Experience:

 Linear Algebra helps to provide better graphical processing in Machine Learning like
Image, audio, video, and edge detection.
 These are the various graphical representations supported by Machine Learning projects
that you can work on.
 Further, parts of the given data set are trained based on their categories by classifiers
provided by machine learning algorithms.
 These classifiers also remove the errors from the trained data.

Moreover, Linear Algebra helps solve and compute large and complex data set through a specific
terminology named Matrix Decomposition Techniques. There are two most popular matrix
decomposition techniques, which are as follows:

o Q-R
o L-U

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Improved Statistics:

Statistics is an important concept to organize and integrate data in Machine Learning. Also,
linear Algebra helps to understand the concept of statistics in a better manner. Advanced
statistical topics can be integrated using methods, operations, and notations of linear algebra.

Creating better Machine Learning algorithms:

Linear Algebra also helps to create better supervised as well as unsupervised Machine Learning
algorithms.

Few supervised learning algorithms can be created using Linear Algebra, which is as follows:

o Logistic Regression
o Linear Regression
o Decision Trees
o Support Vector Machines (SVM)

Further, below are some unsupervised learning algorithms listed that can also be created with the
help of linear algebra as follows:

o Single Value Decomposition (SVD)

o Clustering
o Components Analysis

With the help of Linear Algebra concepts, you can also self-customize the various parameters in
the live project and understand in-depth knowledge to deliver the same with more accuracy and
precision.

Estimating the forecast of Machine Learning:

If you are working on a Machine Learning project, then you must be a broad-minded person and
also, you will be able to impart more perspectives. Hence, in this regard, you must increase the
awareness and affinity of Machine Learning concepts. You can begin with setting up different
graphs, visualization, using various parameters for diverse machine learning algorithms or taking
up things that others around you might find difficult to understand.

Easy to Learn:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Linear Algebra is an important department of Mathematics that is easy to understand. It is taken

into consideration whenever there is a requirement of advanced mathematics and its applications.

1.2 Minimum Linear Algebra for Machine Learning

Notation:

Notation in linear algebra enables you to read algorithm descriptions in papers, books, and
websites to understand the algorithm's working. Even if you use for-loops rather than matrix
operations, you will be able to piece things together.

Operations:

Working with an advanced level of abstractions in vectors and matrices can make concepts
clearer, and it can also help in the description, coding, and even thinking capability. In linear
algebra, it is required to learn the basic operations such as addition, multiplication, inversion,
transposing of matrices, vectors, etc.

Matrix Factorization:

One of the most recommended areas of linear algebra is matrix factorization, specifically
matrix deposition methods such as SVD and QR.

Describe briefly about some examples of linear algebra in machine learning?

2. EXAMPLES OF LINEAR ALGEBRA IN MACHINE LEARNING

Below are some popular examples of linear algebra in Machine learning:

• Datasets and Data Files

• Linear Regression
• Recommender Systems
• One-hot encoding
• Regularization
• Principal Component Analysis
• Images and Photographs
• Singular-Value Decomposition
• Deep Learning
• Latent Semantic Analysis

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

1. Datasets and Data Files

Each machine learning project works on the dataset, and we fit the machine learning
model using this dataset.
Each dataset resembles a table-like structure consisting of rows and columns. Where each
row represents observations, and each column represents features/Variables. This dataset
is handled as a Matrix, which is a key data structure in Linear Algebra.
Further, when this dataset is divided into input and output for the supervised learning
model, it represents a Matrix(X) and Vector(y), where the vector is also an important
concept of linear algebra.

2. Images and Photographs

In machine learning, images/photographs are used for computer vision applications. Each
Image is an example of the matrix from linear algebra because an image is a table structure
consisting of height and width for each pixel.
Moreover, different operations on images, such as cropping, scaling, resizing, etc., are
performed using notations and operations of Linear Algebra.

3. One Hot Encoding

In machine learning, sometimes, we need to work with categorical data. These

categorical variables are encoded to make them simpler and easier to work with, and the
popular encoding technique to encode these variables is known as one-hot encoding.
In the one-hot encoding technique, a table is created that shows a variable with one
column for each category and one row for each example in the dataset. Further, each row
is encoded as a binary vector, which contains either zero or one value. This is an example
of sparse representation, which is a subfield of Linear Algebra.

4. Linear Regression

Linear regression is a popular technique of machine learning borrowed from statistics.

It describes the relationship between input and output variables and is used in machine
learning to predict numerical values.
The most common way to solve linear regression problems using Least Square
Optimization is solved with the help of Matrix factorization methods.
Some commonly used matrix factorization methods are LU decomposition, or Singular-
value decomposition, which are the concept of linear algebra.

5. Regularization

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

In machine learning, we usually look for the simplest possible model to achieve the best
outcome for the specific problem.
Simpler models generalize well, ranging from specific examples to unknown datasets.
These simpler models are often considered models with smaller coefficient values.
A technique used to minimize the size of coefficients of a model while it is being fit on
data is known as regularization.
Common regularization techniques are L1 and L2 regularization.
Both of these forms of regularization are, in fact, a measure of the magnitude or length of
the coefficients as a vector and are methods lifted directly from linear algebra called the
vector norm.

6. Principal Component Analysis

Generally, each dataset contains thousands of features, and fitting the model with such a
large dataset is one of the most challenging tasks of machine learning.
Moreover, a model built with irrelevant features is less accurate than a model built with
relevant features.
There are several methods in machine learning that automatically reduce the number of
columns of a dataset, and these methods are known as Dimensionality reduction.
The most commonly used dimensionality reductions method in machine learning is
Principal Component Analysis or PCA.
This technique makes projections of high-dimensional data for both visualizations and
training models. PCA uses the matrix factorization method from linear algebra.

7. Singular-Value Decomposition

Singular-Value decomposition is also one of the popular dimensionality reduction

techniques and is also written as SVD in short form.
It is the matrix-factorization method of linear algebra, and it is widely used in different
applications such as feature selection, visualization, noise reduction, and many more.

8. Latent Semantic Analysis

Natural Language Processing or NLP is a subfield of machine learning that works with text
and spoken words.
NLP represents a text document as large matrices with the occurrence of words.
For example, the matrix column may contain the known vocabulary words, and rows may
contain sentences, paragraphs, pages, etc., with cells in the matrix marked as the count or
frequency of the number of times the word occurred.
It is a sparse matrix representation of text. Documents processed in this way are much
easier to compare, query, and use as the basis for a supervised machine learning model.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

This form of data preparation is called Latent Semantic Analysis, or LSA for short, and is
also known by the name Latent Semantic Indexing or LSI.

9. Recommender System

A recommender system is a sub-field of machine learning, a predictive modelling

problem that provides recommendations of products.
For example, online recommendation of books based on the customer's previous purchase
history, recommendation of movies and TV series, as we see in Amazon & Netflix.
The development of recommender systems is mainly based on linear algebra methods.
We can understand it as an example of calculating the similarity between sparse customer
behaviour vectors using distance measures such as Euclidean distance or dot products.
Different matrix factorization methods such as singular-value decomposition are used in
recommender systems to query, search, and compare user data.

10. Deep Learning

Artificial Neural Networks or ANN are the non-linear ML algorithms that work to
process the brain and transfer information from one layer to another in a similar way.
Deep learning studies these neural networks, which implement newer and faster hardware
for the training and development of larger networks with a huge dataset.
All deep learning methods achieve great results for different challenging tasks such as
machine translation, speech recognition, etc.
The core of processing neural networks is based on linear algebra data structures, which
are multiplied and added together.
Deep learning algorithms also work with vectors, matrices, tensors (matrix with more than
two dimensions) of inputs and coefficients for multiple dimensions.

Explain in detail about machine learning concepts with differentlearning types.

Discuss with examples some useful applications of machine learning.

Differentiate supervised, unsupervised and reinforcement learning.

3 .INTRODUCTION TO MACHINE LEARNING

 Machine learning is the field of study that allows computers to learn without being
explicitly programmed.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

 Using machine learning, we don't need to provide explicit instructions to Computers for
reacting to some special situations.
 We need to provide training to the computers to find real-time solutions for the specific
problems.
 The chess game is a famous example where machine learning is being used to play chess.
 The code lets the machine learn and optimizes itself over repeated games.

Applications of Machine learning

Machine learning is a buzzword for today's technology, and it is growing very rapidly day by day.
We are using machine learning in our daily life even without knowing it such as Google
Maps, Google assistant, Alexa, etc. Below are some most trending real-world applications of
Machine Learning:

1. Image Recognition:

Image recognition is one of the most common applications of machine learning. It is used to
identify objects, persons, places, digital images, etc. The popular use case of image recognition
and face detection is, Automatic friend tagging suggestion:

Facebook provides us a feature of auto friend tagging suggestion. Whenever we upload a photo
with our Facebook friends, then we automatically get a tagging suggestion with name, and the
technology behind this is machine learning's face detection and recognition algorithm.

It is based on the Facebook project named "Deep Face," which is responsible for face
recognition and person identification in the picture.

2. Speech Recognition

While using Google, we get an option of "Search by voice," it comes under speech recognition,
and it's a popular application of machine learning.

Speech recognition is a process of converting voice instructions into text, and it is also known as
"Speech to text", or "Computer speech recognition." At present, machine learning algorithms
are widely used by various applications of speech recognition. Google assistant, Siri, Cortana,
and Alexa are using speech recognition technology to follow the voice instructions.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
3. Traffic prediction:

If we want to visit a new place, we take help of Google Maps, which shows us the correct path
with the shortest route and predicts the traffic conditions.
It predicts the traffic conditions such as whether traffic is cleared, slow-moving, or heavily
congested with the help of two ways:

o Real Time location of the vehicle form Google Map app and sensors
o Average time has taken on past days at the same time.

Everyone who is using Google Map is helping this app to make it better. It takes information
from the user and sends back to its database to improve the performance.

4. Product recommendations:

Machine learning is widely used by various e-commerce and entertainment companies such as
Amazon, Netflix, etc., for product recommendation to the user. Whenever we search for some
product on Amazon, then we started getting an advertisement for the same product while internet
surfing on the same browser and this is because of machine learning.

Google understands the user interest using various machine learning algorithms and suggests the
product as per customer interest.

As similar, when we use Netflix, we find some recommendations for entertainment series,
movies, etc., and this is also done with the help of machine learning.

5. Self-driving cars:

One of the most exciting applications of machine learning is self-driving cars. Machine learning
plays a significant role in self-driving cars. Tesla, the most popular car manufacturing company
is working on self-driving car. It is using unsupervised learning method to train the car models to
detect people and objects while driving.

6. Email Spam and Malware Filtering:

Whenever we receive a new email, it is filtered automatically as important, normal, and spam.
We always receive an important mail in our inbox with the important symbol and spam emails in
our spam box, and the technology behind this is Machine learning. Below are some spam filters
used by Gmail:

o Content Filter
o Header filter
o General blacklists filter
o Rules-based filters
o Permission filters
Some machine learning algorithms such as Multi-Layer Perceptron, Decision tree, and
Naïve Bayes classifier are used for email spam filtering and malware detection.
Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

7. Virtual Personal Assistant:

We have various virtual personal assistants such as Google assistant, Alexa, Cortana, Siri. As
the name suggests, they help us in finding the information using our voice instruction. These
assistants can help us in various ways just by our voice instructions such as Play music, call
someone, Open an email, Scheduling an appointment, etc.

These virtual assistants use machine learning algorithms as an important part.

These assistant record our voice instructions, send it over the server on a cloud, and decode it
using ML algorithms and act accordingly.

8. Online Fraud Detection:

Machine learning is making our online transaction safe and secure by detecting fraud transaction.
Whenever we perform some online transaction, there may be various ways that a fraudulent
transaction can take place such as fake accounts, fake ids, and steal money in the middle of a
transaction. So to detect this, Feed Forward Neural network helps us by checking whether it is
a genuine transaction or a fraud transaction.

For each genuine transaction, the output is converted into some hash values, and these values
become the input for the next round. For each genuine transaction, there is a specific pattern
which gets change for the fraud transaction hence, it detects it and makes our online transactions
more secure.

9. Stock Market trading:

Machine learning is widely used in stock market trading. In the stock market, there is always a
risk of up and downs in shares, so for this machine learning's long short term memory neural
network is used for the prediction of stock market trends.

10. Medical Diagnosis:

In medical science, machine learning is used for diseases diagnoses. With this, medical
technology is growing very fast and able to build 3D models that can predict the exact position
of lesions in the brain.

It helps in finding brain tumors and other brain-related diseases easily.

11. Automatic Language Translation:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Nowadays, if we visit a new place and we are not aware of the language then it is not a problem
at all, as for this also machine learning helps us by converting the text into our known languages.
Google's GNMT (Google Neural Machine Translation) provide this feature, which is a Neural
Machine Learning that translates the text into our familiar language, and it called as automatic
translation.

The technology behind the automatic translation is a sequence to sequence learning algorithm,
which is used with image recognition and translates the text from one language to another
language.

4. HYPOTHESIS IN MACHINE LEARNING

What is Hypothesis?

The hypothesis is defined as the supposition or proposed explanation based on insufficient

evidence or assumptions. It is just a guess based on some known facts but has not yet been
proven. A good hypothesis is testable, which results in either true or false.

Example: Let's understand the hypothesis with a common example. Some scientist claims that
ultraviolet (UV) light can damage the eyes then it may also cause blindness.

In this example, a scientist just claims that UV rays are harmful to the eyes, but we assume they
may cause blindness. However, it may or may not be possible. Hence, these types of assumptions
are called a hypothesis.

Hypothesis in Machine Learning (ML)

The hypothesis is one of the commonly used concepts of statistics in Machine Learning. It is
specifically used in Supervised Machine learning, where an ML model learns a function that best
maps the input to corresponding outputs with the help of an available dataset.

In supervised learning techniques, the main aim is to determine the possible hypothesis out of
hypothesis space that best maps input to the corresponding or correct outputs.
Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

There are some common methods given to find out the possible hypothesis from the Hypothesis
space, where hypothesis space is represented by uppercase-h (H) and hypothesis by lowercase-
h (h). Th ese are defined as follows:

Hypothesis space (H):

Hypothesis space is defined as a set of all possible legal hypotheses; hence it is also known as
a hypothesis set. It is used by supervised machine learning algorithms to determine the best
possible hypothesis to describe the target function or best maps input to output.

It is often constrained by choice of the framing of the problem, the choice of model, and the
choice of model configuration.

Hypothesis (h):

It is defined as the approximate function that best describes the target in supervised machine
learning algorithms. It is primarily based on data as well as bias and restrictions applied to data.

Hence hypothesis (h) can be concluded as a single hypothesis that maps input to proper output
and can be evaluated as well as used to make predictions.

The hypothesis (h) can be formulated in machine learning as follows:

y= mx + b
Where,

Y: Range

m: Slope of the line which divided test data or changes in y divided by change in x. x:

domain

c: intercept (constant)

Example: Let's understand the hypothesis (h) and hypothesis space (H) with a two-
dimensional coordinate plane showing the distribution of data as follows:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
Now, assume we have some test data by which ML algorithms predict the outputs for input as
follows:

If we divide this coordinate plane in such as way that it can help you to predict output or result
as follows:

Based on the given test data, the output result will be as follows:

However, based on data, algorithm, and constraints, this coordinate plane can also be divided in the
following ways as follows:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

With the above example, we can conclude that;

Hypothesis space (H) is the composition of all legal best possible ways to divide the
coordinate plane so that it best maps input to proper output.

Further, each individual best possible way is called a hypothesis (h). Hence, the hypothesis and
hypothesis space would be like this:

5. INDUCTIVE BIAS:

Definition

Every machine learning model requires some type of architecture design and possibly some
initial assumptions about the data we want to analyze. Generally, every building block and
every belief that we make about the data is a form of inductive bias.
 Inductive biases play an important role in the ability of machine learning models to
generalize to the unseen data.
 A strong inductive bias can lead our model to converge to the global optimum.
 On the other hand, a weak inductive bias can cause the model to find only the local
optima and be greatly affected by random changes in the initial states.
 Inductive bias helps the algorithm focus on what's important in the data and avoid
getting lost in irrelevant details. It steers the learning process towards specific types of
solutions.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
Types of inductive bias

• Prior Knowledge: Some algorithms incorporate prior knowledge about the problem through their
architecture or initial parameters. For example, convolutional neural networks used for image
recognition are inherently biased towards detecting edges and shapes useful in recognizing
objects in images.
• Learning Algorithm Bias: The way an algorithm processes data also introduces bias. For
instance, Support Vector Machines (SVMs) focus on finding the widest margin between classes.
This can be good for clear separation but might miss subtle patterns in more complex data.
• Occam's Razor Bias: Many machine learning algorithms exhibit a bias towards simpler
explanations. This means they prefer models with fewer parameters to avoid overfitting, which
occurs when the model memorizes the training data too well and fails to generalize to unseen
data.

5.1 Inductive Biases in Machine Learning

In traditional machine learning, every algorithm has its own inductive biases. In this
section, we mention some of these algorithms.
5.1.1 Bayesian Models
 Inductive bias in Bayesian models shows itself in the form of the prior distributions that
we choose for the variables.
 Consequently, the prior can shape the posterior distribution in a way that the latter can
turn out to be a similar distribution to the former.
 In addition, we assume that the variables are conditionally independent, meaning that
given the parents of a node in the network, it’ll be independent from its ancestors.
 As a result, we can make use of conditional probability to make the inference.
 Also, the structure of the Bayesian net can facilitate the analysis of causal
relationships between entities.

5.1.2. k-Nearest Neighbors (k-NN) Algorithm

The k-Nearest Neighbors(k-NN) algorithm assumes that entities belonging to a particular

category should appear near each other, and those that are part of different groups should be
distant. In other words, we assume that similar data points are clustered near each other away
from the dissimilar ones.

5.1.3. Linear Regression

Given the (X,Y)data points, in linear regression, we assume that the variable (Y)is linearly
dependent on the explanatory variables (X). Therefore, the resulting model linearly fits the
training data. However, this assumption can limit the model’s capacity to learn non-linear
functions.

5.1.4. Logistic Regression

In logistic regression, we assume that there’s a hyperplane that separates the two classes from
Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
each other. This simplifies the problem, but one can imagine that if the assumption is not valid,
we won’t have a good model.

5.2 Relational Inductive Biases in Deep Learning

Relational inductive biases define the structure of the relationships between different
entities or parts in our model. These relations can be arbitrary, sequential, local, and so
on.

5.2.1. Weak Relation

Sometimes the relationship between the neural units is weak, meaning that they’re somewhat
independent of each other. The choice of including a fully connected layer in the net can
represent this kind of relationship:

5.2.2. Locality

In order to process an image, we start by capturing the local information. One way to do that is
the use of a convolutional layer. It can capture the local relationship between the pixels of an
image. Then, as we go deeper in the model, the local feature extractors help to extract the
global features:

5.2.3. Sequential Relation

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
Sometimes our data has a sequential characteristic. For instance, time series and sentences
consist of sequential elements that appear one after another. To model this pattern, we can
introduce a recurrent layer to our network:

5.2.4. Arbitrary Relation

To solve problems related to a group of things or people, it might be more informative to see
them as a graph. The graph structure imposes arbitrary relationships between the entities,
which is ideal when there’s no clear sequential or local relation in the model:

6. BIAS AND VARIANCE IN MACHINE LEARNING

 While making predictions, a difference occurs between prediction values made by the
model and actual values/expected values, and this difference is known as bias errors.
 Low Bias: (Fewer assumptions)
 Decision Trees, k-Nearest Neighbours and Support Vector Machines.
 High Bias:(more assumptions)
 Linear Regression, Linear Discriminant Analysis and Logistic Regression.
 if the machine learning model is not accurate, it can make predictions errors, and these
prediction errors are usually known as Bias and Variance.
 In machine learning, these errors will always be present as there is always a slight
difference between the model predictions and actual predictions.
 The main aim of ML/data science analysts is to reduce these errors in order to get more
accurate results.
Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Errors in Machine Learning?

In machine learning, an error is a measure of how accurately an algorithm can make predictions
for the previously unknown dataset. On the basis of these errors, the machine learning model is
selected that can perform best on the particular dataset. There are mainly two types of errors in
machine learning, which are:

o Reducible errors: These errors can be reduced to improve the model accuracy. Such
errors can further be classified into bias and Variance

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Irreducible errors: These errors will always be present in the model

regardless of which algorithm has been used. The cause of these errors is unknown variables
whose value can't be reduced.

What is Bias?

While making predictions, a difference occurs between prediction values made by the model
and actual values/expected values, and this difference is known as bias errors or Errors due to
bias.

A model has either:

o Low Bias: A low bias model will make fewer assumptions about the form of the target
function.
o High Bias: A model with a high bias makes more assumptions, and the model becomes
unable to capture the important features of our dataset. A high bias model also cannot
perform well on new data.

Some examples of machine learning algorithms with low bias are Decision Trees, k- Nearest
Neighbours and Support Vector Machines.

At the same time, an algorithm with high bias is Linear Regression, Linear Discriminant
Analysis and Logistic Regression.

Ways to reduce High Bias:

High bias mainly occurs due to a much simple model. Below are some ways to reduce the high
bias:

o Increase the input features as the model is underfitted.

o Decrease the regularization term.
o Use more complex models, such as including some polynomial features.

What is a Variance Error

Variance tells that how much a random variable is different from its expected value.
Ideally, a model should not vary too much from one training dataset to another, which means the
algorithm should be good in understanding the hidden mapping between inputs and output
variables.

Variance errors are either of low variance or high variance.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

Low variance means there is a small variation in the prediction of the target function with
changes in the training data set.

At the same time, High variance shows a large variation in the prediction of the target function
with changes in the training dataset.

A model that shows high variance learns a lot and perform well with the training dataset, and
does not generalize well with the unseen dataset. As a result, such a model gives good results
with the training dataset but shows high error rates on the test dataset.

Since, with high variance, the model learns too much from the dataset, it leads to overfitting of
the model. A model with high variance has the below problems:

o A high variance model leads to overfitting.

o Increase model complexities.

Usually, nonlinear algorithms have a lot of flexibility to fit the model, have high variance.

Some examples of machine learning algorithms with low variance are, Linear Regression,
Logistic Regression, and Linear discriminant analysis.

At the same time, algorithms with high variance are decision tree, Support Vector Machine,
and K-nearest neighbours.

Ways to Reduce High Variance:

o Reduce the input features or number of parameters as a model is overfitted.

o Do not use a much complex model.
o Increase the training data.
o Increase the Regularization term.

Different Combinations of BiasVariance:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

There are four possible combinations of bias and variances, which are represented by the below
diagram:

1. Low-Bias,Low-Variance:The combination of low bias and low variance shows an ideal

machine learning model. However, it is not possible practically.
2. Low-Bias, High-Variance: With low bias and high variance, model predictions are
inconsistent and accurate on average. This case occurs when the model learns with a large
number of parameters and hence leads to an overfitting
3. High-Bias,Low-Variance: With High bias and low variance, predictions are consistent
but inaccurate on average. This case occurs when a model does not learn well with the
training dataset or uses few numbers of the parameter. It leads to underfitting
problems in the model.
4. High-Bias,High-Variance:With high bias and high variance, predictions are inconsistent
and also inaccurate on average.

How to identify High variance or High Bias? High

variance can be identified if the model has:

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

o Low training error and high test error.

High Bias can be identified if the model has:

o High training error and the test error is almost similar to training error.

BIAS-VARIANCE TRADE-OFF

 While building the machine learning model, it is really important to take care of bias and
variance in order to avoid overfitting and underfitting in the model.
 If the model is very simple with fewer parameters, it may have low variance and high
bias.
 Whereas, if the model has a large number of parameters, it will have high variance and
low bias.
 So, it is required to make a balance between bias and variance errors, and this balance
between the bias error and variance error is known as the Bias-Variancetrade-off.

For an accurate prediction of the model, algorithms need a low variance and low bias. But this is
not possible because bias and variance are related to each other:

o If we decrease the variance, it will increase the bias.

o If we decrease the bias, it will increase thevariance.

Hence, the Bias-Variance trade-off is about finding the sweet spot to make a balance
between bias and variance errors.

7. VAPNIK-CHERVONENKIS (VC) DIMENSION / PAC

 The Vapnik-Chervonenkis (VC) dimension is a measure of the capacity of a

hypothesis set to fit different data sets. It was introduced by Vladimir Vapnik and
Alexey Chervonenkis in the 1970s and has become a fundamental concept in
statistical learning theory.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
 Let H be the hypothesis space for some machine learning problem. The Vapnik-
Chervonenkis dimension of H, also called the VC dimension of H, and denoted by V
C(H), is a measure of the complexity of the space H. To define the VC dimension we
require the notion of the shattering of a set of instances.
Shattering of a set

 A set of N points is set to be shattered by a hypothesis space, if there are

hypothesis H in H that separates positive examples from the negative examples in
all of the 2^n possible ways

 Let D be a dataset containing N examples for a binary classification problem with class
Labels 0 and 1. Let H be a hypothesis space for the problem. Each hypothesis h in H
partitions D into two disjoint subsets as follows:
 {x ∈ D ∣h(x) = 0} and {x ∈ D ∣h(x) = 1}.

For 2n different classifiers

Example
Let the instance space X be the set of all real numbers. Consider the hypothesis space
defined by

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
 Let D be a subset of X containing only a single number, say, D = {3.5}. There are 2
dichotomies for this set. These correspond to the following assignment of class
labels

 Let D be a subset of X containing two elements, say, D = {3.25, 4.75}. There are 4
dichotomies in D and they correspond to the assignment of class labels shown in Table .
 In these dichotomies, h5 is consistent with (a), h4 is consistent with (b) and h3 is
consistent with (d). But there is no hypothesis hm ∈ H consistent with (c).
 Thus the two-element set D is not shattered by H. In a similar way it can be shown
that there is no two-element subset of X which is shattered by H.
 It follows that the size of the largest finite subset of X shattered by H is [Link]
number is the VC dimension of H.

PAC:
 probably approximately correct learning (PAC learning) is a framework for
mathematical analysis of machine learning algorithms. It was proposed in 1984 by
Leslie Valiant.
 The goal is that, with high probability (the “probably” part), the selected hypothesis
will have low generalization error

Definition :

 The concept class C is said to be PAC-learnable if there is an algorithm A which, for

samples drawn with any probability distribution F and any concept c Є C, will with
high probability produce a hypothesis h Є C whose error is small.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING

 Given a set of sample points labeled positive or negative

 The concept class C is PAC-learnable by the algorithm L using the hypothesis space
H of all axis-aligned rectangles.

Dr. M. SUPRIYA,
AP/CSE
AL3451- MACHINE LEARNING
Applications of PAC Learning
1. Machine Learning Algorithm Design
PAC learning provides a theoretical basis for designing and evaluating machine
learning algorithms. It helps in understanding the trade-offs between sample size,
accuracy, and confidence, guiding the development of efficient and reliable
algorithms.
Example: Support Vector Machines (SVMs) are designed with the goal of
maximizing the margin between classes, which is linked to minimizing the VC
dimension and ensuring good generalization as per PAC learning principles.
2. Model Selection and Evaluation
In practice, PAC learning aids in model selection by providing criteria for choosing
models that balance complexity and performance. It helps in determining the
appropriate hypothesis class and regularization techniques to avoid overfitting.
Example: In decision tree learning, PAC principles guide the choice of tree depth
and pruning strategies to ensure the model generalizes well to new data.
3. Active Learning
Active learning is a paradigm where the learning algorithm selectively queries the
most informative examples from a large pool of unlabeled data. PAC learning
principles can be applied to determine the optimal querying strategy to achieve the
desired accuracy and confidence with fewer labeled examples.
Example: In a text classification task, an active learning algorithm might query
examples that are closest to the decision boundary, as these provide the most
information for improving the model.
Challenges and Limitations
While PAC learning provides a solid theoretical foundation, it has some limitations
and challenges:
 Assumptions: PAC learning assumes that the data distribution is fixed and known, which
may not hold in real-world scenarios where distributions can change over time.
 Computational Complexity: Calculating the exact VC dimension for complex hypothesis
classes can be computationally infeasible, limiting practical applications.
 Model Expressiveness: The hypothesis class needs to be expressive enough to contain a
good approximation of the target function, which can be challenging for complex problems.

Dr. M. SUPRIYA,
AP/CSE
 UNIT II

2 SUPERVISED LEARNING

Syllabus
Introduction to machine learning-Linear Regression Models:Leastsquares,single&
multiples variables ,Bayesian linear regression ,gradient descent ,Linear Classification
Models :Discriminant function-Probablistic discriminative model -Logistic
regression,Probablistic generative model-Navive Bayes ,Maximum margin classifier-
Support Vector machine,Decision Tree,Random forests

2.1 INTRODUCTION TO MACHINE LEARNING

• Machine Learning (ML) is a sub-field of Artificial Intelligence (Al) which concerns with
developing computational theories of learning and building learning machines.

• Learning is a phenomenon and process which has manifestations of various aspects. Learning
proCess includes gaining of new symbolic knowledge and development of cognitive skills
through instruction and practice. It is also discovery of new facts and theories through
observation and experiment.

• Machine Learning Definition : A computer program is said to learn from experience E with
respect to some class of tasks T and performance measure p, if its performance at tasks in T,
as measured by P, improves with experience E.

• Machine learning is programming computers to optimize a performance criterion using

example data or past experience. Application of machine learning methods to large databases
is called data mining.

• The goal of machine learning is to build computer systems that can adapt and learn from their
experience.

Examples:
i) Handwriting recognition learning problem

• Task T: Recognising and classifying handwritten words within images

• Performance P: Percent of words correctly classified

Dr. M. SUPRIYA,
AP/CSE
 • Training experience E: A dataset of handwritten words with given classifications

ii) A robot driving learning problem

• Task T: Driving on highways using vision sensors

• Performance measure P: Average distance traveled before an error

• training experience: A sequence of images and steering commands recorded while

observing a human driver

Phases of machine learning:

1. Training: A training set of examples of correct behavior is analyzed and of the
newly learnt knowledge is stored. is form of rules.

2. Validation: The rules are checked and, if necessary, additional training is given.

3. Application: The rules are used in responding to some new situation.

Fig 3.1 Phases of ML

Applications of machine learning
1. In retail business, machine learning is used to study consumer behaviour
2. In finance, banks analyze their past data to build models to use in credit applications, fraud
detection, and the stock market.
3. In manufacturing, learning models are used for optimization, control, and troubleshooting. 3 4.
In medicine, learning programs are used for medical diagnosis.
5. In telecommunications, call patterns are analyzed for network optimization and maximizing the
quality of service.
6. In science, large amounts of data in physics, astronomy, and biology can only be analyzed fast
enough by computers. The World Wide Web is huge; it is constantly growing and searching for
relevant information cannot be done manually.

Dr. M. SUPRIYA,
AP/CSE
 7. In artificial intelligence, it is used to teach a system to learn and adapt to changes so that the
system designer need not foresee and provide solutions for all possible situations.
8. It is used to find solutions to many problems in vision, speech recognition, and robotics.
How machines learn
Basic components of learning process
The learning process, whether by a human or a machine, can be divided into four components,
namely, data storage, abstraction, generalization and evaluation. Figure 3.2 illustrates the various
components and the steps involved in the learning process.

Fig 3.2 components of learning process

1. Data storage
Facilities for storing and retrieving huge amounts of data are an important component
of the learning process
2. Abstraction
Abstraction is the process of extracting knowledge about stored data
3. Generalization
The term generalization describes the process of turning the knowledge about stored
data into a form that can be utilized for future action.
4. Evaluation
It is the process of giving feedback to the user to measure the utility of the learned
knowledge.
Different types of learning
1. Supervised learning
2. Unsupervised learning
3. Semi-Supervised Learning
4. Reinforcement learning
1. Supervised learning
 Supervised learning is the machine learning task of learning a function that maps an
input to an output based on example input-output pairs.
 In supervised learning, each example in the training set is a pair consisting of an input
object (typically a vector) and an output value.
 A supervised learning algorithm analyzes the training data and produces a function,
which can be used for mapping new examples. In the optimal case, the function will
correctly determine the class labels for unseen instances.
 Both classification and regression problems are supervised learning problems.

Dr. M. SUPRIYA,
AP/CSE
 Fig. Supervised learning

Example
Consider the following data regarding patients entering a clinic. The data consists of the
gender and age of the patients and each patient is labeled as “healthy” or “sick”.

2. Unsupervised learning
 Unsupervised learning is a type of machine learning algorithm used to draw
inferences from datasets consisting of input data without labeled responses.
 In unsupervised learning algorithms, a classification or categorization is not included
in the observations.
 The most common unsupervised learning method is cluster analysis, which is used for
exploratory data analysis to find hidden patterns or grouping in data.
Example
Consider the following data regarding patients entering a clinic. The data consists of the
gender and age of the patients.

3 .Semi-Supervised Learning

Dr. M. SUPRIYA,
AP/CSE
 Semi-Supervised learning is a type of Machine Learning algorithm that represents the
intermediate ground between Supervised and Unsupervised learning algorithms. It uses the
combination of labeled and unlabeled datasets during the training period.
 Semi-supervised learning is motivated by its practical value in learning
faster, better and cheaper.
 In many real world applications, it is relatively easy to acquire a large
amount of unlabeled data x.
 Semi-supervised learning sometimes enables predictive model testing at
reduced cost.
 Semi-supervised classification : Training on labeled data exploits additional
unlabeled data, frequently resulting in a more accurate classifier.
 Semi-supervised clustering : Uses small amount of labeled data to aid and
bias the clustering of unlabeled data.

For example, documents can be crawled from the Web, images can be obtained surveillance cameras,
and speech can be collected from broadcast.

4 Reinforcement learning
 This is somewhere between supervised and unsupervised learning.
 User will get immediate feedback in supervised learning and no feedback from
unsupervised learning. But in the reinforced learning, you will get delayed scalar
feedback.

Fig:3.4 Reinforcement learning

 Reinforcement learning is the problem of getting an agent to act in the world so as to

maximize its rewards.
 A learner (the program) is not told what actions to take as in most forms of machine learning,
but instead must discover which actions yield the most reward by trying them. In the most
interesting and challenging cases, actions may affect not only the immediate reward but also
the next situations and, through that, all subsequent rewards.
 Example :Consider teaching a dog a new trick: we cannot tell it what to do, but we can
reward/punish it if it does the right/wrong thing. It has to find out what it did that made it get
the reward/punishment.

Dr. M. SUPRIYA,
AP/CSE
 Fig:3.5 Example for Reinforcement learning

2.2 LINEAR REGRESSION:

 Linear Regression is a supervised and statistical ML method that is used for predictive analysis.
This uses the relationship between the data-points to draw a straight line through them.
 Linear regression makes predictions for continuous/real or numeric variables such as sales, age,
price, income etc.
 Linear regression algorithm shows a linear relationship between a dependent (y) and one or
more independent (y) variables.

Dr. M. SUPRIYA,
AP/CSE
  Regression finds how the value of the dependent variable is changing according to the value
of the independent variable.
 The model provides a sloped straight line representing the relationship betweenthe variables.
 The mathematical equation of simple linear regression is given below:
Here,

o Yi is the predicted output for the instance i and is the dependent or explained variable.
o ẞ0 is the intercept of the line while ẞ1 is the slope or scaling factor for each input.
o Xi is the independent variable or explanatory variable or predictor or feature that
governs the entire learning process.
o i is the error component.
2.2.1 Least Squares Regression
o “A least-squares regression method is a form of statistical regression analysis that
establishes the relationship between the dependent (Y) and independent variable (X)
through linear line, referred as line of best fit”.
o The least squares method is a statistical procedure to find the best fit for a set of data
points by minimizing the sum of the offsets or residuals of points from the plotted
curve.
o Least squares regression is used to predict the behavior of dependent variables. This
method of regression analysis begins with a set of data points to be plotted on an x-
and y-axis graph.
o If the data shows a leaner relationship between two variables, the line that best fits
this linear relationship is known as a least-squares regression line, which minimizes
the vertical distance from the data points to the regression line.
o The term "least squares" indicates the smallest sum of squares of errors otherwise
known as variance.
o
The least-squares method is often applied in data fitting.
.
There are two basic categories of least-squares problems:
▪ Ordinary or linear least squares: used in statistical regression analysis
▪ Nonlinear least squares: iterative method to approximate the model to a linear model with
each iteration.
Advantages
o The least-squares method of regression analysis is best suited for prediction models
and trend analysis.
o It is best used in the fields of economics, finance, and stock markets wherein the
value of any future variable is predicted with the help of existing variables and the
relationship between the same.
o The least-squares method provides the closest relationship between the variables.
o The difference between the sums of squares of residuals to the line of best fit is
minimal under this method.
o The computation mechanism is simple and easy to apply.

Dr. M. SUPRIYA,
AP/CSE
Disadvantages
o This method relies on establishing the closest relationship between a given s of
variables.
o The computation mechanism is sensitive to the data, and in case of any outliers, the
results may affect severally.
o More exhaustive computation mechanisms are applied for non linear problems.
Least Square Algorithm
 For each (x, y) point calculate x and xy
 Sum all x, y, x and xy, which gives us x, y, x2 and xy
 Calculate Slope b

 Calculate Intercept a:

 Assemble the equation of a line: Y= bx+a

Example 1: The below table give the statistics about the number of hours or rainfall in Chennai and
the number of French fries sold on a week from Monday to Friday in a canteen. Predict the number of
French fries to be prepared on Saturday, if a rainfall of 8 hours is expected.
Hours of rain [Link] French fries sold
2 4
3 5
5 7
7 10
9 15

Solution:
1. For each (x,y) point calculate x2 and xy.
2. Find x, y, x2 and xy.
x y X2 xy
2 4 4 8
3 5 9 15
5 7 25 35
7 10 49 70
9 15 81 135
x=26 y=41 x =168
2
xy=263
3. Find the slope (b)

Dr. M. SUPRIYA,
AP/CSE
b=(263-(26*41))/5)/(168(26*26)/5)=1.5182
4. Calculate the Intercept a

a=(41-(1.5182*26))/5
a=0.3049
5. Form the equation:Y=1.5182x+0.3049
Compute the error
x y Y=1.5182x+0.3049 Error(Y-y)
2 4 3.3413 -0.6587
3 5 4.8595 -0.1405
5 7 7.8959 0.8959
7 10 10.9323 0.9323
9 15 13.9687 -1.0313

Visualizing the line of fit:

Fig 3.5:Line of fit

Number of French fries to be prepared if it rains for 8 hours Substitute x=8 in Y= 1.5182x+0.3049,
then Y=12.45
So, approximately 13 French fries will be sold on Saturday.
2.2.2 Single and Multiple Variables

Dr. M. SUPRIYA,
AP/CSE
• Simple or single linear regression performs regression analysis of two variables The single
independent variable impacts the slope of the regression line.
• Multiple regression is a broader class of regressions that encompasses linear and nonlinear regressions
with multiple explanatory variables.
• Each independent variable in multiple regression has its own coefficient ensure each variable is
weighted appropriately to establish complex connections between variables.
• Two main operations are done in multiple variable regression:
i) Determine the dependent variable based on multiple independent variables
ii) Determine the strength of the relationship is between each variable.
 Multiple regression assumes there is not a strong relationship between each independent
variable.
 It also assumes there is a correlation between each independent variable and the single
dependent variable.
 Each of these relationships is weighted to ensure more impactful independent variables
drive the dependent value by adding a unique regression coefficient to each independent
variable.
 Using multiple variables for regression is more specific calculation than simple linear
regression. More complex relationships can be acquired through multiple linear regression.
 All the multiple variables use multiple slopes to predict the outcome of single target
variableY=a+b1x1+b2x2+...+bnxn
 In the above equation, b1, b2, ..., bn are the slopes for the individual variables x1 , x2…..xn

Assumptions of Linear Regression

 Linear regression have some fundamental assumptions:
 Linearity: There must be a linear relationship between the dependent and independent
variables.
 Homoscedasticity: The residuals must have a constant variance.
 Normality: Normally distributed error
 No Multicollinearity: No high correlation between the independent variables

Dr. M. SUPRIYA,
AP/CSE
 Fig 3.6 Linear vs Multivariate regression

2.2.3 Bayesian Regression:

(REFER NAÏVE BAYES THEOREM)

3 .GRADIENT DESCENT IN MACHINE LEARNING

 Gradient Descent is known as one of the most commonly used optimization algorithms
to train machine learning models by means of minimizing errors between actual and
expected results. Further, gradient descent is also used to train Neural Networks.
 In mathematical terminology, Optimization algorithm refers to the task of
minimizing/maximizing an objective function f(x) parameterized by x. Similarly, in
machine learning, optimization is the task of minimizing the cost function
parameterized by the model's parameters.
 The main objective of gradient descent is to minimize the convex function using
iteration of parameter updates.
 Once these machine learning models are optimized, these models can be used as
powerful tools for Artificial Intelligence and various computer science applications.
 In this tutorial on Gradient Descent in Machine Learning, we will learn in detail about
gradient descent, the role of cost functions specifically as a barometer within Machine
Learning, types of gradient descents, learning rates, etc.

What is Gradient Descent or Steepest Descent?

Gradient descent was initially discovered by "Augustin-Louis Cauchy" in mid of 18th century.
Gradient Descent is defined as one of the most commonly used iterative optimization algorithms of
machine learning to train the machine learning and deep learning models. It helps infinding the
local minimum of a function.

The best way to define the local minimum or local maximum of a function using gradient descent is as
follows:

o If we move towards a negative gradient or away from the gradient of the function at the current
point, it will give the local minimum of that function.

o Whenever we move towards a positive gradient or towards the gradient of the function at the
current point, we will get the local maximum of that function.

Dr. M. SUPRIYA,
AP/CSE
 0000000

This entire procedure is known as Gradient Ascent, which is also known as steepest descent. The main
objective of using a gradient descent algorithm is to minimize the cost function using iteration. To
achieve this goal, it performs two steps iteratively:

o Calculates the first-order derivative of the function to compute the gradient or slope of that
function.

o Move away from the direction of the gradient, which means slope increased from the current
point by alpha times, where Alpha is defined as Learning Rate. It is a tuning parameter in the
optimization process which helps to decide the length of the steps.

Cost-function

The cost function is defined as the measurement of difference or error between actual
values and expected values at the current position and present in the form of a single real number.
It helps to increase and improve machine learning efficiency by providing feedback to this model so
that it can minimize error and find the local or global minimum.

How does Gradient Descent work?

Before starting the working principle of gradient descent, we should know some basic concepts to find
out the slope of a line from linear regression. The equation for simple linear regression is given as:

1. Y=mX+c

Where 'm' represents the slope of the line, and 'c' represents the intercepts on the y-axis.

Dr. M. SUPRIYA,
AP/CSE
The starting point(shown in above fig.) is used to evaluate the performance as it is considered just as an
arbitrary point. At this starting point, we will derive the first derivative or slope and then use a tangent
line to calculate the steepness of this slope. Further, this slope will inform the updates to the parameters
(weights and bias).

The slope becomes steeper at the starting point or arbitrary point, but whenever new parameters are
generated, then steepness gradually reduces, and at the lowest point, it approaches the lowest point,
which is called a point of convergence.

Direction & Learning Rate

These two factors are used to determine the partial derivative calculation of future iteration and allow
it to the point of convergence or local minimum or global minimum.

It is defined as the step size taken to reach the minimum or lowest point. This is typically a
small value that is evaluated and updated based on the behavior of the cost function. If the learning rate
is high, it results in larger steps but also leads to risks of overshooting the minimum. At the same time,
a low learning rate shows the small step sizes, which compromises overall efficiency but gives the
advantage of more precision.

Dr. M. SUPRIYA,
AP/CSE
Types of Gradient Descent

Based on the error in various training models, the Gradient Descent learning algorithm can be divided
into Batch gradient descent, stochastic gradient descent, and mini-batch gradient descent. Let's
understand these different types of gradient descent:

1. Batch Gradient Descent:

Batch gradient descent (BGD) is used to find the error for each point in the training set and update the
model after evaluating all training examples. This procedure is known as the training epoch. In simple
words, it is a greedy approach where we have to sum over all examples for each update.

Advantages of Batch gradient descent:

o It produces less noise in comparison to other gradient descent.

o It produces stable gradient descent convergence.
o It is Computationally efficient as all resources are used for all training samples.

2. Stochastic gradient descent

Stochastic gradient descent (SGD) is a type of gradient descent that runs one training example
per iteration. Or in other words, it processes a training epoch for each example within a dataset and
updates each training example's parameters one at a time.

As it requires only one training example at a time, hence it is easier to store in allocated memory.
However, it shows some computational efficiency losses in comparison to batch gradient systems as it
shows frequent updates that require more detail and speed. Further, due to frequent updates, it is also
treated as a noisy gradient. However, sometimes it can be helpful in finding the global minimum and
also escaping the local minimum.

Advantages of Stochastic gradient descent:

In Stochastic gradient descent (SGD), learning happens on every example, and it consists of a few
advantages over other gradient descent.

o It is easier to allocate in desired memory.

o It is relatively fast to compute than batch gradient descent.
o It is more efficient for large datasets.

3. MiniBatch Gradient Descent:

Mini Batch gradient descent is the combination of both batch gradient descent and stochastic
gradient descent. It divides the training datasets into small batch sizes then performs the updates on
those batches separately.

Splitting training datasets into smaller batches make a balance to maintain the computational
efficiency of batch gradient descent and speed of stochastic gradient descent. Hence, we can achieve a
special type of gradient descent with higher computational efficiency and less noisy gradient descent.

Dr. M. SUPRIYA,
AP/CSE
Advantages of Mini Batch gradient descent:

o It is easier to fit in allocated memory.

o It is computationally efficient.
o It produces stable gradient descent convergence.

Challenges with the Gradient Descent

Although we know Gradient Descent is one of the most popular methods for optimization
problems, it still also has some challenges. There are a few challenges as follows:

1. Local Minima and Saddle Point:

For convex problems, gradient descent can find the global minimum easily, while for non-
convex problems, it is sometimes difficult to find the global minimum, where the machine learning
models achieve the best results.

Whenever the slope of the cost function is at zero or just close to zero, this model stops learning
further. Apart from the global minimum, there occur some scenarios that can show this slop, which is
saddle point and local minimum. Local minima generate the shape similar to the global minimum, where
the slope of the cost function increases on both sides of the current points.

In contrast, with saddle points, the negative gradient only occurs on one side of the point, which
reaches a local maximum on one side and a local minimum on the other side. The name of a saddle
point is taken by that of a horse's saddle.

The name of local minima is because the value of the loss function is minimum at that point in
a local region. In contrast, the name of the global minima is given so because the value of the loss
function is minimum there, globally across the entire domain the loss function.

2. Vanishing and Exploding Gradient

In a deep neural network, if the model is trained with gradient descent and backpropagation,
there can occur two more issues other than local minima and saddle point.

Vanishing Gradients:

Vanishing Gradient occurs when the gradient is smaller than expected. During
backpropagation, this gradient becomes smaller that causing the decrease in the learning rate of earlier

Dr. M. SUPRIYA,
AP/CSE
layers than the later layer of the network. Once this happens, the weight parameters update until they
become insignificant.

Exploding Gradient:

Exploding gradient is just opposite to the vanishing gradient as it occurs when the Gradient is
too large and creates a stable model. Further, in this scenario, model weight increases, and they will be
represented as NaN. This problem can be solved using the dimensionality reduction technique, which
helps to minimize complexity within the model.

4 .CLASSIFICATION MODELS
 The Classification algorithm is a Supervised Learning technique that is used to
identify the category of new observations on the basis of training data.
 In Classification, a program learns from the given dataset or observations and then
classifies new observation into a number of classes or groups. Such as, Yes or No,
0 or 1, Spam or Not Spam, cat or dog, etc.
 Classes can be called as targets/labels or categories.
 Unlike regression, the output variable of Classification is a category, not a value,
such as "Green or Blue", "fruit or animal", etc.
 Since the Classification algorithm is a Supervised learning technique, hence it takes
labeled input data, which means it contains input with the corresponding output.

4.1 Discriminant Functions

 Discriminant Function Analysis is a dimensionality reduction technique that is
commonly used for supervised classification problems.
 It is used for modelling differences in groups i.e., separating two or more classes. It
is used to project the features in higher dimension space into a lower dimension
space.
 For example, we have two classes, and we need to separate them efficiently. Classes
can have multiple features. Using only a single feature to classify them may result
in some overlapping as shown in the below figure. So, we will keep on increasing
the number of features for proper classification.

Dr. M. SUPRIYA,
AP/CSE
Example:
 Suppose we have two sets of data points belonging to two different classes that we
want to classify. As shown in the given 2D graph, when the data points are plotted
on the 2D plane, there’s no straight line that can separate the two classes of the data
points completely. Hence, in this case, LDA (Linear Discriminant Analysis) is used
which reduces the 2D graph into a 1D graph in order to maximize the separability
between the two classes.

 Here, Linear Discriminant Analysis uses both the axes (X and Y) to create a new
axis and projects data onto a new axis in a way to maximize the separation of the
two categories and hence, reducing the 2D graph into a 1D graph.
 Two criteria are used by LDA to create a new axis:
 Maximize the distance between means of the two classes.
 Minimize the variation within each class.

 In the above graph, it can be seen that a new axis (in red) is generated and plotted
in the 2D graph such that it maximizes the distance between the means of the two
classes and minimizes the variation within each class.

Dr. M. SUPRIYA,
AP/CSE
  In simple terms, this newly generated axis increases the separation between the data
points of the two classes. After generating this new axis using the above-mentioned
criteria, all the data points of the classes are plotted on this new axis and are shown
in the figure given below.

 But Linear Discriminant Analysis fails when the mean of the distributions are
shared, as it becomes impossible for LDA to find a new axis that makes both the
classes linearly separable. In such cases, we use non-linear discriminant analysis.

5. PROBABILISTIC DISCRIMINANT FUNCTIONS

 Probabilistic LDA or PLDA is a generative model which assumes that given data
samples are generated from a distribution. We need to find the parameters of model
which best describe the training data.

 The discriminative model refers to a class of models used in Statistical Classification,

mainly used for supervised machine learning. These types of models are also known

as conditional models since they learn the boundaries between classes or labels in a

dataset.
 Discriminative models focus on modeling the decision boundary between classes in a
classification problem. The goal is to learn a function that maps inputs to binary outputs,
indicating the class label of the input. Maximum likelihood estimation is often used to
estimate the parameters of the discriminative model, such as the coefficients of a logistic
regression model or the weights of a neural network.

Dr. M. SUPRIYA,
AP/CSE
Examples of Discriminative Models

 Logistic regression
 Support vector machines(SVMs)
 Traditional neural networks
 Nearest neighbor
 Conditional Random Fields (CRFs)
 Decision Trees and Random Forest

6. LOGISTIC REGRESSION IN MACHINE LEARNING

 Logistic regression is one of the most popular Machine Learning algorithms, which
comes under the Supervised Learning technique. It is used for predicting the
categorical dependent variable using a given set of independent variables.
 Logistic regression predicts the output of a categorical dependent variable.
Therefore the outcome must be a categorical or discrete value. It can be either Yes
or No, 0 or 1, true or False, etc. but instead of giving the exact value as 0 and 1, it
gives the probabilistic values which lie between 0 and 1.
 Logistic Regression is much similar to the Linear Regression except that how they
are used. Linear Regression is used for solving Regression problems, whereas
Logistic regression is used for solving the classification problems.
 In Logistic regression, instead of fitting a regression line, we fit an "S" shaped
logistic function, which predicts two maximum values (0 or 1).
 The curve from the logistic function indicates the likelihood of something such as
whether the cells are cancerous or not, a mouse is obese or not based on its weight,
etc.
 Logistic Regression is a significant machine learning algorithm because it has the
ability to provide probabilities and classify new data using continuous and discrete
datasets.
 Logistic regression is a simple and more efficient method for binary and linear
classification problems.
 Logistic Regression can be used to classify the observations using different types
of data and can easily determine the most effective variables used for the
classification. The below image is showing the logistic function:

Dr. M. SUPRIYA,
AP/CSE
Logistic Function (Sigmoid Function):

 The sigmoid function is a mathematical function used to map the predicted values to
probabilities.
 It maps any real value into another value within a range of 0 and 1.
 The value of the logistic regression must be between 0 and 1, which cannot go beyond
this limit, so it forms a curve like the "S" form. The S-form curve is called the
Sigmoid function or the logistic function.
 In logistic regression, we use the concept of the threshold value, which defines the
probability of either 0 or 1. Such as values above the threshold value tends to 1, and a
value below the threshold values tends to 0.

Assumptions for Logistic Regression:

 The dependent variable must be categorical in nature.

 The independent variable should not have multi-collinearity.

Logistic Regression Equation:

 The Logistic regression equation can be obtained from the Linear Regression equation.
The mathematical steps to get Logistic Regression equations are given below:
 We know the equation of the straight line can be written as:

 In Logistic Regression y can be between 0 and 1 only, so for this let's divide the
above equation by (1-y):

Dr. M. SUPRIYA,
AP/CSE
  But we need range between -[infinity] to +[infinity], then take logarithm of the
equation it will become:

 The above equation is the final equation for Logistic Regression.

Type of Logistic Regression:

 On the basis of the categories, Logistic Regression can be classified into three types:
 Binomial: In binomial Logistic regression, there can be only two possible types of
dependent variables, such as 0 or 1, Pass or Fail, etc.
 Multinomial: In multinomial Logistic regression, there can be 3 or more possible
unordered types of the dependent variable, such as "cat", "dogs", or "sheep"
 Ordinal: In ordinal Logistic regression, there can be 3 or more possible ordered
types of dependent variables, such as "low", "Medium", or "High".

PROBABLISTIC GENERATIVE MODELS

Generative models are considered a class of statistical models that can generate new data
instances. These models are used in unsupervised machine learning as a means to perform tasks
such as

 Probability and Likelihood estimation,

 Modeling data points
 To describe the phenomenon in data,
 To distinguish between classes based on these probabilities.

 Since these models often rely on the Bayes theorem to find the joint probability,
generative models can tackle a more complex task than analogous discriminative
models.

 These models use probability estimates and likelihood to model data points and
differentiatebetween different class labels present in a dataset.
 Unlike discriminative models, these modelscan also generate new data points.

 However, they also have a major drawback – If there is a presence of outliers in the dataset,
then it affects these types of models to a significant extent.

Dr. M. SUPRIYA,
AP/CSE
Examples of Generative Models

 Naïve Bayes
 Bayesian networks
 Markov random fields
 Hidden Markov Models (HMMs)
 Latent Dirichlet Allocation (LDA)
 Generative Adversarial Networks (GANs)
 Autoregressive Model

FIG: Example of generative models

7. NAÏVE BAYES CLASSIFIER ALGORITHM

 Naïve Bayes algorithm is a supervised learning algorithm, which is based on Bayes

theorem and used for solving classification problems.
 It is mainly used in text classification that includes a high-dimensional training dataset.
 Naïve Bayes Classifier is one of the simple and most effective Classification algorithms
which helps in building the fast machine learning models that can make quick
predictions.
 It is a probabilistic classifier, which means it predicts on the basis of the probability of
an object.

Dr. M. SUPRIYA,
AP/CSE
  Some popular examples of Naïve Bayes Algorithm are spam filtration, Sentimental
analysis, and classifying articles.

Why is it called Naïve Bayes?

 The Naïve Bayes algorithm is comprised of two words Naïve and Bayes, Which can be
described as:
 Naïve: It is called Naïve because it assumes that the occurrence of a certain
feature is independent of the occurrence of other features. Such as if the fruit is
identified on the bases of color, shape, and taste, then red, spherical, and sweet
fruit is recognized as an apple. Hence each feature individually contributes to
identify that it is an apple without depending on each other.
 Bayes: It is called Bayes because it depends on the principle of Bayes' Theorem.

Bayes' Theorem:

 Bayes' theorem is also known as Bayes' Rule or Bayes' law, which is used to determine
the probability of a hypothesis with prior knowledge. It depends on the conditional
probability.
 The formula for Bayes' theorem is given as:

Where,

 P(A|B) is Posterior probability: Probability of hypothesis A on the observed event B.

 P(B|A) is Likelihood probability: Probability of the evidence given that the probability
of a hypothesis is true.
 P(A) is Prior Probability: Probability of hypothesis before observing the
evidence.
 P(B) is Marginal Probability: Probability of Evidence.

Working of Naïve Bayes' Classifier:

 Working of Naïve Bayes' Classifier can be understood with the help of the below
example:
 Suppose we have a dataset of weather conditions and corresponding target variable
"Play". So using this dataset we need to decide that whether we should play or not on a

Dr. M. SUPRIYA,
AP/CSE
 particular day according to the weather conditions. So to solve this problem, we need
to follow the below steps:
 Convert the given dataset into frequency tables.
 Generate Likelihood table by finding the probabilities of given features.
 Now, use Bayes theorem to calculate the posterior probability.
 Problem: If the weather is sunny, then the Player should play or not?
Solution: To solve this, first consider the below dataset:

Likelihood table weather condition:

Dr. M. SUPRIYA,
AP/CSE
Applying Bayes’ theorem:

P(Yes|Sunny)= P(Sunny|Yes)*P(Yes)/P(Sunny)

P(Sunny|Yes)= 3/10= 0.3

P(Sunny)= 0.35

P(Yes)=0.71

So P(Yes|Sunny) = 0.3*0.71/0.35= 0.60

P(No|Sunny)= P(Sunny|No)*P(No)/P(Sunny)

P(Sunny|NO)= 2/4=0.5

P(No)= 0.29

P(Sunny)= 0.35

So P(No|Sunny)= 0.5*0.29/0.35 = 0.41

So as we can see from the above calculation that P(Yes|Sunny)>P(No|Sunny)

Hence on a Sunny day, Player can play the game.

Advantages of Naïve Bayes Classifier:

 Naïve Bayes is one of the fast and easy ML algorithms to predict a class of datasets.
 It can be used for Binary as well as Multi-class Classifications.
 It performs well in Multi-class predictions as compared to the other Algorithms.
 It is the most popular choice for text classification problems.

Disadvantages of Naïve Bayes Classifier:

 Naive Bayes assumes that all features are independent or unrelated, so it cannot learn
the relationship between features.

Applications of Naïve Bayes Classifier:

 It is used for Credit Scoring.

 It is used in medical data classification.
 It can be used in real-time predictions because Naïve Bayes Classifier is an eager
learner.
 It is used in Text classification such as Spam filtering and Sentiment analysis.

Dr. M. SUPRIYA,
AP/CSE
Types of Naïve Bayes Model:

There are three types of Naive Bayes Model, which are given below:

o Gaussian: The Gaussian model assumes that features follow a normal distribution. This
means if predictors take continuous values instead of discrete, then the model assumes
that these values are sampled from the Gaussian distribution.
o Multinomial: The Multinomial Naïve Bayes classifier is used when the data is
multinomial distributed. It is primarily used for document classification problems, it
means a particular document belongs to which category such as Sports, Politics,
education, etc.
The classifier uses the frequency of words for the predictors.
o Bernoulli: The Bernoulli classifier works similar to the Multinomial classifier, but the
predictor variables are the independent Booleans variables. Such as if a particular word
is present or not in a document. This model is also famous for document classification
tasks.

8 MAXIMUM MARGIN CLASSIFIER

Support Vector Machine Algorithm(SVM)

Support Vector Machine or SVM is one of the most popular Supervised Learning
algorithms, which is used for Classification as well as Regression problems. However, primarily,
it is used for Classification problems in Machine Learning.
The goal of the SVM algorithm is to create the best line or decision boundary that can segregate
n-dimensional space into classes so that we can easily put the new data point in thecorrect
category in the future. This best decision boundary is called a hyperplane.
A support vector machine (SVM) is a machine learning algorithm that uses supervised learning
models to solve complex classification, regression, and outlier detection problems by performing
optimal data transformations that determine boundaries between data points based on predefined
classes, labels, or outputs.
The popularity of SVMs is likely due to their amenability to theoretical analysis, their flexibility
in being applied to a wide variety of tasks, including structured prediction problems.
SVM chooses the extreme points/vectors that help in creating the hyperplane. These extreme
cases are called as support vectors, and hence algorithm is termed as Support Vector Machine.
Consider the below diagram in which there are two different categories that are classified using
a decision boundary or hyperplane:

Dr. M. SUPRIYA,
AP/CSE
Example:
SVM can be understood with the example that we have used in the KNN classifier. Suppose
we see a strange cat that also has some features of dogs, so if we want a model that can
accurately identify whether it is a cat or dog, so such a model can be created by using the SVM
algorithm. We will first train our model with lots of images of cats and dogs so that it can learn
about different features of cats and dogs, and then we test it with this strange creature. So as
support vector creates a decision boundary between these two data (cat and dog) and choose
extreme cases (support vectors), it will see the extreme case of cat and dog. On the basis of the
support vectors, it will classify it as a cat. Consider the below diagram:

SVM algorithm can be used for Face detection, image classification, text categorization, etc.
Types of SVM
SVM can be of two types:
Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can
be classified into two classes by using a single straight line, then such data is termed as
linearly separable data, and classifier is used called as Linear SVM classifier.
Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if a
dataset cannot be classified by using a straight line, then such data is termed as non-linear
data and classifier used is called as Non-linear SVM classifier.

Dr. M. SUPRIYA,
AP/CSE
Hyperplane and Support Vectors in the SVM algorithm:
Hyperplane: There can be multiple lines/decision boundaries to segregate the classes in n-
dimensional space, but we need to find out the best decision boundary that helps to classifythe
data points. This best boundary is known as the hyperplane of SVM.
The dimensions of the hyperplane depend on the features present in the dataset, which meansif
there are 2 features (as shown in image), then hyperplane will be a straight line. And if there are
3 features, then hyperplane will be a 2-dimensional plane.
We always create a hyperplane that has a maximum margin, which means the maximumdistance
between the data points.

Support Vectors: The data points or vectors that are the closest to the hyperplane and which
affect the position of the hyperplane are termed as Support Vector. Since these vectors support
the hyperplane, hence called a Support vector.

How does SVM works?

Linear SVM:
The working of the SVM algorithm can be understood by using an example. Suppose we have
a dataset that has two tags (green and blue), and the dataset has two features x1 and [Link] want
a classifier that can classify the pair(x1, x2) of coordinates in either green or blue. Consider the
below image:

So, as it is 2-d space so by just using a straight line, we can easily separate these two classes.
But there can be multiple lines that can separate these classes. Consider the below image:

Hence, the SVM algorithm helps to find the best line or decision boundary; this best boundary
or region is called a hyperplane. SVM algorithm finds the closest point of the linesfrom both
the classes. These points are called support vectors. The distance between the vectors and the
hyperplane is called margin. And the goal of SVM is to maximize this margin. The hyperplane

Dr. M. SUPRIYA,
AP/CSE
with maximum margin is called the optimal hyperplane.

Non-Linear SVM:
If data is linearly arranged, then we can separate it by using a straight line, but for non-linear
data, we cannot draw a single straight line. Consider the below image:

So, to separate these data points, we need to add one more dimension. For linear data, we
have used two dimensions x and y, so for non-linear data, we will add a third dimension z. It
can be calculated as:
z=x2 +y2

Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis. If we convert
it in 2d space with z=1, then it will become as:

Dr. M. SUPRIYA,
AP/CSE
Hence we get a circumference of radius 1 in case of non-linear data.
Python Implementation of Support Vector Machine
Now we will implement the SVM algorithm using Python. Here we will use the same dataset
user data, which we have used in Logistic regression and KNN classification.
Data Pre-processing step
Till the Data pre-processing step, the code will remain the same. Below is the code:
#Data Pre-processing Step
# importing libraries
import numpy as nm
import [Link] as mtp
import pandas as pd
#importing datasets
data_set= pd.read_csv('user_data.csv')

#Extracting Independent and dependent Variable

x= data_set.iloc[:, [2,3]].values
y= data_set.iloc[:, 4].values

# Splitting the dataset into training and test set.

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test= train_test_split(x, y, test_size= 0.25, random_state=0)
#feature Scaling

Dr. M. SUPRIYA,
AP/CSE
from [Link] import StandardScaler
st_x= StandardScaler()
x_train= st_x.fit_transform(x_train)
x_test= st_x.transform(x_test)

9 .Decision Tree Classification Algorithm

o Decision Tree is a Supervised learning technique that can be used for both
classification and Regression problems, but mostly it is preferred for solving
Classification problems. It is a tree-structured classifier, where internal nodes represent
the features of a dataset, branches represent the decision rules and each leaf node
represents the outcome.
o In a Decision tree, there are two nodes, which are the Decision Node and Leaf Node.
o Decisionnodes are used to make any decision and have multiple branches, whereas Leaf
nodes are the output of those decisions and do not contain any further branches.
o The decisions or the test are performed on the basis of features of the given dataset.
o It is a graphical representation for getting all the possible solutions to a
problem/decisionbased on given conditions.
o It is called a decision tree because, similar to a tree, it starts with the root node, which
expandson further branches and constructs a tree-like structure.
o In order to build a tree, we use the CART algorithm, which stands for Classification
andRegression Tree algorithm.
o A decision tree simply asks a question, and based on the answer (Yes/No), it further
split thetree into subtrees.
o Below diagram explains the general structure of a decision tree:

Dr. M. SUPRIYA,
AP/CSE
Why use Decision Trees?

There are various algorithms in Machine learning, so choosing the best algorithm for the given
datasetand problem is the main point to remember while creating a machine learning model.
Below are the two reasons for using the Decision tree:

o Decision Trees usually mimic human thinking ability while making a decision, so it is
easy tounderstand.
o The logic behind the decision tree can be easily understood because it shows a
tree-likestructure.

Decision Tree Terminologies

Root Node: Root node is from where the decision tree starts. It represents the entire dataset,
whichfurther gets divided into two or more homogeneous sets.
Leaf Node: Leaf nodes are the final output node, and the tree cannot be segregated further
aftergetting a leaf node.
Splitting: Splitting is the process of dividing the decision node/root node into sub-nodes
accordingto the given conditions.
Branch/Sub Tree: A tree formed by splitting the tree.
Pruning: Pruning is the process of removing the unwanted branches from the tree.
Parent/Child node: The root node of the tree is called the parent node, and other nodes are
calledthe child nodes.

How does the Decision Tree algorithm Work?

In a decision tree, for predicting the class of the given dataset, the algorithm starts from the root
node of the tree. This algorithm compares the values of root attribute with the record (real
dataset) attribute and, based on the comparison, follows the branch and jumps to the next node.

For the next node, the algorithm again compares the attribute value with the other sub-nodes and
movefurther. It continues the process until it reaches the leaf node of the tree. The complete
process can be better understood using the

Dr. M. SUPRIYA,
AP/CSE
  Step-1: Begin the tree with the root node, says S, which contains the complete dataset.
 Step-2: Find the best attribute in the dataset using Attribute Selection Measure (ASM).
 Step-3: Divide the S into subsets that contains possible values for the best attributes.
 Step-4: Generate the decision tree node, which contains the best attribute.
 Step-5: Recursively make new decision trees using the subsets of the dataset created in step -
3. Continue this process until a stage is reached where you cannot further classify the nodes
and called the final node as a leaf node.

Example: Suppose there is a candidate who has a job offer and wants to decide whether he should
accept the offer or Not. So, to solve this problem, the decision tree starts with the root node (Salary
attribute by ASM). The root node splits further into the next decision node (distance from the office)
and one leaf node based on the corresponding labels. The next decision node further gets split into one
decision node (Cab facility) and one leaf node. Finally, the decision node splits into two leaf nodes
(Accepted offers and Declined offer). Consider the below diagram:

Attribute Selection Measures

While implementing a Decision tree, the main issue arises that how to select the best attribute for the
root node and for sub-nodes. So, to solve such problems there is a technique which is called as Attribute
selection measure or ASM. By this measurement, we can easily select the best attribute for the nodes
of the tree. There are two popular techniques for ASM, which are:

o Information Gain
o Gini Index

1. Information Gain:

o Information gain is the measurement of changes in entropy after the segmentation of a dataset
based on an attribute.
o It calculates how much information a feature provides us about a class.
o According to the value of information gain, we split the node and build the decision tree.

Dr. M. SUPRIYA,
AP/CSE
 o A decision tree algorithm always tries to maximize the value of information gain, and a
node/attribute having the highest information gain is split first. It can be calculated using the
below formula:

Information Gain= Entropy(S)- [(Weighted Avg) *Entropy(each feature)

(Or)

Entropy: Entropy is a metric to measure the impurity in a given attribute. It specifies randomness in
data. Entropy can be calculated as:

Entropy(s)= -P(yes)log2 P(yes)- P(no) log2 P(no)

(Or)

Where,

o S= Total number of samples

o P(yes)= probability of yes
o P(no)= probability of no

2. Gini Index:

o Gini index is a measure of impurity or purity used while creating a decision tree in the
CART(Classification and Regression Tree) algorithm.
o An attribute with the low Gini index should be preferred as compared to the high Gini index.
o It only creates binary splits, and the CART algorithm uses the Gini index to create binary splits.
o Gini index can be calculated using the below formula:
Gini Index= 1- ∑ P 2
Pruning: Getting anj O
j
ptimal Decision tree

“Pruning is a process of deleting the unnecessary nodes from a tree in order to get the optimal
decision tree”.

A too-large tree increases the risk of overfitting, and a small tree may not capture all the important
features of the dataset. Therefore, a technique that decreases the size of the learning tree without
reducing accuracy is known as Pruning. There are mainly two types of tree pruning technology used:

o Cost Complexity Pruning

o Reduced Error Pruning.

3.7.1 Algoritm for decision tree

Dr. M. SUPRIYA,
AP/CSE
There are many algorithms there to build a decision tree. They are

1. CART (Classification and Regression Trees) — This makes use of Gini impurity as the
metric.

2. ID3 (Iterative Dichotomiser 3) — This uses entropy and information gain as metric.

Classification using the ID3 algorithm

Consider whether a dataset based on which we will determine whether to play football or not.

Dr. M. SUPRIYA,
AP/CSE
Here There are for independent variables to determine the dependent variable. The independent
variables are Outlook, Temperature, Humidity, and Wind. The dependent variable is whether to play
football or not.
As the first step, we have to find the parent node for our decision tree. For that follow the steps:

Step 1:Find the entropy of the class variable.

E(S) = -[(9/14)log(9/14) + (5/14)log(5/14)] = 0.94

note: Here typically we will take log to base [Link] total there are 14 yes/no. Out of which 9 yes and 5
[Link] on it we calculated probability above.

From the above data for outlook we can arrive at the following table easily

Step2:Now we have to calculate average weighted entropy.

ie, we have found the total of weights of each feature multiplied by probabilities.

E(S, outlook) = (5/14)*E(3,2) + (4/14)*E(4,0) + (5/14)*E(2,3) = (5/14)(-(3/5)log(3/5)-(2/5)log(2/5))+

(4/14)(0) + (5/14)((2/5)log(2/5)-(3/5)log(3/5)) = 0.693

Step 3:The next step is to find the information gain. It is the difference between parent entropy and
average weighted entropy we found above.
IG(S, outlook) = 0.94 - 0.693 = 0.247
Similarly find Information gain for Temperature, Humidity, and Windy.
IG(S, Temperature) = 0.940 - 0.911 = 0.029
IG(S, Humidity) = 0.940 - 0.788 = 0.152
IG(S, Windy) = 0.940 - 0.8932 = 0.048

Step 4:Now select the feature having the largest entropy gain. Here it is Outlook. So it forms the first
node(root node) of our decision tree.

Now our data look as follows

Dr. M. SUPRIYA,
AP/CSE
Since overcast contains only examples of class ‘Yes’ we can set it as yes. That means If outlook is
overcast football will be played. Now our decision tree looks as follows.

Step 5:The next step is to find the next node in our decision tree.

Now we will find one under sunny. We have to determine which of the following Temperature,
Humidity or Wind has higher information gain.

Calculate parent entropy E(sunny)

E(sunny) = (-(3/5)log(3/5)-(2/5)log(2/5)) = 0.971.
Now Calculate the information gain of Temperature. IG(sunny, Temperature)

E(sunny, Temperature) = (2/5)*E(0,2) + (2/5)*E(1,1) + (1/5)*E(1,0)=2/5=0.4

Now calculate information gain.

Dr. M. SUPRIYA,
AP/CSE
IG(sunny, Temperature) = 0.971–0.4 =0.571
Similarly we get
IG(sunny, Humidity) = 0.971
IG(sunny, Windy) = 0.020
Here IG(sunny, Humidity) is the largest value. So Humidity is the node that comes under sunny.

For humidity from the above table, we can say that play will occur if humidity is normal and will not
occur if it is high. Similarly, find the nodes under rainy.

Note: A branch with entropy more than 0 needs further splitting.

Finally, our decision tree will look as below:

Chapter 3 Classification using CART algorithm

Classification using CART is similar to it. But instead of entropy, we use Gini impurity.

So as the first step we will find the root node of our decision tree. For that Calculate the Gini
index of the class variable

Gini(S) = 1 - [(9/14)² + (5/14)²] = 0.4591

As the next step, we will calculate the Gini gain. For that first, we will find the average weighted
Gini impurity of Outlook, Temperature, Humidity, and Windy.

First, consider case of Outlook

Dr. M. SUPRIYA,
AP/CSE
Gini(S, outlook) = (5/14)gini(3,2) + (4/14)*gini(4,0)+ (5/14)*gini(2,3) = (5/14)(1 - (3/5)² - (2/5)²) +
(4/14)*0 + (5/14)(1 - (2/5)² - (3/5)²)= 0.171+0+0.171 = 0.342

Gini gain (S, outlook) = 0.459 - 0.342 = 0.117

Gini gain(S, Temperature) = 0.459 - 0.4405 = 0.0185

Gini gain(S, Humidity) = 0.459 - 0.3674 = 0.0916

Gini gain(S, windy) = 0.459 - 0.4286 = 0.0304

Choose one that has a higher Gini gain. Gini gain is higher for outlook. So we can choose it as our root
node.

Note:Repeat the same steps we used in the ID3 algorithm.

3.7.2 Advantages of the Decision Tree

o It is simple to understand as it follows the same process which a human follow while making
any decision in real-life.
o It can be very useful for solving decision-related problems.
o It helps to think about all the possible outcomes for a problem.
o There is less requirement of data cleaning compared to other algorithms.

3.7.3 Disadvantages of the Decision Tree

o The decision tree contains lots of layers, which makes it complex.

o It may have an overfitting issue, which can be resolved using the Random Forest algorithm.
o For more class labels, the computational complexity of the decision tree may increase

[Link] Forest Algorithm

Random Forest is a popular machine learning algorithm that belongs to the supervised
learningtechnique. It can be used for both Classification and Regression problems in ML. It is
based on the concept of ensemble learning, which is a process of combining multiple classifiers
to solve a complexproblem and to improve the performance of the model.

As the name suggests, "Random Forest is a classifier that contains a number of decision trees
on various subsets of the given dataset and takes the average to improve the predictive
accuracy of thatdataset." Instead of relying on one decision tree, the random forest takes the
prediction from each treeand based on the majority votes of predictions, and it predicts the final
output.

The greater number of trees in the forest leads to higher accuracy and prevents the
problem ofoverfitting.

The below diagram explains the working of the Random Forest algorithm:

Dr. M. SUPRIYA,
AP/CSE
 Fig.3.8.1 Random forest

Assumptions for Random Forest

Since the random forest combines multiple trees to predict the class of the dataset, it is possible
that some decision trees may predict the correct output, while others may not. But together, all
the trees predict the correct output. Therefore, below are two assumptions for a better Random
forest classifier:

o There should be some actual values in the feature variable of the dataset so that the
classifiercan predict accurate results rather than a guessed result.

o The predictions from each tree must have very low correlations.

Why use Random Forest?

Below are some points that explain why we should use the Random Forest algorithm:

 It takes less training time as compared to other algorithms.

 It predicts output with high accuracy, even for the large dataset it runs efficiently.
 It can also maintain accuracy when a large proportion of data is missing.

How does Random Forest algorithm work?

Random Forest works in two-phase first is to create the random forest by combining N decision
tree, and second is to make predictions for each tree created in the first phase.

3.8.1 The Working process

Step-1: Select random K data points from the training set.

Step-2: Build the decision trees associated with the selected data points (Subsets).

Step-3: Choose the number N for decision trees that you want to build.

Step-4: Repeat Step 1 & 2.

Dr. M. SUPRIYA,
AP/CSE
Step-5: For new data points, find the predictions of each decision tree, and assign the new data
points to the category that wins the majority votes.

The working of the algorithm can be better understood by the below example:

Example: Suppose there is a dataset that contains multiple fruit images. So, this dataset is
given to theRandom forest classifier. The dataset is divided into subsets and given to each
decision tree. During thetraining phase, each decision tree produces a prediction result, and
when a new data point occurs, then based on the majority of results, the Random Forest
classifier predicts the final decision. Consider the below image:

3.8.2 Applications of Random Forest

There are mainly four sectors where Random forest mostly used:

1. Banking: Banking sector mostly uses this algorithm for the identification of loan risk.

2. Medicine: With the help of this algorithm, disease trends and risks of the disease
can beidentified.
3. Land Use: We can identify the areas of similar land use by this algorithm.

4. Marketing: Marketing trends can be identified using this algorithm.

3.8.3 Advantages of Random Forest

o Random Forest is capable of performing both Classification and Regression tasks.

o It is capable of handling large datasets with high dimensionality.
o It enhances the accuracy of the model and prevents the overfitting issue.

3.8.4 Disadvantages of Random Forest

o Although random forest can be used for both classification and regression tasks, it is not
moresuitable for Regression tasks.

Dr. M. SUPRIYA,
AP/CSE
 Two marks
1) What is Machine Learning?
Definition : A computer program is said to learn from experience E with respect to some class of tasks
T and performance measure p, if its performance at tasks in T, as measured by P, improves with
experience [Link] learning is programming computers to optimize a performance criterion using
example data or past experience. Application of machine learning methods to large databases is called
data mining.
2) What are the phases of machine learning?
Phases of machine learning:
1. Training: A training set of examples of correct behavior is analyzed and of the
newly learnt knowledge is stored. is form of rules.

2. Validation: The rules are checked and, if necessary, additional training is given.

3. Application: The rules are used in responding to some new situation.

3) What are the different types of learning

1 Supervised learning
2 Unsupervised learning
3 Reinforcement learning
4) Define Supervised learning
o Supervised learning is the machine learning task of learning a function that maps an
input to an output based on example input-output pairs.
o In supervised learning, each example in the training set is a pair consisting of an input
object (typically a vector) and an output value.
o A supervised learning algorithm analyzes the training data and produces a function,
which can be used for mapping new examples. In the optimal case, the function will
correctly determine the class labels for unseen instances.
5) Define Unsupervised learning

 Unsupervised learning is a type of machine learning algorithm used to draw

inferences from datasets consisting of input data without labeled responses.
 In unsupervised learning algorithms, a classification or categorization is not included
in the observations.
 The most common unsupervised learning method is cluster analysis, which is used for
exploratory data analysis to find hidden patterns or grouping in data.
6) What is Semi-Supervised Learning

Semi-Supervised learning is a type of Machine Learning algorithm that represents the

intermediate ground between Supervised and Unsupervised learning algorithms. It uses the
combination of labeled and unlabeled datasets during the training period.
7. Define Reinforcement learning
 This is somewhere between supervised and unsupervised learning.

Dr. M. SUPRIYA,
AP/CSE
  User will get immediate feedback in supervised learning and no feedback from
unsupervised learning. But in the reinforced learning, you will get delayed scalar
feedback.

Reinforcement learning

8. What is Linear regression

 Linear Regression is a supervised and statistical ML method that is used for predictive analysis.
This uses the relationship between the data-points to draw a straight line through them.
 Linear regression makes predictions for continuous/real or numeric variables such as sales, age,
price, income etc.
 Linear regression algorithm shows a linear relationship between a dependent (y) and one or
more independent (y) variables.

9. Define Bayesian Regression:

Bayesian is just an approach to defining and estimating statistical models. Bayesian

Regression can be very useful when we have insufficient data in the dataset or the data is poorly
distributed.

The output of a Bayesian Regression model is obtained from a probability distribution, as

compared to regular regression techniques where the output is just obtained from a single value
of each attributeIn order to explain Naive Bayes we need to first explain Bayes theorem. The
foundationof Bayes theorem is conditional probability (figure 1). In fact, Bayes theorem (figure
1) is just an alternate or reverse way to calculate conditional probability. When the joint
probability, P(A∩B), is hardto calculate or if the inverse or Bayes probability, P(B|A), is easier
to calculate then Bayes theorem canbe applied.

Dr. M. SUPRIYA,
AP/CSE
10) Difference between single and multiple regression

11) What is Gradient Descent or Steepest Descent?

Gradient descent was initially discovered by "Augustin-Louis Cauchy" in mid of 18th century.
Gradient Descent is defined as one of the most commonly used iterative optimization algorithms of
machine learning to train the machine learning and deep learning models. It helps in finding the local
minimum of a function.

12) What is the use of Logistic Regression in Machine Learning

 Logistic regression is one of the most popular Machine Learning algorithms, which
comes under the Supervised Learning technique. It is used for predicting the
categorical dependent variable using a given set of independent variables.
 Logistic regression predicts the output of a categorical dependent variable.
Therefore the outcome must be a categorical or discrete value. It can be either Yes
or No, 0 or 1, true or False, etc. but instead of giving the exact value as 0 and 1, it
gives the probabilistic values which lie between 0 and 1.

13) What is meant by Decision Tree Classification

Dr. M. SUPRIYA,
AP/CSE
o Decision Tree is a Supervised learning technique that can be used for both
classification and Regression problems, but mostly it is preferred for solving
Classification problems. It is a tree- structured classifier, where internal nodes
represent the features of a dataset, branches represent the decision rules and each
leaf node represents the outcome.
o In a Decision tree, there are two nodes, which are the Decision Node and Leaf Node.
Decisionnodes are used to make any decision and have multiple branches, whereas Leaf
nodes are the output of those decisions and do not contain any further branches.

Dr. M. SUPRIYA,
AP/CSE
 UNIT III

Ensemble Techniques And Unsupervised Learning

Syllabus
Combining multiple learners: Model combination schemes, Voting, Ensemble
Learning - bagging, boosting, stacking, Unsupervised learning: K. means, Instance
Based Learning: KNN, Gaussian mixture models and Expectation maximization.

3.1 Combining Multiple Learners

 When designing a learning machine, we generally make some choices like parameters
of machine, training data, representation, etc. This implies some sort of variance in
performance. For example, in a classification setting, we can use a parametric classifier
or in a multilayer perceptron, we should also decide on the number of hidden units.
 Each learning algorithm dictates a certain model that comes with a set of assumptions.
This inductive bias leads to error if the assumptions do not hold for the data.
 Different learning algorithms have different accuracies. The no free lunch theorem
asserts that no single learning algorithm always achieves the best performance in any
domain. They can be combined to attain higher accuracy.
 Data fusion is the process of fusing multiple records representing the same real-world
object into a single, consistent, and clean representation. Fusion of data for improving
prediction accuracy and reliability is an important problem in machine learning.
 Combining different models is done to improve the performance of deep learning
models. Building a new model by combination requires less time, data, and
computational resources. The most common method to combine models is by averaging
multiple models, where taking a weighted average improves the accuracy.

Dr. M. SUPRIYA,
AP/CSE
Voting

 The simplest way to combine multiple classifiers is by voting, which corresponds to

take a linear combination of the learners. Voting is an ensemble machine learning
algorithm.
 For regression, a voting ensemble involves make a prediction that the average of
multiple other regression models.
Hard Voting – The class that receives the majority of votes is selected as the final
prediction. It is commonly used in classification problems. In regression, it predicts the
average of the individual predictions.
Soft Voting – Weighted average of predicted probabilities is used to make the final
prediction. It is suitable when classifiers provide probability estimates. In other words,
for each class, it sums the predicted probabilities and predicts the class with the highest
sum.

Fig:voting

 In this methods, the first step is to create multiple classification/ regression models
using some training dataset. Each base model can be created using different splits of

Dr. M. SUPRIYA,
AP/CSE
 the same training dataset and same algorithm, or using the same dataset with different
algorithms, or any other methods.

Base learners with model combiner

 When combining multiple independent and diverse decisions each of which is at least
more accurate than random guessing, random errors cancel each other out, and correct
decisions are reinforced. Human ensembles are demonstrably better.
 Use a single, arbitrary learning algorithm but manipulate training data to make it learn
multiple models.

Bagging

 Bagging is also called Bootstrap aggregating. Bagging and boosting are meta-
algorithms that pool decisions from multiple classifiers. It creates ensembles by
repeatedly randomly resampling he training data.
 Bagging was the first effective method of ensemble learning and is one of the simplest
methods of arching. The meta – algorithm, which is a special case of the model
averaging, was originally designed for classification and is usually applied to decision
tree models, but it can be used with any type of model for classification or regression
Applications of Bagging
[Link]
[Link]
[Link]
[Link]

Dr. M. SUPRIYA,
AP/CSE
 Ensemble classifiers such as bagging, boosting and model averaging are known to have
improved accuracy and robustness over a single model. Although unsupervised models,
such as clustering, do not directly generate label prediction for each individual, they
provide useful constraints for the joint prediction of a set of related objects.
 For given a training set of size n, create m samples of size n by drawing n examples
from the original data, with replacement. Each bootstrap sample will on average contain
63.2% of the unique training examples, the rest are replicates. It combines the m
resulting models using simple majority vote.
 In particular, on each round, the base learner is trained on what is often called a
“bootstrap replicate” of the original training set. Suppose the training set consists of
n examples. Then a bootstrap replicate is a new training set that also consists of n
examples, and which is formed by repeatedly selecting uniformly at random and with
replacement n examples from the original training set. This means that the same
example may appear multiple times in the bootstrap replicate, or it may appear not at
all.
 It also decreases error by decreasing the variance in the results due to unstable learners,
algorithms (like decision trees) whose out put can change dramatically when the
training date is slightly changed.
 Pseudocode:
1. Given Training data (x1,y1)........... (xm,ym)
2. For t = 1,… T :
a. Form bootstrap replicate dataset St by selecting m random examples from the
training set with replacement.
b. Let ht be the result of training base learn4ng algorithm on st

Dr. M. SUPRIYA,
AP/CSE
 3. Output combined classifier :

H(×) = majority (h1(×) ...........hT(×)

Bagging Steps :

1. Suppose there are N observations and M features in training data set. A sample
from training data set is taken randomly with replacement.
2. A subset of M features is selected randomly and whichever feature gives the best
split is used to split the node iteratively.
3. The tree is grown to the largest.
4. Above steps are repeated n times and predictions is given based on the aggregation
of predictions from n number of trees.

Advantages of Bagging :

1. Reduces over – fitting of the model.

2. Handles higher dimensionality data very well.
3. Maintains accuracy for missing data.

Disadvantages of Bagging:

1. Since final prediction is based on the mean predictions from subset trees, it won’t
give precise values for the classification and regression model.

Boosting

 Boosting is an ensemble learning method that combines a set of weak learners into a
strong learner to minimize training errors.
 In boosting, a random sample of data is selected, fitted with a model and then trained
sequentially—that is, each model tries to compensate for the weaknesses of its
predecessor..

 Originally developed by computational learning theorists to guarantee performance

improvements on fitting training data for a weak learner that only needs to generate a
hypothesis with a training accuracy greater than 0.5.

 Final result is the weighted sum of the results of weak classifiers.

Dr. M. SUPRIYA,
AP/CSE
 A learner is weak if it produces a classifier that is only slightly better than random
guessing, while a learner is said to be strong if it produces a classifier that achieves a
low error with high confidence for a given concept.
 Revised to be a practical algorithm, AdaBoost, for building ensembles that empirically
improves generalization performance. Examples are given weights. At each iteration, a
new hypothesis is learned and the examples are reweighted to focus the system on
examples that the most recently learned classifier got wrong.
 Boosting is a bias reduction technique.
 It typically improves the performance of a single tee model.
 A reason for this is that we often cannot construct trees which are sufficiently large due
to thinning out of observations in the terminal nodes.
 Boosting is then a device to come up with a more complex solution by taking linear
combination of trees.
 In presence of high – dimensional predictors, boosting is also very useful as a
regularization technique for additive or interaction modeling.
 To begin, we define an algorithm for finding the rules of thumb, which we call a weak
learner. The boosting algorithm repeatedly calls this weak learner, each time feeding it
a different distribution over the training data.
 Each call generates a weak classifier and we must combine all of these into a single
classifier that, hopefully, is much more accurate than any one of the rules.
 Train a set of weak hypotheses ; h1 ….,[Link] combined hypothesis H is a weighted
majority vote of the T weak hypotheses. During the training, focus on the examples
that are misclassified.

Dr. M. SUPRIYA,
AP/CSE
 Fig. 4.2.1combined hypothesis

AdaBoost:

AdaBoost is an ensemble learning technique that aims to create a strong classifier by

combining the performance of several weak classifiers. A weak classifier is a model that performs
slightly better than random guessing. By iteratively adjusting the weights of misclassified
instances, AdaBoost focuses on the difficult cases that need more attention, thereby improving the
overall accuracy of the model.
Stump

Fig. 4.2.3 Adaboost

 The working of the AdaBoost :

 Creation of the learner.
 Calculation of the total error via the beneath formulation.
Sum of total error = Total number of wrongly predicted instances
 Calculation of performance of the decision stumps.
 Updating the weights in line with the misclassified factors.
Advantages of AdaBoost:
1. Very simple to implement
2. fairly good generalization
3. The prior error need not be known ahead of time.
Disadvantages of AdaBoost:
1. Suboptimal solution

Dr. M. SUPRIYA,
AP/CSE
 2. Can over fit in presence of noise.
Boosting Steps:

1. Draw a random subset of training samples d1 without replacement from the training set
D to train a weak learner C1
2. Draw second random training subset d2 without replacement from the training set add
add 50 percent of the samples that were previously falsely classified / misclassified to
train a weak learner C2
3. Find the training samples d3 in the training set D on which C1 and C2 disagree to train
a third weak learner C3
4. Combine all the weak learners via majority voting.

Advantages of Boosting:
1. Supports different loos function.
2. Works well with interactions.
Disadvantages of Boosting:
1. Prone to over – fitting.
2. Requires careful tuning of different hyper – parameters.
Stacking
 Stacking, sometimes called stacked generalization, is an ensemble machine learning
method that combines multiple heterogeneous base or component models via a meta -
model.
 The base model is trained on the complete training data, and then the meta – model is
trained on the predictions of the base models. The advantage of stacking is the ability
to explore the solution space with different models in the same problem.
 The stacking base model can be visualized in levels and has at least two levels of the
models. The first level typically trains the two or more base learners (can be
heterogeneous) and the second level might be a single meta learner that utilizes the
base models predictions as input and gives the final result as output . A stacked model
can have more than two such levels but increasing the levels doesn’t always guarantee
better performance.
 In the classification tasks, often logistic regression is used as a meta learner, while linear
regression is more suitable as a meta learner for regression – based tasks.
 Stacking is concerned with combining multiple classifiers generated by different
learning algorithms L1,….LN on a single dataset S, which is composed by a feature

Dr. M. SUPRIYA,
AP/CSE
 vectors Si= (xi, ti)

 The stacking process can be broken into two phases:

1. Generate a set of base – level classifiers C1,…..CN where Ci = Bi (S)
2. Train a meta – level classifier to combine the output of the bae – level
classifiers.
 Fig. 4.2.2 shows stacking frame.

Fig.4.2.2 Stacking frame.

 Based on one of two basic observations:

1. Variance reduction: If the training sets are completely independent, it will
always helps to average an ensemble because this will reduce variance without
affecting bias (e.g. – bagging) and reduce sensitivity to individual data points.
2. Bias reduction : For simple models, average of models has much greater
capacity than single model Averaging models can reduce bias substantially by
increasing capacity and control variance by Citting one component at a time.

Dr. M. SUPRIYA,
AP/CSE
3.2.5 Difference between Bagging and Boosting

Sr. No. Bagging Boosting

1. Bagging is a technique that builds Boosting refers to a group of
multiple homogeneous models from algorithms that utilize weighted
Different subsamples of the same averages to make weak learning
training dataset to obtain more algorithms stronger learning
accurate predictions than its algorithms.
individual models
2 Learns them independently from Learns them sequentially in very
each other in parallel adaptative way
3 It helps in reducing variance. It helps in reducing bias and
variance.
4. Every model receives an equal Models are weighted by their
weight. performance.

Clustering
 Cluster analysis or clustering is the task of grouping a set of objects in such a way that
objects in the same group (called a cluster) are more similar (in some sense) to each
other than to those in other groups (clusters).

 Cluster analysis can be a powerful data-mining tool for any organization that needs to
identity discrete groups of customers, sales transaction, or other types of behaviors and

Dr. M. SUPRIYA,
AP/CSE
 things. For example, insurance providers use cluster analysis to detect fraudulent claims
and banks used it for credit scoring.
 Cluster analysis uses mathematical models to discover groups of similar customers
based on the smallest variations among customers within each group.
 Cluster is a group of objects that belongs to the same class. In another words the similar
object are grouped in one cluster and dissimilar are grouped in other cluster.
 Clustering is a process of partitioning a set of data in a set of meaningful subclasses.
Every data in the sub class shares a common trait. It helps a user understand the natural
grouping or structure in a data set.
 Various types of clustering methods are partitioning methods, hierarchical clustering,
fuzzy clustering, density based clustering and model based clustering.
 Cluster anlysis is process of grouping a set of data objects into clusters.

Desirable properties of a clustering algorithm are as follows:

Fig. 4.3.1clustering
1. Scalability (in terms of both time and space )
2. Ability to deal with different data types
3. Minimal requirements for domain knowledge to determine input parameters.
4. Interpretability and usability.
 Clustering of date is a method by which large sets of data are grouped into clusters of
smaller sets of similar data. Clustering can be considered the most important
unsupervised learning problem.
 A cluster is therefore a collection of objects which are “similar” between them and are
dissimilar” to the objects belonging to other clusters. Fig. 4.3.1 shows cluster.
 In this case we easily identify the 4 clusters into which the data can be divided; the
similarity criterion is distance : two or more objects belong to the same cluster if they
are “close” according to a given distance (in this case geometrical distance ) This is
called distance – based clustering.

Raw data Clustering algorithm Clusters of data

Dr. M. SUPRIYA,
AP/CSE
 Fig. 4.3.2 cluster centroid

 Clustering means grouping of data or dividing a large data set into smaller data sets of
some similarity.
 A clustering algorithm attempts to find natural groups components or data based on
some similarity. Also, the clustering algorithm finds the centroid of a group of data sets.

 To determine cluster membership, most algorithms evaluate the distance between a

point and the cluster centroids. The output from a clustering algorithm is basically a
statistical description of the cluster centroids with the number of components in each
cluster.
 Cluster centroid : The centroid of a cluster is a point whose parameter values are the
means of the parameter values of all the points in the cluster. Each cluster has a well
defined centroid.
 Distance : The distance between two points is taken as a common metric to as see the
similarity among he components of population. The commonly used distance measure
is the Euclidean metric which defines the distance between two points p= (p1, p2…)
and q=(q1, q2,…) is given by,

D = ∑ (𝑃𝑖- qi) 2
𝑖=1

 The goal of clustering is to determine the intrinsic grouping in a set of unlabeled data.
But how to decide what constitutes a good clustering? It can be shown that there is no
absolute “best” criterion which would be independent of the final aim of the clustering.
Consequently, it is the user which must supply criterion, in such a way that the result of
the clustering will suit their needs.
 Clustering analysis helps construct meaningful partitioning of a large set of objects
cluster analysis has been widely used in numerous application, including, pattern

Dr. M. SUPRIYA,
AP/CSE
 recognition, data analysis, image processing etc.
 Clustering algorithms may be classified as listed below:
1. Exclusive clustering
2. Overlapping clustering
3. Hierarchical clustering
4. Probabilistic clustering.
 A good clustering method will produce high quality clusters high intra – class similarity
and low inter – class similarity. The quality of a clustering result depends

on both the similarity measure used by the method and its implementation. The quality
of a clustering method is also measured by it’ s ability to discover some or all of the
hidden patterns.
 Clustering techniques types : The major clustering techniques are
a) Partitioning methods
b) Hierarchical methods
c) Density – based methods.

3.2.1 Unsupervised Learning: K-means

K-Means Clustering Algorithm

K-Means Clustering is an unsupervised learning algorithm that is used to solve the

clustering problems in machine learning or data science. In this topic, we will learn what is
K-means clustering algorithm, how the algorithm works, along with the Python
implementation of k-means clustering.

What is K-Means Algorithm?

 K-Means Clustering is an Unsupervised Learning algorithm, which groups the

unlabeled dataset into different clusters. Here K defines the number of pre-defined
clusters that need to be created in the process, as if K=2, there will be two clusters, and
for K=3, there will be three clusters, and so on.
 It is an iterative algorithm that divides the unlabeled dataset into k different clusters
in such a way that each dataset belongs only one group that has similar properties.
 It allows us to cluster the data into different groups and a convenient way to discover
the categories of groups in the unlabeled dataset on its own without the need for any
training.
 It is a centroid-based algorithm, where each cluster is associated with a centroid. The
main aim of this algorithm is to minimize the sum of distances between the data point
and their corresponding clusters.
 The algorithm takes the unlabeled dataset as input, divides the dataset into k-number of
clusters, and repeats the process until it does not find the best clusters. The value of k
should be predetermined in this algorithm.

Dr. M. SUPRIYA,
AP/CSE
The k-means clustering algorithm mainly performs two tasks:

o Determines the best value for K center points or centroids by an iterative process.
o Assigns each data point to its closest k-center. Those data points which are near to the
particular k-center, create a cluster.

Hence each cluster has datapoints with some commonalities, and it is away from other
clusters.

The below diagram explains the working of the K-means Clustering Algorithm:

How does the K-Means Algorithm Work?

The working of the K-Means algorithm is explained in the below steps:

Step-1: Select the number K to decide the number of clusters.

Step-2: Select random K points or centroids. (It can be other from the input dataset).

Step-3: Assign each data point to their closest centroid, which will form the predefined K
clusters.

Step-4: Calculate the variance and place a new centroid of each cluster.

Step-5: Repeat the third steps, which means reassign each datapoint to the new closest
centroid of each cluster.

Step-6: If any reassignment occurs, then go to step-4 else go to FINISH.

Step-7: The model is ready.

Let's understand the above steps by considering the visual plots:

Suppose we have two variables M1 and M2. The x-y axis scatter plot of these two variables is
given below:

Dr. M. SUPRIYA,
AP/CSE
 o Let's take number k of clusters, i.e., K=2, to identify the dataset and to put them into
different clusters. It means here we will try to group these datasets into two different
clusters.
o We need to choose some random k points or centroid to form the cluster. These points
can be either the points from the dataset or any other point. So, here we are selecting
the below two points as k points, which are not the part of our dataset. Consider
thebelow image:

o Now we will assign each data point of the scatter plot to its closest K-point or centroid.
We will compute it by applying some mathematics that we have studied to calculate the
distance between two points. So, we will draw a median between
[Link] the below image:

From the above image, it is clear that points left side of the line is near to the K1 or blue
centroid, and points to the right of the line are close to the yellow centroid. Let's color them as
blue and yellow for clear visualization.

Dr. M. SUPRIYA,
AP/CSE
 o As we need to find the closest cluster, so we will repeat the process by choosing a
new centroid. To choose the new centroids, we will compute the center of gravity of
these centroids, and will find new centroids as below:

o Next, we will reassign each datapoint to the new centroid. For this, we will repeat the
same process of finding a median line. The median will be like below image:

From the above image, we can see, one yellow point is on the left side of the line, and two
blue points are right to the line. So, these three points will be assigned to new centroids.

Dr. M. SUPRIYA,
AP/CSE
As reassignment has taken place, so we will again go to the step-4, which is finding new
centroids or K-points.
o We will repeat the process by finding the center of gravity of centroids, so the new
centroids will be as shown in the below image:

o As we got the new centroids so again will draw the median line and reassign the data
points. So, the image will be:

o We can see in the above image; there are no dissimilar data points on either side of
the line, which means our model is formed. Consider the below image:

Dr. M. SUPRIYA,
AP/CSE
As our model is ready, so we can now remove the assumed centroids, and the two final
clusters will be as shown in the below image:

How to choose the value of "K number of clusters" in K-means Clustering?

The performance of the K-means clustering algorithm depends upon highly efficient clusters
that it forms. But choosing the optimal number of clusters is a big task. There are some different
ways to find the optimal number of clusters, but here we are discussing the most appropriate
method to find the number of clusters or value of K. The method is given below:

Elbow Method

The Elbow method is one of the most popular ways to find the optimal number of clusters. This
method uses the concept of WCSS value. WCSS stands for Within Cluster Sum of Squares,
which defines the total variations within a cluster. The formula to calculate the value of WCSS
(for 3 clusters) is given below:

In the above formula of WCSS,

∑Pi in Cluster1 distance(Pi C1)2: It is the sum of the square of the distances between each data point
and its centroid within a cluster1 and the same for the other two terms.

To measure the distance between data points and centroid, we can use any method such as

Dr. M. SUPRIYA,
AP/CSE
Euclidean distance or Manhattan distance.

To find the optimal value of clusters, the elbow method follows the below steps:

o It executes the K-means clustering on a given dataset for different K values (ranges
from 1-10).
o For each value of K, calculates the WCSS value.
o Plots a curve between calculated WCSS values and the number of clusters K.
o The sharp point of bend or a point of the plot looks like an arm, then that point is
considered as the best value of K.
Since the graph shows the sharp bend, which looks like an elbow, hence it is known as the
elbow method. The graph for the elbow method looks like the below image:

Instance Based Learning: KNN

 KNN is a simple, supervised machine learning (ML) algorithm that can be used
for classification or regression tasks - and is also frequently used in missing value
imputation.
 It is based on the idea that the observations closest to a given data point are the
most "similar" observations in a data set, and we can therefore classify unforeseen
points based on the values of the closest existing points. By choosing K, the user
can select the number of nearby observations to use in the algorithm.
 K-NN set of rules can be used for regression as well as for classification .
 K-NN is a non-parametric algorithm, because of this it does no longer makes any
assumption on underlying data.
 k-Nearest Neighbors (k-NN) is considered a "lazy learner" because it doesn't
perform any generalization or model training in advance.

Dr. M. SUPRIYA,
AP/CSE
  In k-NN, the entire dataset is kept in memory, and the algorithm doesn't build a
model or abstract any patterns from the data during a training phase. This is
different from "eager" learners like decision trees or neural networks, which
create a model based on the training data and then use that model to make
predictions.
 Example : Suppose, we’ve an picture of a creature that looks much like cat and
dog, but we want to know both it is a cat or dog. So for this identity, we are able

to use the KNN algorithm, because it works on a similarity degree. Our KNN
version will discover the similar features of the new facts set to the cats and dogs
snap shots and primarily based on the most similar functions it will place it in both
cat or canine class.

Why do We Need KNN ?

 Suppose there are two categories, i.e., category A and category B and we’ve a brand
new statistics point x1, so this fact point will lie within of these classes. To solve this
sort of problem, we need a K-NN set of rules. With the help of K-NN, we will without
difficulty discover the category or class of a selected dataset. Consider the underneath
diagram :

Fig.9.4.1 why do we need KNN

How Does KNN Work?

 The K-NN working can be explained on the basis of the below algorithm :

Step -1 :Select the wide variety of dataset

Step -2 :Calculate the Euclidean distance of K datasets.

Dr. M. SUPRIYA,
AP/CSE
Step -3 :Take the K nearest neighbors as according to the calculated Euclidean distance.

Step -4 : count the number of the data points in each class.

Step -5 :Assign the brand new records points to that category for which the quantity of the
neighbor is maximum.

Step -6 :Our model is ready.

 Suppose we’ve got a brand new information point and we want to place it in the required
category, Consider the under image

Fig4.4.2 KNN example

 Firstly, we are able to pick the number of friends, so we are able to select the ok=5
 Next, we will calculate the Euclidean distance between the facts points. The Euclidean
distance is the gap between points, which we’ve got already studied in geometry. It may
be calculated as :

Dr. M. SUPRIYA,
AP/CSE
 Fig4.4.3 KNN example continue

 By calculating the Euclidean distance we got the nearest acquaintances, as 3 nearest

neigh bours in category A and two nearest associates in class B. Consider the
underneath image.

Fig:KNN example continue

 As we are able to see the three nearest acquaintances are from category A, subsequently
this new fact point must belong to category A.

4.3.3 Difference between K- means and KNN

Sr. No K-means KNN

1 K-Means is an unsupervised machine KNN is a supervised machine
learning algorithm used for clustering. learning algorithm used for
classification.
2 K- Means is an eager learner k-NN is a lazy learner
3 It is used for Clustering It is used mostly for Classification,
and sometimes even for
Regression

Dr. M. SUPRIYA,
AP/CSE
4 K in K-Means is the number of clusters the K’ in KNN is the number of
algorithm is trying to identify/ learn from nearest neigh bours used classify
the data or predict a test sample
5 K-means require unlabeled data. It gathers
and groups data into k number of clusters.

Gaussian Mixture models and Expectation Maximization

 Gaussian Mixture Models is a “soft” clustering algorithm, where each point
probabilistically “belongs” to all clusters. This is different than k-means where each
point belongs to one cluster.

 The Gaussian mixture model is a probabilistic model that assumes all the data points
are generated from a mix of Gaussian distributions with unknown parameters.
 For example, in modeling human height data, height is typically modeled as a normal
distribution for each gender with a mean of approximately 5’10” for males and 5’5” for
females. Given only the height data and not the gender assignments for each data point,
the distribution of all heights would follow the sum of two scaled (different variance)
and shifted (different mean) normal distributions. A model making this assumption is
an example of a Gaussian mixture model.
 Gaussian mixture models do not rigidly classify each and every instance into one class
or the other. The algorithm attempts to produce K-Gaussian distributions that would
take into account the entire training space. Every point can be associated with one or
more distributions. Consequently, the deterministic factor would be the probability that
each point belongs to a certain Gaussian distribution.
 GMMs have a variety of real - world applications. Some of them are listed below.
a) Used for signal processing
b) Used for customer churn analysis
c) Used for language identification
d) Used in video game industry
e) Genre classification of songs
4.4.1 Expectation – maximization
 In Gaussian mixture models, an expectation-maximization method is a
powerful tool for estimating he parameters of a Gaussian mixture model.
The expectation is termed E and maximization is termed M.
 Expectation is used to find the Gaussian parameters which are used to represent each

Dr. M. SUPRIYA,
AP/CSE
 component of gaussian mixture models. Maximization is termed M and it is involved
in determining whether new data points can be added or not.
 The Expectation – Maximization (EM) algorithm is used in maximum likelihood
estimation where the problem involves two sets of random variables of which one, X,
is observable and the other, Z, is hidden.
 The goal of the algorithm is to find the parameter vector ∅ that maximizes the likelihood
of the observed values of X, L (∅ | X)

 But in cases where this is not feasible, we associated the extra hidden variables Z and
express the underlying model using both, to maximize the likelihood of the joint
distribution of X and Z the complete likelihood Lc (∅|X,Z)
 Expectation -maximization (EM) is an iterative method used to find maximum
likelihood estimates of parameters in probabilistic models, where the model depends
on unobserved, also called latent, variables.
 EM alternates between performing an expectation € step, which computes an
expectation of the likelihood by including the latent variables as if they were obserrved,
and maximization (M) step, which computes the maximum likelihood estimates of the
parameters by maximizing the expected likelihood found in the E step.
 The Parameters found on the M step are then used to start another E step, and the
process is repeated until some criterion is satisfied. EM is frequently used for data
clustering like for example in Gaussian mixtures.
 In the Expectation step, find the expected values of the latent variables (here you need
to use the current parameter values)
 In the Maximization step, first plug in the expected values of the latent variables in the
log-likelihood of the augmented data. The maximize this log-likelihood to reevaluate
the parameters.
 Expectation – Maximization (EM) is a technique used in point estimation. Given a set
a observable variables X and unknown (latent) variables Z we want to estimate
parameters ∅ in a model.
 The expectation maximization (EM ) algorithm is a widely used maximum likely- hood
estimation procedure for statistical models when the values of some of the variables in
the model are not observed
 The EM algorithm is an elegant and powerful method for finding the maximum
likelihood of models with hidden variables. The key concept in the EM algorithm is
that it iterates between the expectation step (E-setp ) and maximization step (M-step )

Dr. M. SUPRIYA,
AP/CSE
 until convergence.
 In the E-step, the algorithm estimates the posterior distribution of the hidden variables
Q given the observed data and the current parameter settings; and in the M-step the
algorithm calculates the ML parameter settings with Q fixed.

Dr. M. SUPRIYA,
AP/CSE
 At the end of each iteration the lower bound on the likelihood is optimized for the given
parameter setting (M-step) and the likelihood is set to the bound (E-setp ) which
guarantees an increase in the likelihood and convergence to a local maximum, or
global maximum if the likelihood function is unimodal.
 Generally, EM works best when the fraction of missing information is small and the
dimensionality of he data is not too large. EM can require many iterations, and higher
dimensionality can dramatically slow down the E-setp.
 EM is useful for several reasons: conceptual simplicity, ease of implementation, and
the fact that each iteration improves l (ø ). The rate of convergence on the first few
steps is typically quite good, but can become excruciatingly slow as you approach local
optima.
 Sometimes the M- step is a constrained maximization, which means that there are
constraints on valid solutions not encoded in the function itself.
 Expectation maximization is an effective technique that is often used in data analysis
to manage missing data. Indeed, expectation maximization overcomes some of the
limitations of other techniques, such as mean substitution or regression substitution.
These alternative techniques generate biased estimates – and specifically,
underestimate the standard errors. Expectation maximization overcomes this problem.

Dr. M. SUPRIYA,
AP/CSE
 Two Marks

Q:1 What is unsupervised learning?

Ans : In an unsupervised learning, the network adapts purely in response to its inputs. Such
networks can learn to pick out structure in their input.
Q.2 What is semi – supervised learning :
Ans: Semi – supervised learning uses both labeled and unlabeled data to improve supervised
learning.
Q.3 What is ensemble method?
Ans: Ensemble methods is a machine learning technique that combines several base models
in order to produce on optimal predictive model. It combine the insights obtained from
multiple learning models to facilitate accurate and improved decisions.
Q.4 What is cluster ?
Ans: Cluster is a group of objects that belong to the same class. In other words the similar
object are grouped in one cluster and dissimilar are grouped in other cluster
Q5 Explain clustering.
Ans: Clustering is a process of partitioning a set of data ion a set of meaningful subclass.
Every data in the subclass shares a common trait. It helps a user understand the natural
grouping or structure in a data set.
Q6 What is Bagging ?
Ans: Bagging is also known as Bootstrap aggregation, ensemble method works by training
multiple models independently and combining later to result in a strong model.
Q7 Define bosting.
Ans : Boosting refers to a group of algorithms that utilize weighted averages to make weak
learning algorithms stronger learning algorithms
Q8 What is K- Nearest Neighbour Methods?
Ans :· The K-Nearest Neighbor (KNN) is a classical classification method and requires no
training effort, critically depends on the quality of the distance measures among examples
The KNN classifier uses mahalanobis distance function. A sample is classified according
to the majority vote of the its nearest K training samples in the feature space. Distance of a
sample to its neighbors is defined using a distance function.
Q.9 Which are the performance factors that influence KNN algorithm?
Ans: The performance of the KNN algorithm is influenced by three main factors:
1. The distance function or distance metric used to determine the nearest neighbors.

Dr. M. SUPRIYA,
AP/CSE
 2. The decision rule used to drive a classification from the K-nearest neighbors.
3. The number of neighbors used to classify the new example.
Q10. What is K-means clustering?
Ans: k-means clustering is heuristic method. Here each cluster is represented by the center
off the cluster. The k-means algorithm takes the input parameter, k, and partitions a set of a
objects into k-clusters so that the resulting intracluster similarity is high but the intracluster
similarity is low
Q.11 List the properties of K-Means algorithm.
Ans : 1. There are always k clusters.
2. There is always at least one item in each cluster.
3. The clusters are non – hierarchical and they do not overlap.
Q.12 What is stacking ?
Ans : Staking, sometimes called stacked generalization. Is an ensemble machine learning
method that combines multiple heterogeneous base or component models via a meta – model.
Q.13 How do GMMs differentiate from K- means clustering ?
Ans : GMMs and K-means, both are clustering algorithms used for unsupervised learning
tasks. However, the basic difference between the is that k-means is a distance -based
clustering method while GMMs is a distribution based clustering method.

PART B

1. What is ensemble technique. How it works on different dataset.

2. Explain:

1) Bagging
2)stacking
3)Boosting
3. With a suitable example explain K means clustering.
4. What is the difference between k means and [Link] the cocept of KNN with
suitable example.
5. Expalin about Gaussian mixture and expectation maximization
6. How do the voting concept works?

PART C

Dr. M. SUPRIYA,
AP/CSE
1) Outline the steps in the AdaBoost algorithm with an suitable example.
2) Write the steps in expectation maximization algorithm.
3)

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
UNIT IV

NEURAL NETWORKS

Perceptron- Multilayer perceptron, activation functions, network training – gradient descent

optimization – stochastic gradient descent, error backpropagation, from shallow networks to
deep networks –Unit saturation (aka the vanishing gradient problem) – ReLU, hyperparameter
tuning, batch normalization, regularization, dropout.

4.1 PERCEPTRON IN MACHINE LEARNING

Perceptron is Machine Learning algorithm for supervised learning of various binary classification
tasks. Further, Perceptron is also understood as an Artificial Neuron or neural network unit that
helps to detect certain input data computations in business intelligence.

Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
However, it is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a
single-layer neural network with four main parameters, i.e., input values, weights and Bias, net
sum, and an activation function.

Basic Components of Perceptron

Mr. Frank Rosenblatt invented the perceptron model as a binary classifier which contains three
main components. These are as follows:

Fig: 5.1

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
o Input Nodes or Input Layer:

This is the primary component of Perceptron which accepts the initial data into the system for
further processing. Each input node contains a real numerical value.

o Wight and Bias:

Weight parameter represents the strength of the connection between units. This is another most
important parameter of Perceptron components. Weight is directly proportional to the strength of
the associated input neuron in deciding the output. Further, Bias can be considered as the line of
intercept in a linear equation.

o Activation Function:

These are the final and important components that help to determine whether the neuron will fire
or not. Activation Function can be considered primarily as a step function.

Types of Activation functions:

o Sign function
o Step function, and
o Sigmoid function

Fig:5.2

 The data scientist uses the activation function to take a subjective decision based on various
problem statements and forms the desired outputs.
 Activation function may differ (e.g., Sign, Step, and Sigmoid) in perceptron models by
checking whether the learning process is slow or has vanishing or exploding gradients.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
How does Perceptron work?

In Machine Learning, Perceptron is considered as a single-layer neural network that consists of

four main parameters named input values (Input nodes), weights and Bias, net sum, and an
activation function. The perceptron model begins with the multiplication of all input values and
their weights, then adds these values together to create the weighted sum. Then this weighted sum
is applied to the activation function 'f' to obtain the desired output. This activation function is also
known as the step function and is represented by 'f'.

Fig:5.3

This step function or Activation function plays a vital role in ensuring that output is mapped
between required values (0,1) or (-1,1). It is important to note that the weight of input is indicative
of the strength of a node. Similarly, an input's bias value gives the ability to shift the activation
function curve up or down.

Perceptron model works in two important steps as follows:

Step-1

In the first step first, multiply all input values with corresponding weight values and then add them
to determine the weighted sum. Mathematically, we can calculate the weighted sum as follows:

∑wi*xi = x1*w1 + x2*w2 +…wn*xn

Add a special term called bias 'b' to this weighted sum to improve the model's performance.

∑wi*xi + b

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Step-2

In the second step, an activation function is applied with the above-mentioned weighted sum,
which gives us output either in binary form or a continuous value as follows:

Y = f(∑wi*xi + b)

Types of Perceptron Models

Based on the layers, Perceptron models are divided into two types. These are as follows:

1. Single-layer Perceptron Model

2. Multi-layer Perceptron model

Single Layer Perceptron Model:

This is one of the easiest Artificial neural networks (ANN) types. A single-layered perceptron
model consists feed-forward network and also includes a threshold transfer function inside the
model. The main objective of the single-layer perceptron model is to analyze the linearly separable
objects with binary outcomes.

In a single layer perceptron model, its algorithms do not contain recorded data, so it begins with
inconstantly allocated input for weight parameters. Further, it sums up all inputs (weight). After
adding all inputs, if the total sum of all inputs is more than a pre-determined value, the model gets
activated and shows the output value as +1.

If the outcome is same as pre-determined or threshold value, then the performance of this model
is stated as satisfied, and weight demand does not change. However, this model consists of a few
discrepancies triggered when multiple weight inputs values are fed into the model. Hence, to find
desired output and minimize errors, some changes should be necessary for the weights input.

Multi-Layered Perceptron Model:

Like a single-layer perceptron model, a multi-layer perceptron model also has the same model
structure but has a greater number of hidden layers.

The multi-layer perceptron model is also known as the Backpropagation algorithm, which executes
in two stages as follows:

o Forward Stage: Activation functions start from the input layer in the forward stage and
terminate on the output layer.
o Backward Stage: In the backward stage, weight and bias values are modified as per the
model's requirement. In this stage, the error between actual output and demanded
originated backward on the output layer and ended on the input layer.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

4.1 MULTILAYER PERCEPTRON:

 Multilayer perceptron is one of the most commonly used machine learning method.
 The Multi-layer Perceptron network, consisting of multiple layers of connected neurons.
 Multilayer perceptron is an artificial neural network structure and is a non parametric
estimator that can be used for classification and regression.

Fig: 5.4 The Multi-layer Perceptron network

 In the multi-layer perceptron diagram above, we can see that there are three inputs and thus
three input nodes and the hidden layer has three nodes.
 The output layer gives two outputs, therefore there are two output nodes.
 The nodes in the input layer take input and forward it for further process, in the diagram
above the nodes in the input layer forwards their output to each of the three nodes in the
hidden layer, and in the same way, the hidden layer processes the information and passes
it to the output layer.
 Every node in the multi-layer perception uses a sigmoid activation function. The sigmoid
activation function takes real values as input and converts them to numbers between 0 and
1 using the sigmoid formula.
 The most commonly used form of this function (where β is some positive parameter) is:

 The multi-layer perceptron is also known as back propagation algorithm, which executes
in two stages as follows:
i. Forward stage:

In Figure 5.1, we start at the left by filling in the values for the inputs. We then use these
inputs and the first level of weights to calculate the activations of the hidden layer, and then
we use those activations and the next set of weights to calculate the activations of the output

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
layer. Now that we’ve got the outputs of the network, we can compare them to the targets
and compute the error.

ii. Backward stage: BACK-PROPAGATION OF ERROR

Backpropagation, or backward propagation of errors, is an algorithm that is designed to

test for errors working back from output nodes to input nodes.

The error function that we used for the Perceptron was

Where N is the number of output nodes.

4.1.1 The Multi-layer Perceptron Algorithm:

The MLP training algorithm using back-propagation of error is described below:

1. an input vector is put into the input nodes

2. the inputs are fed forward through the network

• the inputs and the first-layer weights (here labelled as v) are used to decide
whether the hidden nodes fire or not. The activation function g(·) is the sigmoid
function given in

• the outputs of these neurons and the second-layer weights (labelled as w) are
used to decide if the output neurons fire or not
3. the error is computed as the sum-of-squares difference between the network outputs
and the targets

4. this error is fed backwards through the network in order to

• first update the second-layer weights
• and then afterwards, the first-layer weights

The steps for multilayer perceptron is given below:

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

Advantages of Multi-layer perceptron:

 It can be used to solve complex nonlinear problems.
 It handles large amounts of input data well.
 Makes quick predictions after training.
 The same accuracy ratio can be achieved even with smaller samples.
Disadvantages of Multi-layer perceptron:
 In Multi-layer perceptron, computations are difficult and time-consuming.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
 In multi-layer Perceptron, it is difficult to predict how much the dependent variable
affects each independent variable.
 The model functioning depends on the quality of the training.

4.1.2 Activation Functions:

 Artificial neurons are elementary units in an artificial neural network. The artificial
neuron receives one or more inputs and sums them to produce an output. Each input is
separately weighted, and the sum is passed through a function known as an activation
function or transfer function.
 In an artificial neural network, the function which takes the incoming signals as input and
produces the output signal is known as the activation function.

Fig: 5.5 Artificial neuron

x1, x2,……,xn : input signals

w1,w2,……,wn : weights associated with input signals
x0 : input signal taking the constant value 1
w0 : weight associated with x0 (called bias)
Ʃ : indicates summation of input signals
f : function which produces the output
y : output signal

 The function f can be expressed in the following form:

Some simple activation functions

The following are some of the simple activation functions.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
1. Threshold activation function
 The threshold activation function is defined by

 The graph of this function is shown as follows:

Fig: 5.6 Threshold activation function

2. Unit step functions:

 Sometimes, the threshold activation function is also defined as a unit step function in
which case it is called a unit-step activation function.
 This is defined as follows:

 The graph of this function is shown as follows:

Fig: 5.7 Unit step functions

3. Sigmoid activation function (logistic function):

 One of the most commonly used activation functions is the sigmoid activation function.

 It is a function which is plotted as ‘S’ shaped graph

 This is defined as follows:

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

 Value Range :- 0to +1

 Nature :- non-linear
 Uses : Usually used in output layer of a binary classification, where result is either 0 or 1,
as value for sigmoid function lies between 0 and 1 only so, result can be predicted easily
to be 1 if value is greater than 0.5 and 0 otherwise.
 The graph of this function is shown as follows:

Fig: 5.8 Sigmoid activation function

4. Linear activation function

 The linear activation function is defined by
F(x) = mx + c
 This defines a straight line in the xy-plane.

Fig: 5.9 Linear activation function

5. Tanh or Hyperbolic tangential activation function

 The activation that works almost always better than sigmoid function is Tanh function also
known as Tangent Hyperbolic function. It’s actually mathematically shifted version of the
sigmoid function. Both are similar and can be derived from each other.
 Value Range :- -1 to +1
 Nature :- non-linear
 Uses :- Usually used in hidden layers of a neural network as it’s values lies between -1 to
1 hence the mean for the hidden layer comes out be 0 or very close to it, hence helps in

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
centering the data by bringing mean close to 0. This makes learning for the next layer much
easier.
 This is defined by

Fig: 5.10 Hyperbolic tangent activation function

6. RELU Activation Function

It Stands for Rectified linear unit. It is the most widely used activation function. Chiefly
implemented in hidden layers of Neural network.

Equation :- A(x) = max(0,x). It gives an output x if x is positive and 0 otherwise.

Value Range :- [0, inf)

Nature :- non-linear, which means we can easily backpropagate the errors and have multiple layers
of neurons being activated by the ReLU function.

Uses :- ReLu is less computationally expensive than tanh and sigmoid because it involves simpler
mathematical operations. At a time only a few neurons are activated making the network sparse
making it efficient and easy for computation.

Fig: 5.11 RELU activation Function

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
4.2 GRADIENT DESCENT OPTIMIZATION:

Gradient Descent is known as one of the most commonly used optimization algorithms to train
machine learning models by means of minimizing errors between actual and expected results. Further,
gradient descent is also used to train Neural Networks.

➢ In mathematical terminology, Optimization algorithm refers to the task of minimizing/maximizing

an objective function f(x) parameterized by x. Similarly, in machine learning, optimization is the task
of minimizing the cost function parameterized by the model's parameters.

➢ The main objective of gradient descent is to minimize the convex function using iteration of
parameter updates.

➢ Once these machine learning models are optimized, these models can be used as powerful tools for
Artificial Intelligence and various computer science applications.

➢ In this tutorial on Gradient Descent in Machine Learning, we will learn in detail about gradient
descent, the role of cost functions specifically as a barometer within Machine Learning, types of
gradient descents, learning rates, etc.

What is Gradient Descent or Steepest Descent?

Gradient descent was initially discovered by "Augustin-Louis Cauchy" in mid of 18th century.
Gradient Descent is defined as one of the most commonly used iterative optimization algorithms of
machine learning to train the machine learning and deep learning models. It helps in finding the
local minimum of a function.

The best way to define the local minimum or local maximum of a function using gradient descent is as
follows:

• If we move towards a negative gradient or away from the gradient of the function at the current
point, it will give the local minimum of that function.

• Whenever we move towards a positive gradient or towards the gradient of the function at the
current point, we will get the local maximum of that function.

This entire procedure is known as Gradient Ascent, which is also known as steepest descent. The main
objective of using a gradient descent algorithm is to minimize the cost function using iteration. To
achieve this goal, it performs two steps iteratively:

• Calculates the first-order derivative of the function to compute the gradient or slope of that
function.

• Move away from the direction of the gradient, which means slope increased from the current
point by alpha times, where Alpha is defined as Learning Rate. It is a tuning parameter in the
optimization process which helps to decide the length of the steps.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Cost-function

The cost function is defined as the measurement of difference or error between actual
values and expected values at the current position and present in the form of a single real number.
It helps to increase and improve machine learning efficiency by providing feedback to this model so that
it can minimize error and find the local or global minimum.

How does Gradient Descent work?

Before starting the working principle of gradient descent, we should know some basic concepts to find
out the slope of a line from linear regression. The equation for simple linear regression is given as:

1. Y=mX+c

Where 'm' represents the slope of the line, and 'c' represents the intercepts on the y-axis.

The starting point(shown in above fig.) is used to evaluate the performance as it is considered just as an
arbitrary point. At this starting point, we will derive the first derivative or slope and then use a tangent
line to calculate the steepness of this slope. Further, this slope will inform the updates to the parameters
(weights and bias).

The slope becomes steeper at the starting point or arbitrary point, but whenever new parameters are
generated, then steepness gradually reduces, and at the lowest point, it approaches the lowest point,
which is called a point of convergence.

Direction & Learning Rate

These two factors are used to determine the partial derivative calculation of future iteration and allow
it to the point of convergence or local minimum or global minimum.

It is defined as the step size taken to reach the minimum or lowest point. This is typically a small
value that is evaluated and updated based on the behavior of the cost function. If the learning rate is
high, it results in larger steps but also leads to risks of overshooting the minimum. At the same time, a
low learning rate shows the small step sizes, which compromises overall efficiency but gives the
advantage of more precision.

It produces less noise in comparison to other gradient descent.

It produces stable gradient descent convergence.

It is Computationally efficient as all resources are used for all training samples.

Types of Gradient Descent

Based on the error in various training models, the Gradient Descent learning algorithm can be divided
into Batch gradient descent, stochastic gradient descent, and mini-batch gradient descent. Let's
understand these different types of gradient descent:

1. Batch Gradient Descent:

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Batch gradient descent (BGD) is used to find the error for each point in the training set and update the
model after evaluating all training examples. This procedure is known as the training epoch. In simple
words, it is a greedy approach where we have to sum over all examples for each update.

Advantages of Batch gradient descent:

2. Stochastic gradient descent

Stochastic gradient descent (SGD) is a type of gradient descent that runs one training example per
iteration. Or in other words, it processes a training epoch for each example within a dataset and
updates each training example's parameters one at a time.

As it requires only one training example at a time, hence it is easier to store in allocated memory.
However, it shows some computational efficiency losses in comparison to batch gradient systems as it
shows frequent updates that require more detail and speed. Further, due to frequent updates, it is also
treated as a noisy gradient. However, sometimes it can be helpful in finding the global minimum and
also escaping the local minimum.

Advantages of Stochastic gradient descent:

In Stochastic gradient descent (SGD), learning happens on every example, and it consists of a few
advantages over other gradient descent.

• It is easier to allocate in desired memory.

• It is relatively fast to compute than batch gradient descent.

• It is more efficient for large datasets.

3. MiniBatch Gradient Descent:

Mini Batch gradient descent is the combination of both batch gradient descent and stochastic gradient
descent. It divides the training datasets into small batch sizes then performs the updates on those
batches separately.

Splitting training datasets into smaller batches make a balance to maintain the computational
efficiency of batch gradient descent and speed of stochastic gradient descent. Hence, we can achieve a
special type of gradient descent with higher computational efficiency and less noisy gradient descent.

Advantages of Mini Batch gradient descent:

• It is easier to fit in allocated memory.

• It is computationally efficient.

• It produces stable gradient descent convergence.

Challenges with the Gradient Descent

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Although we know Gradient Descent is one of the most popular methods for optimization problems, it
still also has some challenges. There are a few challenges as follows:

1. Local Minima and Saddle Point:

For convex problems, gradient descent can find the global minimum easily, while for non- convex
problems, it is sometimes difficult to find the global minimum, where the machine learning models
achieve the best results.

Whenever the slope of the cost function is at zero or just close to zero, this model stops learning
further. Apart from the global minimum, there occur some scenarios that can show this slop, which is
saddle point and local minimum. Local minima generate the shape similar to the global minimum,
where the slope of the cost function increases on both sides of the current points.

In contrast, with saddle points, the negative gradient only occurs on one side of the point, which
reaches a local maximum on one side and a local minimum on the other side. The name of a saddle
point is taken by that of a horse's saddle.

The name of local minima is because the value of the loss function is minimum at that point in a local
region. In contrast, the name of the global minima is given so because the value of the loss function is
minimum there, globally across the entire domain the loss function.

2. Vanishing and Exploding Gradient

In a deep neural network, if the model is trained with gradient descent and backpropagation, there can
occur two more issues other than local minima and saddle point.

Vanishing Gradients:

Vanishing Gradient occurs when the gradient is smaller than expected. During backpropagation, this
gradient becomes smaller that causing the decrease in the learning rate of earlier

layers than the later layer of the network. Once this happens, the weight parameters update until they
become insignificant.

Exploding Gradient:

Exploding gradient is just opposite to the vanishing gradient as it occurs when the Gradient is too large
and creates a stable model. Further, in this scenario, model weight increases, and they will be
represented as NaN. This problem can be solved using the dimensionality reduction technique, which
helps to minimize complexity within the model.
4.3 ERROR BACKPROPAGATION, FROM SHALLOW NETWORKS TO DEEP
NETWORKS:
Refer the section 4.1 Back propagation

4.4 UNIT SATURATION (aka the vanishing gradient problem)

Unit saturation occurs when the output of an activation function reaches its maximum or
minimum value and stops responding to changes in input. This can hinder the learning process of

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
neural networks.

The key point is that the calculated partial derivatives used to compute the gradient as one goes
deeper into the network. Since the gradients control how much the network learns during training,
the gradients are very small or zero, then little to no training can take place, leading to poor
predictive performance.

The problem:

As more layers using certain activation functions are added to neural networks, the gradients
of the loss function approaches zero, making the network hard to train.

Why:

Certain activation functions, like the sigmoid function, squishes a large input space into a small
input space between 0 and 1. Therefore, a large change in the input of the sigmoid function will
cause a small change in the output. Hence, the derivative becomes small.

Fig:5.16

The sigmoid function and its derivative

As an example, the above image is the sigmoid function and its derivative. Note how when the
inputs of the sigmoid function becomes larger or smaller (when |𝑥| becomes bigger), the derivative
becomes close to zero.

Why it's significant:

For shallow network with only a few layers that use these activations, this isn't a big problem.
However, when more layers are used, it can cause the gradient to be too small for training to work
effectively. Gradients of neural networks are found using backpropagation. Simply put,
backpropagation finds the derivatives of the network by moving layer by layer from the final layer
to the initial one. By the chain rule, the derivatives of each layer are multiplied down the network
(from the final layer to the initial) to compute the derivatives of the initial layers.

However, when n hidden layers use activation like the sigmoid an function, n small derivatives

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
are multiplied together. Thus, the gradient decreases exponentially as we propagate down to the
initial layers. A small gradient means that the weights and biases of the initial layers will not be
updated effectively with each training session. Since these initial layers are often crucial to
recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole
network.

Solution:

The simplest solution is to use other activation functions, such as ReLU, which doesn't cause
a small derivative. Residual networks are another solution, as they provide residual connections
straight to earlier layers. The residual connection directly adds the value at the beginning of the
block, x, to the end of the block (F(x) + x). This residual connection doesn't go through activation
functions that "squashes" the derivatives, resulting in a higher overall derivative of the block.

Fig:5.17

What is an activation function?

 Activation function is a simple mathematical function that transforms the given

input to the required output that has a certain range. From their name they activate
the neuron when output reaches the set threshold value of the function.

 Basically are responsible for switching the neuron ON/OFF. The neuron receives
the sum of the product of inputs and randomly initialized weights along with a static
bias for each layer.

 The activation function is applied on to this sum, and an output is generated.

Activation functions introduce a non-linearity, so as to make the network learn
complex patterns in the data such as in the case of images, text, videos or sounds.

 Without an activation function our model is going to behave like a linear regression
model that has limited learning capacity.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
4.5 ReLU

The rectified linear activation unit, or ReLU, is one of the few landmarks in the deep learning
revolution. It's simple, yet it's far superior to previous activation functions like sigmoid or tanh.

ReLU formula is: f(x) = max(0,x)

Both the ReLU function and its derivative are monotonic. If the function receives any
negative input, it returns 0; however, if the function receives any positive value x, it returns that
value. As a result, the output has a range of 0 to infinite. ReLU is the most often used activation
function in neural networks, especially CNNs, and is utilized as the default activation function.

Fig: 5.18 RELU activation Function

Implementing ReLu function in Python

The code for ReLu is as follows :

def relu(x):

return max(0.0, x)

To test the function, let’s run it on a few inputs.

x = 1.0

print('Applying Relu on (%.1f) gives %.1f' % (x, relu(x)))

x = -10.0

print('Applying Relu on (%.1f) gives %.1f' % (x, relu(x)))

x = 0.0

print('Applying Relu on (%.1f) gives %.1f' % (x, relu(x)))

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
x = 15.0

print('Applying Relu on (%.1f) gives %.1f' % (x, relu(x)))

x = -20.0

print('Applying Relu on (%.1f) gives %.1f' % (x, relu(x)))

Fig: 5.19

We see from the plot that all the negative values have been set to zero, and the positive values
are returned as it is. Note that we've given a set of consecutively increasing numbers as input, so
we've a linear output with an increasing slope.

Advantages of ReLU:

ReLU is used in the hidden layers instead of Sigmoid or tanh as using sigmoid or tanh in the
hidden layers leads to the infamous problem of "Vanishing Gradient". The "Vanishing Gradient"
prevents the earlier layers from learning important information when the network is
backpropagating. The sigmoid which is a logistic function is more preferrable to be used in
regression or binary classification related problems and that too only in the output layer, as the
output of a sigmoid function ranges from 0 to 1.

Some of the advantages of ReLU are:

→ Simpler Computation: Derivative remains constant i.e 1 for a positive input and thus
reduces the time taken for the model to learn and in minimizing the errors.

→ Mitigates Vanishing Gradient Problem:Unlike activation functions like sigmoid or tanh,

ReLU maintains a gradient of 1 for positive inputs, helping to avoid the vanishing gradient
problem in deep networks.
→ Less Computational Cost:

ReLU is computationally cheaper than functions like sigmoid or tanh, which involve exponentials.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Disadvantages of ReLU:

 Exploding Gradient: This occurs when the gradient gets accumulated, this causes a large
differences in the subsequent weight updates.
 Dying ReLU: The problem of "dead neurons" occurs when the neuron gets stuck in the
negative side and constantly outputs zero. Because gradient of 0 is also 0, it's unlikely for
the neuron to ever recover. This happens when the learning rate is too high or negative bias
is quite large.
 Sensitivity to Outliers:

ReLU is sensitive to outliers in the input data, which can lead to unexpected behavior
during training.

4.6 HYPERPARAMETER TUNING

A Machine Learning model is defined as a mathematical model with a number of parameters

that need to be learned from the data. By training a model with existing data, we are able to fit the
model parameters, However, there is another kind of parameter, known as Hyperparameters, that
cannot be directly learned from the regular training process. They are usually fixed before the
actual training process begins. These parameters express important properties of the model such
as its complexity or how fast it should learn.

Some examples of model hyperparameters include:

 The penalty in Logistic Regression Classifier i.e. L, or L, regularization

 The learning rate for training a neural network.
 The C and sigma hyperparameters for support vector machines.
 The k in k-nearest neighbors.
Models can have many hyperparameters and finding the best combination of parameters can
be treated as a search problem. The two best strategies for Hyperparameter tuning are:

Grid Search CV

In GridSearchCV approach, the machine learning model is evaluated for a range of

hyperparameter values. This approach is called GridSearchCV, because it searches for the best set
of hyperparameters from a grid of hyperparameters values. For example, if we want to set two
hyperparameters C and Alpha of the Logistic Regression Classifier model, with different sets of
values. The grid search technique will construct many versions of the model with all possible
combinations of hyperparameters and will return the best one.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

Fig: 5.20

As in the image, for C = [0.1, 0.2, 0.3, 0.4, 0.5] and Alpha= [0.1, 0.2, 0.3, 0.4]. For a
combination of C = 0.3 and Alpha = 0.2, the performance score comes out to be 0.726(Highest),
therefore it is selected.

The following code illustrates how to use GridSearchCV

# Necessary imports

from sklearn.linear_model import Logistic Regression

from sklearn.model_selection import GridSearchCV

# Creating the hyperparameter grid

c_space = [Link](-5, 8, 15) param_grid = {'C': c_space}

#Instantiating logistic regression classifier

logreg = Logistic Regression()

# Instantiating the GridSearchCV object

logreg_cv= GridSearchCV(logreg, param_grid, cv = 5)

logreg_cv.fit(X,y)

# Print the tuned parameters and score

print("Tuned Logistic Regression Parameters:{}".format(logreg_cv.best_params_))

print("Best score is {}".format(logrcg_cv.best_score_))

Output:

Tuned Logistic Regression Parameters: {'C': 3.7275937203149381) Best score is

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
0.7708333333333334

Drawback:

GridSearch CV will go through all the intermediate combinations of hyperparameters which

makes grid search computationally very expensive.

4.7 BATCH NORMALIZATION

Normalization is a data pre-processing tool used to bring the numerical data to a common scale
without distorting its shape. -Generally, when we input the data to a machine or deep learning
algorithm we tend to change the values to a balanced scale. The reason we normalize is partly to
ensure that our model can generalize appropriately. Now coming back to Batch normalization, it
is a process to make neural networks faster and more stable through adding extra layers in a deep
neural network. The new layer performs the standardizing and normalizing operations on the input
of a layer coming from a previous layer. A typical neural network is trained using a collected set
of input data called batch. Similarly, the normalizing process in batch normalization takes place in
batches, not as a single input.

Fig: 5.21

L = Number of layers

Bias = 0

Activation Function = Sigmoid

Initially, our inputs X1, X2, X3, X4 are in normalized form as they are coming from the pre-
processing stage.

When the input passes through the first layer, it transforms, as a sigmoid function applied over the
dot product of input X and the weight matrix W.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

h1 = 𝜎(W1X)

Fig: 5.22

Similarly, this transformation will take place for the second layer and go till the last layer L as
shown in the following image.

Fig:5.23

Although, our input X was normalized with time the output will no longer be on the same scale.
As the data go through multiple layers of the neural network and L activation functions are applied,
it leads to an internal co-variate shift in the data.

How does Batch Normalization work?

Since by now we have a clear idea of why we need Batch normalization, let's understand how it
works. It is a two-step process. First, the input is normalized, and later rescaling and offsetting is
performed.

Normalization of the Input

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
Normalization is the process of transforming the data to have a mean zero and standard deviation
one. In this step we have our batch input from layer h, first, we need to calculate the mean of this
hidden activation.

𝜇= 1 Ʃhi
𝑚

Here, m is the number of neurons at layer h. Once we have meant at our end, the next step is to
calculate the standard deviation of the hidden activations.
1
𝜇= √ Ʃ(hi − µ)2
𝑚

Further, as we have the mean and the standard deviation ready. We will normalize the hidden
activations using these values. For this, we will subtract the mean from each input and divide the
whole value with the sum of standard deviation and the smoothing term (ε). The smoothing term(ε)
assures numerical stability within the operation by stopping a division by a zero value.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
(ℎ𝑖− 𝜇)
h i(norm) =
𝜎+ 𝗌

Advantages of Batch Normalization

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

Now let's look into the advantages the BN process offers.

Speed Up the Training

By Normalizing the hidden layer activation the Batch normalization speeds up the training process.

Handles internal covariate shift

It solves the problem of internal covariate shift. Through this, we ensure that the input for every
layer is distributed around the same mean and standard deviation. If you are unaware of what is an
internal covariate shift, look at the following example.

Internal covariate shift

Suppose we are training an image classification model, that classifies the images into Dog or Not
Dog. Let's say we have the images of white dogs only, these images will have certain distribution
as well. Using these images model will update its parameters.

Smoothens the Loss Function

Batch normalization smoothens the loss function that in turn by optimizing the model parameters
improves the training speed of the model.

4.8 REGULARIZATION

The Problem of Overfitting

So, before diving into regularization, let's take a step back to understand what bias-variance is and
its impact. Bias is the deviation between the values predicted by the model and the actual values
whereas, variance is the difference between the predictions when the model fits different datasets.

When a model performs well on the training data and does not perform well on the testing data,
then the model is said to have high generalization error. In other words, in such a scenario, the
model has low bias and high variance and is too complex. This is called overfitting. Overfitting
means that the model is a good fit on the train data compared to the data. Overfitting is also a result
of the model being too complex.

What Is Regularization in Machine Learning?

Regularization is one of the key concepts in Machine learning as it helps choose a simple
model rather than a complex one. We want our model to perform well both on the train and the
new unseen data, meaning the model must have the ability to be generalized. Generalization error
is "a

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

measure of how accurately an algorithm can predict outcome values for previously unseen data."
Regularization refers to the modifications that can be made to a leaming algorithm that helps to
reduce this generalization error and not the training error. It reduces by ignoring the less
important features. It also helps prevent overfitting, making the model more robust and
decreasing the complexity of a model.

How Does Regularization Work?

Regularization works by shrinking the beta coefficients of a regression model. To understand why
we need to shrink the coefficients, let us see the below example:

Fig: 5.24

In the above graph, the two lines represent the relationship between total years of experience and
salary, where salary is the target variable. These are slopes indicating the change in salary per unit
change in total years of experience. As the slope b1+b3 decreases to slope b₁ , we see that the salary
is less sensitive to the total years of experience. By decreasing the slope, the target variable (salary)
became less sensitive to the change in the independent X variables, which increases the bias into
the model. Remember, bias is the difference between the predicted and the actual values.

With the increase in bias to the model, the variance (which is the difference between the predictions
when the model fits different datasets.) decreases. And, by decreasing the variance, the overfitting
gets reduced. The models having the higher variance leads to overfitting, and we saw above, we
will shrink or reduce the beta coefficients to overcome the overfitting. The beta coefficients or the
weights of the features converge towards zero, which is known as shrinkage.

What Is the Regularization Parameter?

For linear regression, the regularization has two terms in the loss function:

The Ordinary Least Squares (OLS) function, and

The penalty term

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
It becomes :

Loss function regularization =Loss function ols+ Penalty term

The goal of the linear regression model is to minimize the loss function. Now for Regularization,
the goal becomes to minimize the following cost function:
𝒏

∑( 𝐲𝐚𝐜𝐭– 𝐲𝐩𝐫𝐞𝐝)𝟐 + 𝐩𝐞𝐧𝐚𝐥𝐭𝐲

𝒊=𝟏

Where, the penalty term comprises the regularization parameter and the weights associated with
the variables. Hence, the penalty term is:

𝒑𝒆𝒏𝒂𝒍𝒕𝒚 = 𝝀 ∗ 𝒘

where,

λ = Regularization parameter

w = weight associated with the variables; generally considered to be L-p norms

The regularization parameter in machine learning is λ: It imposes a higher penalty on the variable
having higher values, and hence, it controls the strength of the penalty term. This tuning parameter
controls the bias-variance trade-off.

λ can take values 0 to infinity. If λ = 0, then means there is no difference between a model with
and without regularization.

Regularization Techniques in Machine Learning

Each of the following techniques uses different regularization norms (L-p) based on the
mathematical methodology that creates different kinds of regularization. These methodologies
have different effects on the beta coefficients of the features. The regularization techniques in
machine learning as follows:

(a) Ridge Regression

The Ridge regression technique is used to analyze the model where the variables may be having
multicollinearity. It reduces the insignificant independent variables though it does not remove
them completely. This type of regularization uses the L₂ norm for regularization
𝒏

𝒄𝒐𝒔𝒕 𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏 = ∑( 𝐲𝐚𝐜𝐭– 𝐲𝐩𝐫𝐞𝐝)𝟐 + 𝛌 ‖𝐰‖𝟐𝟐

𝒊=𝟏

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
(b) Lasso Regression
Least Absolute Shrinkage and Selection Operator (or LASSO) Regression penalizes the
coefficients to the extent that it becomes zero. It eliminates the insignificant independent
variables. This regularization technique uses the L1 norm for regularization.
𝒏

𝒄𝒐𝒔𝒕 𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏 = ∑( 𝐲𝐚𝐜𝐭– 𝐲𝐩𝐫𝐞𝐝)𝟐 + 𝛌 ‖𝐰‖𝟐

𝒊=𝟏

(c) Elastic Net Regression

The Elastic Net Regression technique is a combination of the Ridge and Lasso regression
technique. It is the linear combination of penalties for both the L, -norm and L₂ -norm
regularization.

The model using elastic net regression allows the learning of the sparse model where some of
the points are zero, similar to Lasso regularization, and yet maintains the Ridge regression
properties. Therefore, the model is trained on both the L, and L₂ norms.

The cost function of Elastic Net Regression is:

𝒏
𝒄𝒐𝒔𝒕 𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏 = ∑( 𝐲𝐚𝐜𝐭– 𝐲𝐩𝐫𝐞𝐝)𝟐 + 𝛌𝐫𝐢𝐝𝐠𝐞 ‖𝐰‖𝟐 + 𝛌𝐥𝐚𝐬𝐬𝐨‖𝐰‖𝟐
𝟐 𝟐
𝒊=𝟏

When to Use Which Regularization Technique?

The regularization in machine learning is used in following scenarios:

 Ridge regression is used when it is important to consider all the independent variables
in the model or when many interactions are present. That is where collinearity or
codependency is present amongst the variables.
 Lasso regression is applied when there are many predictors available and would want
the model to make feature selection as well for us.

When many variables are present, and we can't determine whether to use Ridge or Lasso
regression, then the Elastic-Net regression is your safe bet.

4.9 DROUPOUT:

"Dropout" in machine learning refers to the process of randomly ignoring certain nodes in a layer
during training. In the figure below, the neural network on the left represents a typical neural
network where all units are activated. On the right, the red units have been dropped out of the
model- the values of their weights and biases are not considered during training.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]

Fig:5.25

Dropout is used as a regularization technique - it prevents overfitting by ensuring that no units are
codependent.

Common Regularization Methods

Common regularization techniques include:

Early stopping: stop training automatically when a specific performance measure (eg. Validation
loss, accuracy) stops improving

Weight decay: incentivize the network to use smaller weights by adding a penalty to the loss
function (this ensures that the norms of the weights are relatively evenly distributed amongst all
the weights in the networks, which prevents just a few weights from heavily influencing network
output)

Noise: allow some random fluctuations in the data through augmentation (which makes the
network robust to a larger distribution of inputs and hence improves generalization)

Model combination: average the outputs of separately trained neural networks (requires a lot of
computational power, data, and time)

Dropout remains an extremely popular protective measure against overfitting because of its
efficiency and effectiveness.

How Does Dropout Work?

When we apply dropout to a neural network, we're creating a "thinned" network with unique
combinations of the units in the hidden layers being dropped randomly at different points in time
during training. Each time the gradient of our model is updated, we generate a new thinned neural
network with different units dropped based on a probability hyperparameter p. Training a network
using dropout can thus be viewed as training loads of different thinned neural networks and
merging them into one network that picks up the key properties of each thinned network. This
process allows dropout to reduce the overfitting of models on training data.

Dr. M. SUPRIYA,
AP/CSE
 [Type the document title]
This graph, taken from the paper "Dropout: A Simple Way to Prevent Neural Networks from
Overfitting" by Srivastava et al., compares the change in classification error of models without
dropout to the same models with dropout (keeping all other hyperparameters constant). All the
models have been trained on the MNIST dataset.

Fig:5.26

It is observed that the models with dropout had a lower classification error than the same models
without dropout at any given point in time. A similar trend was observed when the models were
used to train other datasets in vision, as well as speech recognition and text analysis. The lower
error is because dropout helps prevent overfitting on the training data by reducing the reliance of
each unit in the hidden layer on other units in the hidden layers.

The Downside of Dropout

Although dropout is clearly a highly effective tool, it comes with certain drawbacks. A network
with dropout can take 2-3 times longer to train than a standard network. One way to attain the
benefits of dropout without slowing down training is by finding a regularizer that is essentially
equivalent to a dropout layer. For linear regression, this regularizer has been proven to be a
modified form of L2 regularization.

Dr. M. SUPRIYA,
AP/CSE
[Type the document title]

Dr. M. SUPRIYA,
AP/CSE
 UNIT 5

Guidelines for machine learning experiments, Cross Validation (CV) and resampling –
Kfold CV, bootstrapping, measuring classifier performance, assessing a single
classification algorithm and comparing two classification algorithms – t test, Mc
Nemar’s test, K-fold CV paired t test

GUIDELINES FOR MACHINE LEARNING EXPERIMENTS

 The Machine Learning pipeline starts with data collection and integration. After data is
collected analysis and visualization of data is done. Further, the most crucial step feature
selection and engineering is performed then the model is trained. After that model,
evaluation is done and our model becomes ready for prediction!
 To understand the pipeline well, consider building an ML model for a company’s customer
care service. Consider a company XYZ, as an online book shop, that delivers books and
kindle to its customer and this company wants an improved customer care service. It wants
that if a customer calls the helpline for any kind of issue, say for replacement of books,
complaint of kindle purchased or some other services. The company wants to ensure that
the customer’s call gets directed to the right service person in minimum time and that the
process should be smooth. To build a model for customer care services of the company, we
will use the ML pipeline for the systematic development of the model.

Here is the diagrammatic view of the ML pipeline:

MACHINE LEARNING PIPELINE

1. Data Collection and integration:

 The first step of the ML pipeline involves the collection of data and integration of data.
 Data collected acts as an input to the model (data preparation phase)
 Inputs are called features.

Dr. M. SUPRIYA,

AP/CSE
  Data collected in the case of our considered example involves a lot of data. The collected
data should answer the following questions- What is past customer history? What were the
past orders? Is the customer a prime member of our bookstore? Does the customer own a
kindle? Has the customer made any previous complaints? What was the most number of
complaints?
 The more the data is, more the better our model becomes.
 Once the data is collected we need to integrate and prepare the data.
 Integration of data means placing all related data together.
 Then data preparation phase starts in which we manually and critically explore the data.
 The data preparation phase tells the developer that is the data matching the expectations. Is
there enough info to make an accurate prediction? Is the data consistent?
2. Exploratory Data Analysis and Visualisation:
 Once the data is prepared developer needs to visualize the data to have a better
understanding of relationships within the dataset.
 When we get to see data, we can notice the unseen patterns that we may not have noticed
in the first phase.
 It helps developers easily identify missing data and outliers.
 Data visualization can be done by plotting histograms, scatter plots, etc.
 After visualization is done data is analyzed so that developer can decide what ML
technique he may use.
 In the considered example case unsupervised learning may be used to analyze customer
purchasing habits.
3. Feature Selection and Engineering:
 Feature selection means selecting what features the developer wants to use within the
model.
 Features should be selected so that a minimum correlation exists between them and a
maximum correlation exists between the selected features and output.
 Feature engineering is the process to manipulate the original data into new and potential
data that has a lot many features within it.
 In simple words Feature engineering is converting raw data into useful data or getting the
maximum out of the original data.
 Feature engineering is arguably the most crucial and time-consuming step of the ML
pipeline.
 Feature selection and engineering answers questions – Are these features going to make
any sense in our prediction?
 It deals with the accuracy and precision of data.

Dr. M. SUPRIYA,

AP/CSE
4. Model Training:
 After the first three steps are done completely we enter the model training phase.
 It is the first step officially when the developer gets to train the model on basis of data.
 To train the model, data is split into three parts- Training data, validation data, and test
data.
 Around 70%-80% of data goes into the training data set which is used in training the
model.
 Validation data is also known as development set or dev set and is used to avoid overfitting
or underfitting situations i.e. enabling hyperparameter tuning.
 Hyperparameter tuning is a technique used to combat overfitting and underfitting.
 Validation data is used during model evaluation.
 Around 10%-15% of data is used as validation data.
 Rest 10%-15% of data goes into the test data set. Test data set is used for testing after the
model preparation.
 It is crucial to randomize data sets while splitting the data to get an accurate model.
 Data can be randomized using Scikit learn in python.
5. Model Evaluation:
 After the model training, validation, or development data is used to evaluate the model.
 To get the most accurate predictions to test data may be used for further model evaluation.
 A confusion matrix is created after model evaluation to calculate accuracy and precision
numerically.
 After model evaluation, our model enters the final stage that is prediction.
6. Prediction:
 In the prediction phase developer deploys the model.
 After model deployment, it becomes ready to make predictions.
 Predictions are made on training data and test data to have a better understanding of the
build model.
The deployment of the model isn’t a one-time exercise. As more and more data gets generated, the
model is trained on new data, evaluated again, and deployed again. Model training, model
evaluation, and prediction phase circulate each other.

CROSS-VALIDATION
In machine learning, we couldn’t fit the model on the training data and can’t say that the model
will work accurately for the real data. For this, we must assure that our model got the correct patterns

Dr. M. SUPRIYA,

AP/CSE
from the data, and it is not getting up too much noise. For this purpose, we use the cross-validation
technique. In this article, we’ll delve into the process of cross-validation in machine learning.
What is Cross-Validation?
Cross validation is a technique used in machine learning to evaluate the performance of a
model on unseen data. It involves dividing the available data into multiple folds or subsets, using
one of these folds as a validation set, and training the model on the remaining folds. This process is
repeated multiple times, each time using a different fold as the validation set. Finally, the results
from each validation step are averaged to produce a more robust estimate of the model’s
performance. Cross validation is an important step in the machine learning process and helps to
ensure that the model selected for deployment is robust and generalizes well to new data.
What is cross-validation used for?
The main purpose of cross validation is to prevent overfitting, which occurs when a model
is trained too well on the training data and performs poorly on new, unseen data. By evaluating the
model on multiple validation sets, cross validation provides a more realistic estimate of the model’s
generalization performance, i.e., its ability to perform well on new, unseen data.
Types of Cross-Validation
There are several types of cross validation techniques, including k-fold cross validation,
leave-one-out cross validation, and Holdout validation, Stratified Cross-Validation. The choice
of technique depends on the size and nature of the data, as well as the specific requirements of the
modeling problem.
1. Holdout Validation
In Holdout Validation, we perform training on the 50% of the given dataset and rest 50% is
used for the testing purpose. It’s a simple and quick way to evaluate a model. The major drawback
of this method is that we perform training on the 50% of the dataset, it may possible that the
remaining 50% of the data contains some important information which we are leaving while training
our model i.e. higher bias.
2. LOOCV (Leave One Out Cross Validation)
In this method, we perform training on the whole dataset but leaves only one data-point of
the available dataset and then iterates for each data-point. In LOOCV, the model is trained
on n−1n−1 samples and tested on the one omitted sample, repeating this process for each data point
in the dataset. It has some advantages as well as disadvantages also.
An advantage of using this method is that we make use of all data points and hence it is low bias.
The major drawback of this method is that it leads to higher variation in the testing model as we
are testing against one data point. If the data point is an outlier it can lead to higher variation.
Another drawback is it takes a lot of execution time as it iterates over ‘the number of data points’
times.
3. Stratified Cross-Validation

Dr. M. SUPRIYA,

AP/CSE
 It is a technique used in machine learning to ensure that each fold of the cross-validation process
maintains the same class distribution as the entire dataset. This is particularly important when
dealing with imbalanced datasets, where certain classes may be underrepresented. In this method,
1. The dataset is divided into k folds while maintaining the proportion of classes in each fold.
2. During each iteration, one-fold is used for testing, and the remaining folds are used for
training.
3. The process is repeated k times, with each fold serving as the test set exactly once.
Stratified Cross-Validation is essential when dealing with classification problems where
maintaining the balance of class distribution is crucial for the model to generalize well to unseen
data.
4. K-Fold Cross Validation
In K-Fold Cross Validation, we split the dataset into k number of subsets (known as folds)
then we perform training on the all the subsets but leave one(k-1) subset for the evaluation of the
trained model. In this method, we iterate k times with a different subset reserved for testing purpose
each time.
Note: It is always suggested that the value of k should be 10 as the lower value of k takes towards
validation and higher value of k leads to LOOCV method.
Example of K Fold Cross Validation
The diagram below shows an example of the training subsets and evaluation subsets
generated in k-fold cross-validation. Here, we have total 25 instances. In first iteration we use the
first 20 percent of data for evaluation, and the remaining 80 percent for training ([1-5] testing and
[5-25] training) while in the second iteration we use the second subset of 20 percent for evaluation,
and the remaining three subsets of the data for training ([5-10] testing and [1-5 and 10-25]
training), and so on.

Dr. M. SUPRIYA,

AP/CSE
Advantages:
1. Overcoming Overfitting: Cross validation helps to prevent overfitting by providing a more
robust estimate of the model’s performance on unseen data.
2. Model Selection: Cross validation can be used to compare different models and select the
one that performs the best on average.
3. Hyperparameter tuning: Cross validation can be used to optimize the hyperparameters of a
model, such as the regularization parameter, by selecting the values that result in the best
performance on the validation set.
4. Data Efficient: Cross validation allows the use of all the available data for both training and
validation, making it a more data-efficient method compared to traditional validation
techniques.
Disadvantages:
1. Computationally Expensive: Cross validation can be computationally expensive, especially
when the number of folds is large or when the model is complex and requires a long time to
train.
2. Time-Consuming: Cross validation can be time-consuming, especially when there are many
hyperparameters to tune or when multiple models need to be compared.
3. Bias-Variance Tradeoff: The choice of the number of folds in cross validation can impact
the bias-variance tradeoff, i.e., too few folds may result in high variance, while too many
folds may result in high bias.

BOOTSTRAPPING
Bootstrapping
Bootstrapping is a resampling technique that involves repeatedly drawing samples from the
dataset with replacement and estimating model performance on these samples. It provides a way
to assess the uncertainty in the performance metrics.
How Bootstrapping Works
1. Generate Bootstrap Samples:
 Draw nnn samples from the original dataset (with replacement) to create a
bootstrap sample.
 This process is repeated BBB times to create BBB bootstrap samples.
2. Train and Evaluate the Model:
 Train the model on each bootstrap sample and evaluate it on the out-of-bag (OOB)
data, which consists of data points not included in the bootstrap sample.
 The OOB error estimate is used to gauge model performance.

Dr. M. SUPRIYA,

AP/CSE
 3. Aggregate Results:
 Calculate the performance metrics for each bootstrap sample.
 Average the results to get an overall performance estimate.

“Instead of measuring the heights of all the students, we can draw a random sample
of 5 students and measure their heights. We would repeat this process 20 times and
then average the collected height data of 100 students (5 x 20). This average height
would be an estimate of the mean height of all the students of the school”.
Advantages of Bootstrapping
 Flexibility: Can be applied to small datasets where cross-validation might not be feasible.
 Variance Estimation: Provides an estimate of the variability of the performance metrics.
 Bias Correction: Helps in reducing the bias in the performance estimate.
Disadvantages of Bootstrapping
 Overfitting: Since bootstrap samples are drawn with replacement, they might include
duplicate data points, which can lead to overfitting.
 Computationally Demanding: Similar to cross-validation, it requires significant
computational resources, especially for a large number of bootstrap samples.

 Bootstrap Sampling in Machine Learning

 Bootstrap sampling is used in a machine learning ensemble algorithm called bootstrap

aggregating (also called bagging). It helps in avoiding overfitting and improves the stability
of machine learning algorithms.

 In bagging, a certain number of equally sized subsets of a dataset are extracted with

replacement. Then, a machine learning algorithm is applied to each of these subsets and the

outputs are ensembled as I have illustrated below:

Dr. M. SUPRIYA,

AP/CSE
Bootstrapping Pseudo Code
function bootstrap(data, model, num_samples):
scores = []
for i in range(num_samples):
bootstrap_sample = draw_bootstrap_sample(data)
[Link](bootstrap_sample)
score = [Link](data)
[Link](score)
average_score = average(scores)
return average_score
Key Differences Between Cross-Validation and Bootstrapping
While cross-validation and bootstrapping share similarities in their purpose and applications, they
differ significantly in their methodologies, advantages, and ideal use cases. Understanding these
differences is crucial for selecting the appropriate technique for a given task.
1. Data Partitioning
 Cross-Validation: The dataset is split into mutually exclusive subsets or folds. Each fold
is used once as the test set while the remaining folds are used for training.
o Example: In k-fold cross-validation, the dataset is split into k parts. The model is
trained on k-1 of these parts and tested on the remaining part. This process is
repeated k times, with each part used as the test set once.

Dr. M. SUPRIYA,

AP/CSE
  Bootstrapping: Involves sampling with replacement to create multiple bootstrap samples
from the original dataset. Each sample may contain duplicate data points.
o Example: A bootstrap sample of size n is created by randomly drawing n data
points with replacement from the original dataset. This process is repeated B times.
2. Structure of Samples
 Cross-Validation: Each fold is a unique subset of the data with no overlap between
training and test sets in any given iteration.
 Bootstrapping: Each bootstrap sample is likely to contain repeated instances of some data
points and may omit others. The test set (out-of-bag data) consists of data points not
included in the bootstrap sample.
3. Bias and Variance
 Cross-Validation: Typically provides a lower variance estimate of model performance due
to averaging over multiple folds, but may have higher bias if the number of folds is small.
 Bootstrapping: Can provide a lower bias estimate as it uses the full dataset for each
bootstrap sample but may have higher variance due to the nature of resampling with
replacement.
4. Performance Estimation
 Cross-Validation: Typically provides a lower variance estimate of model performance due
to averaging over multiple folds, but may have higher bias if the number of folds is small.
 Bootstrapping: Can provide a lower bias estimate as it uses the full dataset for each
bootstrap sample but may have higher variance due to the nature of resampling with
replacement.
Summary of Differences between Cross-Validation and Bootstrapping

Aspect Cross-Validation Bootstrapping

Splits data into k subsets (folds) for Samples data with replacement to
Definition training and validation. create multiple bootstrap datasets.

Estimate model performance and Estimate the variability of a statistic

Purpose generalize to unseen data. or model performance.

1. Randomly sample data with

1. Split data into k folds. replacement.
2. Train on k-1 folds, validate on the
2. Repeat to create multiple bootstrap
remaining fold.
Process samples.

Dr. M. SUPRIYA,

AP/CSE
Aspect Cross-Validation Bootstrapping

3. Repeat process k times (each fold 3. Evaluate model on each bootstrap

serves as validation once). sample.

1. Helps in model selection and 1. Captures uncertainty in model

tuning. estimates
2. Reduces overfitting by validating 2. Useful for assessing bias and
Advantages on unseen data. variance.

Computationally intensive for large May overestimate performance due to

Disadvantages k or datasets. sample similarity.

Commonly used in model evaluation Useful when dataset size is limited or

Applicability and selection. unknown distribution.

When to Use Cross-Validation

 Model Comparison: When comparing multiple models or algorithms.
 Hyperparameter Tuning: When tuning hyperparameters to find the best model
configuration.
 Balanced Datasets: Works well when the dataset is balanced and large enough to be split
into meaningful folds.
When to Use Bootstrapping
 Small Datasets: More effective for small datasets where splitting into multiple folds might
not be feasible.
 Variance Estimation: When an estimate of the variability of the model performance is
needed.
 Uncertain Data: When the dataset has significant noise or uncertainty.
PERFORMANCE METRICS IN MACHINE LEARNING
Evaluating the performance of a Machine learning model is one of the important steps while
building an effective ML model. To evaluate the performance or quality of the model, different
metrics are used, and these metrics are known as performance metrics or evaluation
metrics. These performance metrics help us understand how well our model has performed for the
given data. In this way, we can improve the model's performance by tuning the hyper-parameters.
Each ML model aims to generalize well on unseen/new data, and performance metrics help
determine how well the model generalizes on the new dataset.

Dr. M. SUPRIYA,

AP/CSE
 In machine learning, each task or problem is divided into classification and Regression.
Not all metrics can be used for all types of problems; hence, it is important to know and understand
which metrics should be used. Different evaluation metrics are used for both Regression and
Classification tasks. In this topic, we will discuss metrics used for classification and regression
tasks.
1. Performance Metrics for Classification
In a classification problem, the category or classes of data is identified based on training data.
The model learns from the given dataset and then classifies the new data into classes or groups
based on the training. It predicts class labels as the output, such as Yes or No, 0 or 1, Spam or Not
Spam, etc. To evaluate the performance of a classification model, different metrics are used, and
some of them are as follows:
o Accuracy
o Confusion Matrix
o Precision
o Recall
o F-Score
o AUC(Area Under the Curve)-ROC
I. Accuracy
The accuracy metric is one of the simplest Classification metrics to implement, and it can be
determined as the number of correct predictions to the total number of predictions.
It can be formulated as:

II. Confusion Matrix

Dr. M. SUPRIYA,

AP/CSE
 A confusion matrix is a tabular representation of prediction outcomes of any binary
classifier, which is used to describe the performance of the classification model on a set of test data
when true values are known.
The confusion matrix is simple to implement, but the terminologies used in this matrix might be
confusing for beginners.
A typical confusion matrix for a binary classifier looks like the below image(However, it can be
extended to use for classifiers with more than two classes).

We can determine the following from the above matrix:

o In the matrix, columns are for the prediction values, and rows specify the Actual values.
Here Actual and prediction give two possible classes, Yes or No. So, if we are predicting the
presence of a disease in a patient, the Prediction column with Yes means, Patient has the
disease, and for NO, the Patient doesn't have the disease.
o In this example, the total number of predictions are 165, out of which 110 time predicted
yes, whereas 55 times predicted No.
o However, in reality, 60 cases in which patients don't have the disease, whereas 105 cases in
which patients have the disease.
In general, the table is divided into four terminologies, which are as follows:
1. True Positive(TP): In this case, the prediction outcome is true, and it is true in reality, also.
2. True Negative(TN): in this case, the prediction outcome is false, and it is false in reality,
also.
3. False Positive(FP): In this case, prediction outcomes are true, but they are false in
actuality.
4. False Negative(FN): In this case, predictions are false, and they are true in actuality.
III. Precision
The precision metric is used to overcome the limitation of Accuracy. The precision
determines the proportion of positive prediction that was actually correct. It can be calculated as the
True Positive or predictions that are actually true to the total positive predictions (True Positive and
False Positive).

Dr. M. SUPRIYA,

AP/CSE
IV. Recall or Sensitivity
It is also similar to the Precision metric; however, it aims to calculate the proportion of actual
positive that was identified incorrectly. It can be calculated as True Positive or predictions that are
actually true to the total number of positives, either correctly predicted as positive or incorrectly
predicted as negative (true Positive and false negative).
The formula for calculating Recall is given below:

V. F-Scores
F-score or F1 Score is a metric to evaluate a binary classification model on the basis of
predictions that are made for the positive class. It is calculated with the help of Precision and Recall.
It is a type of single score that represents both Precision and Recall. So, the F1 Score can be
calculated as the harmonic mean of both precision and Recall, assigning equal weight to each of
them.
The formula for calculating the F1 score is given below:

VI. AUC-ROC
Sometimes we need to visualize the performance of the classification model on charts; then,
we can use the AUC-ROC curve. It is one of the popular and important metrics for evaluating the
performance of the classification model.
Firstly, let's understand ROC (Receiver Operating Characteristic curve) curve. ROC represents
a graph to show the performance of a classification model at different threshold levels. The curve
is plotted between two parameters, which are:
o True Positive Rate
o False Positive Rate
TPR or true Positive rate is a synonym for Recall, hence can be calculated as:

FPR or False Positive Rate can be calculated as:

To calculate value at any point in a ROC curve, we can evaluate a logistic regression model
multiple times with different classification thresholds, but this would not be much efficient. So, for
this, one efficient method is used, which is known as AUC.

Dr. M. SUPRIYA,

AP/CSE
AUC: Area Under the ROC curve
AUC is known for Area Under the ROC curve. As its name suggests, AUC calculates the two-
dimensional area under the entire ROC curve, as shown below image:

AUC calculates the performance across all the thresholds and provides an aggregate measure. The
value of AUC ranges from 0 to 1. It means a model with 100% wrong prediction will have an AUC
of 0.0, whereas models with 100% correct predictions will have an AUC of 1.0.
2. Performance Metrics for Regression
Regression is a supervised learning technique that aims to find the relationships between the
dependent and independent variables. A predictive regression model predicts a numeric or discrete
value. The metrics used for regression are different from the classification metrics. It means we
cannot use the Accuracy metric (explained above) to evaluate a regression model; instead, the
performance of a Regression model is reported as errors in the prediction. Following are the popular
metrics that are used to evaluate the performance of Regression models.
o Mean Absolute Error
o Mean Squared Error
o R2 Score
o Adjusted R2
I. Mean Absolute Error (MAE)
Mean Absolute Error or MAE is one of the simplest metrics, which measures the absolute
difference between actual and predicted values, where absolute means taking a number as Positive.
To understand MAE, let's take an example of Linear Regression, where the model draws a
best fit line between dependent and independent variables. To measure the MAE or error in
prediction, we need to calculate the difference between actual values and predicted values. But in
order to find the absolute error for the complete dataset, we need to find the mean absolute of the
complete dataset.
The below formula is used to calculate MAE:

Here,

Dr. M. SUPRIYA,

AP/CSE
Y is the Actual outcome, Y' is the predicted outcome, and N is the total number of data points.
II. Mean Squared Error
Mean Squared error or MSE is one of the most suitable metrics for Regression evaluation. It
measures the average of the Squared difference between predicted values and the actual value
given by the model.
The formula for calculating MSE is given below:

Here,
Y is the Actual outcome, Y' is the predicted outcome, and N is the total number of data points.
III. R Squared Score
R squared error is also known as Coefficient of Determination, which is another popular
metric used for Regression model evaluation. The R-squared metric enables us to compare our
model with a constant baseline to determine the performance of the model. To select the constant
baseline, we need to take the mean of the data and draw the line at the mean.
The R squared score will always be less than or equal to 1 without concerning if the values
are too large or small.

IV. Adjusted R Squared

Adjusted R squared, as the name suggests, is the improved version of R squared error. R
square has a limitation of improvement of a score on increasing the terms, even though the model
is not improving, and it may mislead the data scientists.
To overcome the issue of R square, adjusted R squared is used, which will always show a
lower value than R². It is because it adjusts the values of increasing predictors and only shows
improvement if there is a real improvement.
We can calculate the adjusted R squared as follows:

Here,
n is the number of observations
k denotes the number of independent variables
and Ra2 denotes the adjusted R2

Dr. M. SUPRIYA,

AP/CSE
 ASSESSING A CLASSIFICATION ALGORITHM’S PERFORMANCE
We will discuss the case of classification error, but the same methodology applies for
squared error in regression, log likelihoods in unsupervised learning, expected reward in
reinforcement learning, and so on, as long as we can write the appropriate parametric form for the
sampling distribution. We will also discuss nonparametric tests when no such parametric form can
be found.
Binomial Test
 Let us start with the case where we have a single training set T and a single validation set V
. We train our classifier on T and test it on V . We denote by p the probability that the
classifier makes a misclassification error.
 We do not know p; it is what we would like to estimate or test a hypothesis about.
 On the instance with index t from the validation set V , let us say xt denotes the correctness
of the classifier’s decision: xt is a 0/1 Bernoulli random variable that takes the value 1 when
the classifier commits an error and 0 when the classifier is correct.
 The binomial random variable X denotes the total number of errors:
X = Σt=1N xt
We would like to test whether the error probability p is less than or equal to some value p0 we
specify:
H0 : p ≤ p0 vs. H1 : p>p0
If the probability of error is p, the probability that the classifier commits j errors out of N is
P{X = j} = ( N/ j ) pj (1 − p)N-j
It is reasonable to reject p ≤ p0 if in such a case, the probability that binomial test we see X = e
errors or more is very unlikely. That is, the binomial test rejects the hypothesis if
P{X ≥ e} = Σx=eN ( N/ x ) p0x(1-p0 )N-x < α (19.10)
where α is the significance, for example, 0.05.
Approximate Normal Test
 If p is the probability of error, our point estimate is pˆ = X/N. Then, it is reasonable
to reject the null hypothesis if pˆ is much larger than p0.
 How large is large enough is given by the sampling distribution of pˆ and the
significance α. Because X is the sum of independent random variables from the same
distribution, the central limit theorem states that for large N, X/N is approximately
normal with mean p0 and variance p0(1 – p0). Then
X/N – p0 / √p0(1 − p0) ∼Z˙ (19.11)
The approximate normal test rejects the null hypothesis if this value for X = e is greater than
zα. Z0.05 is 1.64. This approximation will work well as long as N is not too small and p is

Dr. M. SUPRIYA,

AP/CSE
not very close to 0 or 1; as a rule of thumb, we require Np ≥ 5 and N(1 − p) ≥ 5.

COMPARING TWO CLASSIFICATION ALGORITHMS

T-TEST
Paired t-Test A paired t-test is a statistical test used to compare the means of two related
samples or matched pairs. It is used to test a hypothesis about the difference between the means of
the two samples. It is based on the assumption that the differences between the pairs are normally
distributed.
Dependent vs Independent Samples
There are two types of two-sample t-tests: dependent and independent.
In an independent two-sample t-test (also known as an unpaired t-test), the samples in the two
groups being compared are unrelated. The samples are drawn from two different populations or
groups of subjects, and the difference between the means of the two groups is calculated using the
means and variances of the two separate samples. A separate post covers the independent two-
sample t-test.
In a dependent two-sample t-test (also known as a paired t-test), the samples in the two groups
being compared are related in some way. For example, the samples may be pairs of measurements
taken on the same subjects or on subjects who are closely matched in some other way. In this case,
the difference between the means of the two groups is calculated by taking the differences between
the pairs of measurements and treating these differences as a single sample. This post covers the
paired t-test or dependent two-sample [Link] choice between a dependent or independent t-
test depends on the nature of your data and the research question you are trying to answer.
Steps in Paired t Test
To conduct a paired t-test, the following steps are typically followed:
1. Specify the null and alternative hypotheses. The null hypothesis is usually that there is no
difference between the means of the two samples, while the alternative hypothesis is that
there is a difference between the means.
2. Collect data for the two related samples or matched pairs.
3. Calculate the differences between the pairs and find the mean and standard deviation
of the differences.
4. Calculate the test statistic, which is the difference between the means of the two samples,
divided by the standard error of the mean.
5. Determine the critical value of the test statistic based on the significance level (alpha) of
the test and the degrees of freedom. The degrees of freedom are calculated as the sample
size minus 1.

Dr. M. SUPRIYA,

AP/CSE
 6. Compare the calculated test statistic to the critical value to determine whether to reject
or fail to reject the null hypothesis. If the calculated test statistic exceeds the critical value,
the null hypothesis is rejected, and the alternative hypothesis is accepted.
Conditions for Paired t-Test
To conduct a valid paired t-test, the following conditions must be met:
1. The samples must be related or matched in some way.
2. The differences between the pairs must be approximately normally distributed.
3. The population standard deviation of the differences is unknown, and the sample size is
small (less than 30).
Typical Null and Alternate Hypothesis in Paired t-test
a) Two-Tail Test:
In a paired t-test, the null hypothesis is that there is no average difference between the two samples.
This can be expressed as:
H0: μ1 - μ2 = 0
where μ1 is the mean of the first sample and μ2 is the mean of the second sample.
The alternate hypothesis is the opposite of the null hypothesis and is that there is a difference
between the means of the two samples. This can be expressed as:
Ha: μ1 - μ2 ≠ 0
b) Left Tail Test:
A left-tailed hypothesis is one in which the mean of the first sample is less than the mean of the
second sample. This can be expressed as:
H0: μ1 - μ2 >= 0
Ha: μ1 - μ2 < 0
c) Right Tail Test:
A right-tailed hypothesis is one in which the mean of the first sample is greater than the mean of the
second sample. This can be expressed as:
H0: μ1 - μ2 <= 0
Ha: μ1 - μ2 > 0
Calculating Test Statistic
The t-score represents the number of standard errors that the difference between the means of the
two samples from zero. It is used to determine whether the difference between the means is
statistically significant.
The formula for calculating the t-score in a paired t-test is as follows:

Dr. M. SUPRIYA,

AP/CSE
where:
 d¯ is the mean of the difference scores
 sd is the standard deviation of the difference scores
 n is the number of pairs of observations

McNemar’s Test
Given a training set and a validation set, we use two algorithms to train two
classifiers on the training set and test them on the validation set and compute their
errors. A contingency table, like the one shown here, is an array of natural numbers in
matrix form representing counts, or frequencies:

Under the null hypothesis that the classification algorithms have the same error

Dr. M. SUPRIYA,

AP/CSE
rate, we expect e01 = e10 and these to be equal to (e01+e10)/2. We have the chi-square
statistic with one degree of freedom
((|e01 – e10| − 1)2 /e01 + e10 ) ∼ X2 1
and McNemar’s test rejects the hypothesis that the two classification algorithms
have the same error rate at significance level α if this value is greater than X2 α,1. For
α = 0.05, X2 0.05,1 = 3.84
K-Fold Cross-Validated Paired t Test
This set uses K-fold cross-validation to get K training/validation set pairs. We use
the two classification algorithms to train on the training sets Ti, i = 1,...,K, and test on the
validation sets Vi. The error percentages of the classifiers on the validation sets are
recorded as p1i and p2 i .
If the two classification algorithms have the same error rate, then we expect them
to have the same mean, or equivalently, that the difference of their means is 0. The difference in
error rates on fold i is pi= p1 i −p2 i. This is a paired test; that is, for each i, both algorithms see the
same training and validation sets. When this is done K times, we have a distribution of pi containing
K points. Given that p1i and p2 i are both (approximately) normal, their difference pi is also normal.
The null hypothesis is that this distribution has 0 mean:
H0 : μ = 0 vs. H1 : μ ≠ 0
We define
m = Σi=1K pi /K , S2 = Σi=1K (p i-m)2 /K – 1
Under the null hypothesis that μ = 0, we have a statistic that is t distributed with K − 1
degrees of freedom:
√ K(m − 0)/ S = √ K · m/ S ∼ tK-1 (19.14)
 Thus the K-fold cv paired t test rejects the hypothesis that two classification algorithms
have the same error rate at significance level α if this value is outside the interval
(−tἀ/2,K-1, tἀ/2,K-1). T0.025,9 = 2.26 and t0.025,29 = 2.05.
 If we want to test whether the first algorithm has less error than the second, we need a one-
sided hypothesis and use a one-tailed test:
H0 : μ ≥ 0 vs. H1 : μ < 0
If the test rejects, our claim that the first one has significantly less error is supported.

Dr. M. SUPRIYA,

AP/CSE
 K-Fold Cross Validation
In K-Fold Cross Validation, we split the dataset into k number of subsets (known as folds) then we
perform training on the all the subsets but leave one(k-1) subset for the evaluation of the trained
model. In this method, we iterate k times with a different subset reserved for testing purpose each
time.
Note: It is always suggested that the value of k should be 10 as the lower value of k takes towards
validation and higher value of k leads to LOOCV method.
Example of K Fold Cross Validation
The diagram below shows an example of the training subsets and evaluation subsets generated in
k-fold cross-validation. Here, we have total 25 instances. In first iteration we use the first 20 percent
of data for evaluation, and the remaining 80 percent for training ([1-5] testing and [5-25] training)
while in the second iteration we use the second subset of 20 percent for evaluation,
and the remaining three subsets of the data for training ([5-10] testing and [1-5 and 10-25]
training), and so on.

Advantages:
5. Overcoming Overfitting: Cross validation helps to prevent overfitting by providing a more
robust estimate of the model’s performance on unseen data.
6. Model Selection: Cross validation can be used to compare different models and select the
one that performs the best on average.
7. Hyperparameter tuning: Cross validation can be used to optimize the hyperparameters of a
model, such as the regularization parameter, by selecting the values that result in the best
performance on the validation set.

Dr. M. SUPRIYA,

AP/CSE
 8. Data Efficient: Cross validation allows the use of all the available data for both training and
validation, making it a more data-efficient method compared to traditional validation
techniques.
Disadvantages:
4. Computationally Expensive: Cross validation can be computationally expensive, especially
when the number of folds is large or when the model is complex and requires a long time to
train.
5. Time-Consuming: Cross validation can be time-consuming, especially when there are many
hyperparameters to tune or when multiple models need to be compared.
6. Bias-Variance Tradeoff: The choice of the number of folds in cross validation can impact
the bias-variance tradeoff, i.e., too few folds may result in high variance, while too many
folds may result in high bias.

Dr. M. SUPRIYA,

AP/CSE