UNIT II SUPERVISED LEARNING 11

Supervised Learning, learning a Class from Examples, Noise, Learning Multiple Classes,
Regression, Model Selection and Generalization, Classification, Decision trees – Support vector
machines -Neural networks (Gradient Descent and Back Propagation), Naïve Bayes Algorithm,
Linear Regression and Logistic Regression, Random Forest, Ensemble Learning

2.1 SUPERVISED LEARNING - LEARNING A CLASS FROM EXAMPLES

Some common classes of machine learning algorithms:
● Generalized linear models (e.g., logistic regression)
● Support vector machines (e.g., linear SVM, RBF-kernel SVM)
● Artificial neural networks (e.g., multi-layer perceptrons)
● Tree- or rule-based models (e.g., decision trees)
● Graphical models (e.g., Bayesian networks)
● Ensembles (e.g., Random Forest)
● Instance-based learners (e.g., K-nearest neighbors)

Given a training set of labeled examples (positive and negative) to learn a classifier that can be
used to predict unseen examples, or to understand the data.
• Input representation: To decide what attributes (features) to use to describe the input
patterns (examples, instances). This implies ignoring other attributes as irrelevant.

• Hypothesis (model) class H: the set of classifier functions. Ideally, the true class
distribution C can be represented by a function in H (exactly, or with a small error).
CS1701A Machine Learning Department of CSE- SAEC Unit II 1
 Input representation
• Assigning a class label to an unknown instance from instances of known assignments of labels.
• An object will have large number of features which may contribute to the assignment of the
labels.
• Only important features as inputs for assigning the class labels.
• These features are referred to as the “input features” /”input representation” for the problem.
Example Consider the problem of assigning the label “family car” or “not family car” to cars. Let
us assume that the features that separate a family car from other cars are the price and engine
power. These attributes or features constitute the input representation for the problem. While
deciding on this input representation, we are ignoring various other attributes like seating capacity
or colour as irrelevant.

Hypothesis: In a binary classification problem, a hypothesis is a statement or a proposition

purporting to explain a given set of facts or observations.
Hypothesis space: The hypothesis space for a binary classification problem is a set of hypotheses
for the problem that might possibly be returned by it.
Consistency and satisfying: Let x be an example in a binary classification problem and let c(x)
denote the class label assigned to x (c(x) is 1 or 0). Let D be a set of training examples for the
problem. Let h be a hypothesis for the problem and h(x) be the class label assigned to x by the
hypothesis h.
(a) We say that the hypothesis h is consistent with the set of training examples D if h(x) = c(x) for
all x ∈ D.
(b) We say that an example x satisfies the hypothesis h if h(x) = 1.
Examples 1. Consider the set of observations of a variable x with the associated class labels given in Table

CS1701A Machine Learning Department of CSE- SAEC Unit II 2

 Table Sample data to illustrate the concept of hypotheses
Figure shows the data plotted on the x-axis.

Data in Table with hollow dots representing positive examples and solid dots representing negative
examples

CS1701A Machine Learning Department of CSE- SAEC Unit II 3

Ordering of hypotheses
Let X be the set of all possible examples for a binary classification problem and let h’ and h’’ be
two hypotheses for the problem

Version space
Consider a binary classification problem. Let D be a set of training examples and H a hypothesis
space for the problem. The version space for the problem with respect to the set D and the space
H is the set of hypotheses from H consistent with D; that is, it is the set

CS1701A Machine Learning Department of CSE- SAEC Unit II 4

0

CS1701A Machine Learning Department of CSE- SAEC Unit II 5

 2.2 Noise

Noise and their sources

Noise is any unwanted anomaly in the data . Noise may arise due to several factors:
1. There may be imprecision in recording the input attributes, which may shift the data points in
the input space.
2. There may be errors in labeling the data points, which may relabel positive instances as
negative and vice versa. This is sometimes called teacher noise.
3. There may be additional attributes, which we have not taken into account, that affect the label
of an instance. Such attributes may be hidden or latent in that they may be unobservable. The
effect of these neglected attributes is thus modeled as a random component and is included in
“noise.”
Effect of noise

● Noise distorts data. When there is noise in data, learning problems may not produce
accurate results.

● Also, simple hypotheses may not be sufficient to explain the data and so complicated
hypotheses may have to be formulated.

● This leads to the use of additional computing resources and the needless wastage of such
resources.
For example, in a binary classification problem with two variables, when there is noise, there may
not be a simple boundary between the positive and negative instances and to separate them.
The effect of these neglected attributes is thus modeled as a random component and is
included in “noise.”
A rectangle can be defined by four numbers, but to define a more complicated shape one needs a
more complex model with a much larger number of parameters. With a complex model, one can
make a perfect fit to the data and attain zero error; see the wiggly shape in figure . Another
possibility is to keep the model simple and allow some error; see the rectangle in figure.
Using the simple rectangle (unless its training error is much bigger) makes more sense
because of the following:
1. It is a simple model to use. It is easy to check whether a point is inside or outside a rectangle
and we can easily check, for a future data instance, whether it is a positive or a negative instance.

CS1701A Machine Learning Department of CSE- SAEC Unit II 6

Figure When there is noise, there is not a simple boundary between the positive and negative instances, and zero
misclassification error may not be possible with a simple hypothesis. A rectangle is a simple hypothesis with four
parameters defining the corners. An arbitrary closed form can be drawn by piecewise functions with a larger number
of control points.

2. It is a simple model to train and has fewer parameters. It is easier to find the corner values
of a rectangle than the control points of an arbitrary shape. With a small training set when the
training instances differ a little bit, we expect the simpler model to change less than a complex
model: A simple model is thus said to have less variance. On the other hand, a too simple model
assumes more, is more rigid, and may fail if indeed the underlying class is not that simple: A
simpler model has more bias. Finding the optimal model corresponds to minimizing both the bias
and the variance.
3. It is a simple model to explain. A rectangle simply corresponds to defining intervals on the
two attributes. By learning a simple model, we can extract information from the raw data given in
the training set.
4. If indeed there is mislabeling or noise in input and the actual class is really a simple model
like the rectangle, then the simple rectangle, because it has less variance and is less affected by
single instances, will be a better discriminator than the wiggly shape, although the simple one may
make slightly more errors on the training set.
Given comparable empirical error, we say that a simple (but not too simple) model would
generalize better than a complex model.
This principle Occam’s razor is known as Occam’s razor, which states that simpler
explanations are more plausible and any unnecessary complexity should be shaved off.

CS1701A Machine Learning Department of CSE- SAEC Unit II 7

 2.3 LEARNING MULTIPLE CLASSES
Learning multiple classes
Binary classification problems are more common . In a general case there may be more than two
classes. Two methods are generally used to handle such cases. These methods are known by the
names “one-against-all" and “one-against-one”.
Procedures for learning multiple classes
a) “One-against all” method
Consider the case where there are K classes denoted by C1; : : : ;CK. Each input instance belongs
to exactly one of them.
We view a K-class classification problem as K two-class problems. In the i-th two-class problem,
the training examples belonging to Ci are taken as the positive examples and the examples of all
other classes are taken as the negative examples. So, we have to find K hypotheses h1; : : : ; hK
where hi is defined by

For a given x, ideally only one of hi(x) is 1 and then we assign the class Ci to x. But, when no, or,
two or more, hi(x) is 1, we cannot choose a class. In such a case, we say that the classifier rejects
such cases.
b) “One-against-one” method
In the one-against-one (OAO) (also called one-vs-one (OVO)) strategy, a classifier is constructed
for each pair of classes. If there are K different class labels, a total of K(K – 1)/2 classifiers are
constructed. An unknown instance is classified with the class getting the most votes. Ties are
broken arbitrarily.
For example, let there be three classes, A, B and C. In the OVO method we construct 3(3 − 1)/2 =
3 binary classifiers. Now, if any x is to be classified, we apply each of the three classifiers to x.
Let the three classifiers assign the classes A, B, B respectively to x. Since a label to x is assigned
by the majority voting, in this example, we assign the class label of B to x
In our example of learning a family car, we have positive examples belonging to the class family
car and the negative examples belonging to all other cars. This is a two-class problem. In the
general case, we have K

• With K classes, we can code the label as an integer y = k ∈ {1, . . . ,K}, or as a one-of-K binary
vector y = (y1, . . . , yK)T ∈ {0, 1}K (containing a single 1 in position k).
• One approach for K-class classification: consider it as K two-class classification problems, and
minimize the total empirical error:
CS1701A Machine Learning Department of CSE- SAEC Unit II 8
 where yn is coded as one-of-K and hk is the two-class classifier for problem k, i.e., hk(x) ∈
{0, 1}.
• Ideally, for a given pattern x only one hk(x) is one. When no, or more than one, hk(x) is one
then the classifier is in doubt and may reject the pattern.

Figure There are three classes: family car, sports car, and luxury sedan. There are three hypotheses induced,
each one covering the instances of one class and leaving outside the instances of the other two classes. ‘?’ are
reject regions where no, or more than one, class is chosen.
In our example of learning a family car, we used only one hypothesis and only modeled the positive
examples. Any negative example outside is not a family car. Alternatively, sometimes we may
prefer to build two hypotheses, one for the positive and the other for the negative instances. This
assumes a structure also for the negative instances that can be covered by another hypothesis.
Separating family cars from sports cars is such a problem; each class has a structure of its own.
The advantage is that if the input is a luxury sedan, we can have both hypotheses decide negative
and reject the input.
If in a dataset, we expect to have all classes with similar distribution— shapes in the input space—
then the same hypothesis class can be used for all classes. For example, in a handwritten digit
recognition dataset, we would expect all digits to have similar distributions. But in a medical
diagnosis dataset, for example, where we have two classes for sick and healthy people, we may
have completely different distributions for the two classes; there may be multiple ways for a person
to be sick, reflected differently in the inputs: All healthy people are alike; each sick person is sick
in his or her own way.

CS1701A Machine Learning Department of CSE- SAEC Unit II 9

 2.4 REGRESSION

A regression problem is the problem of determining a relation between one or more independent
variables and an output variable which is a real continuous variable, given a set of observed values
of the set of independent variables and the corresponding values of the output variable.
Examples
1. Let us say we want to have a system that can predict the price of a used car. Inputs are the car
attributes such as brand, year, engine capacity, mileage, and other information that we
believe affect a car’s worth. The output is the price of the car.
2. Consider the navigation of a mobile robot, say an autonomous car. The output is the angle by
which the steering wheel should be turned at each time, to advance without hitting obstacles
and deviating from the route. Inputs are provided by sensors on the car like a video camera,
GPS, and so forth.
3. In finance, the capital asset pricing model uses regression for analyzing and quantifying the
systematic risk of an investment.
4. In economics, regression is the predominant empirical tool. For example, it is used to predict
consumption spending, inventory investment, purchases of a country’s exports, spending on
imports, labor demand, and labor supply.
Different regression models
The different regression models are defined based on type of functions used to represent the
relation between the dependent variable y and the independent variables.
1. Simple linear regression
Assume that there is only one independent variable x. If the relation between x and y is modeled
by the relation
y = a + bx
then we have a simple linear regression.
2. Multiple regression
Let there be more than one independent variable, say x1, x2, : : :, xn, and let the relation between y
and the independent variables be modeled as

then it is case of multiple linear regression or multiple regression.

CS1701A Machine Learning Department of CSE- SAEC Unit II 10

3. Polynomial regression
Let there be only one variable x and let the relation between x y be modeled as

for some positive integer n > 1, then we have a polynomial regression.

4. Logistic regression
Logistic regression is used when the dependent variable is binary (0/1, True/False, Yes/No)
in nature. Even though the output is a binary variable, what is being sought is a probability
function which may take any value from 0 to 1.

2.5 MODEL SELECTION AND GENERALIZATION

Model selection
• There is no universally accepted definition of the term “model”. It may be understood as some
mathematical expression or equation, or some mathematical structures such as graphs and
trees, or a division of sets into disjoint subsets, or a set of logical “if : : : then : : : else : : :”
rules, or some such thing.
• In order to formulate a hypothesis for a problem, we have to choose some model and the term
“model selection” has been used to refer to the process of choosing a model.
• However, the term has been used to indicate several things. In some contexts it has been used
to indicate the process of choosing one particular approach from among several different
approaches. This may be choosing an appropriate algorithms from a selection of possible
algorithms, or choosing the sets of features to be used for input, or choosing initial values for
certain parameters.
• Sometimes “model selection” refers to the process of picking a particular mathematical model
from among different mathematical models which all purport to describe the same data set. It
has also been described as the process of choosing the right inductive bias.
Inductive bias
In a learning problem we only have the data. But data by itself is not sufficient to find the solution.
We should make some extra assumptions to have a solution with the data we have. The set of
assumptions we make to have learning possible is called the inductive bias of the learning
algorithm.
One way we introduce inductive bias is when we assume a hypothesis class.
Examples

CS1701A Machine Learning Department of CSE- SAEC Unit II 11

 ● In learning the class of family car, there are infinitely many ways of separating the positive
examples from the negative examples. Assuming the shape of a rectangle is an inductive
bias.

● In regression, assuming a linear function is an inductive bias.

The model selection is about choosing the right inductive bias.
Advantages of a simple model
Even though a complex model may not be making any errors in prediction, there are certain
advantages in using a simple model.
1. A simple model is easy to use.
2. A simple model is easy to train. It is likely to have fewer parameters.
3. It is easier to find the corner values of a rectangle than the control points of an arbitrary
shape.
4. A simple model is easy to explain.
5. A simple model would generalize better than a complex model. This principle is known as
Occam’s razor, which states that simpler explanations are more plausible and any
unnecessary complexity should be shaved off.
Remarks
A model should not be too simple! With a small training set when the training instances differ a
little bit, we expect the simpler model to change less than a complex model: A simple model is
thus said to have less variance. On the other hand, a too simple model assumes more, is more rigid,
and may fail if indeed the underlying class is not that simple. A simpler model has more bias.
Finding the optimal model corresponds to minimizing both the bias and the variance.
A simple model has less variance and more bias
Generalization
How well a model trained on the training set predicts the right output for new instances is called
generalization.
Generalization refers to how well the concepts learned by a machine learning model apply to
specific examples not seen by the model when it was learning. The goal of a good machine learning
model is to generalize well from the training data to any data from the problem domain. This allows
us to make predictions in the future on data the model has never seen. Overfitting and
underfitting are the two biggest causes for poor performance of machine learning algorithms. The
model should be selected having the best generalisation. This is said to be the case if these
problems are avoided.

CS1701A Machine Learning Department of CSE- SAEC Unit II 12

• Underfitting
Underfitting is the production of a machine learning model that is not complex enough to
accurately capture relationships between a dataset, its features and a target variable.
• Overfitting
Overfitting is the production of an analysis which corresponds too closely or exactly to a particular
set of data, and may therefore fail to fit additional data or predict future observations reliably.

(a) Given dataset (b) “Just right” model

(c) Underfitting model (d) Overfitting model

Testing generalisation: Cross-validation

We can measure the generalization ability of a hypothesis, namely, the quality of its inductive bias,
if we have access to data outside the training set. We simulate this by dividing the training set we
have into two parts. We use one part for training (that is, to find a hypothesis), and the remaining
part is called the validation set and is used to test the generalization ability. Assuming large enough
training and validation sets, the hypothesis that is the most accurate on the validation set is the best
one (the one that has the best inductive bias). This process is called cross-validation.

CS1701A Machine Learning Department of CSE- SAEC Unit II 13

 2.6 DECISION TREES

● A decision tree is a simple model for supervised classification. It is a hierarchical data

structure implementing the divide-and-conquer strategy.
● A decision tree is a graphical representation of all the possible solutions to a decision based
on certain conditions.
● It is an efficient nonparametric method, which can be used for both classification and
regression
● The decision tree algorithms such as ID3, C4.5,CART are very popular inductive inference
algorithms, and they are successfully applied to many learning tasks.
There are two types of decision trees.
1. Classification trees
Tree models where the target variable can take a discrete set of values are called classification
trees. In these tree structures, leaves represent class labels and branches represent conjunctions of
features that lead to those class labels.
2. Regression trees
Decision trees where the target variable can take continuous values (real numbers) like the price
of a house, or a patient’s length of stay in a hospital, are called regression trees.

Structure of a Decision Tree

● A decision tree is composed of internal decision nodes, decision nodes and terminal leaves.
● Each decision node m implements a test function fm(x) with discrete outcomes labeling the
branches.
● Given an input, at each node, a test is applied and one of the branches is taken depending on
the outcome.
● This process starts at the root and is repeated recursively until a leaf node is hit, at which point
the value written in the leaf constitutes the output.

CS1701A Machine Learning Department of CSE- SAEC Unit II 14

CS1701A Machine Learning Department of CSE- SAEC Unit II 15
Most decision tree algorithms differ in the following ways:
• Splitting criterion: information gain (Entropy, Information Gain, Gini impurity,
misclassification error), use of statistical tests, objective function, etc.
• Binary split vs. multi-way splits
• Discrete vs. continuous variables
• Pre- vs. post-pruning
Node purity

● Decision nodes are typically impure, or a mixture of both classes of the target variable

● Pure nodes are those that have one class — hence the term pure.
Variable selection criterion
Variables are selected on a complex statistical criterion which is applied at each decision node.
Now, variable selection criterion in Decision Trees can be done via the following approaches:
1. Entropy
2. Information Gain
3. Gini Index
1. Entropy
● Entropy is a measure of the impurity in a collection of training examples
● Entropy is the measure of the degree of randomness or uncertainty in the dataset.
● In the case of classifications it measures the randomness based on the distribution of class
labels in the dataset.
2. Information Gain
● Information gain is a measure of the effectiveness of an attribute in classifying the training
data.

CS1701A Machine Learning Department of CSE- SAEC Unit II 16

 ● Information gain measures the expected reduction in entropy by partitioning the examples
according to an attribute.
● The standard criterion that is chosen for splitting in decision trees is information gain (the
better the split, the higher the information gain).
3. Gini Impurity or index:
● Gini Impurity is a score that evaluates how accurate a split is among the classified groups.
● The Gini Impurity evaluates a score in the range between 0 and 1, where 0 is when all
observations belong to one class, and 1 is a random distribution of the elements within
classes.
● In this case, we want to have a Gini index score as low as possible.
● Gini Index is the evaluation metric we shall use to evaluate our Decision Tree Model.

Two popular decision tree algorithms are ID3 and CART

1. ID3 (Iterative Dichotomizer 3)

● Purpose: Primarily used for classification tasks.
● Splitting Criteria: Information Gain.
● Tree Structure: Can produce trees with multiple branches at each node.

How it works:

1. Calculate Information Gain: For each attribute, calculate the information gain, which
measures the decrease in entropy (uncertainty) after splitting the data based on that
attribute.
2. Select Best Attribute: Choose the attribute with the highest information gain as the root
node.
3. Create Child Nodes: Split the dataset into subsets based on the values of the selected
attribute.
4. Recursively Apply: Repeat steps 1-3 for each subset until a pure node (all instances belong
to the same class) is reached or there are no more attributes to split on.

CS1701A Machine Learning Department of CSE- SAEC Unit II 17

2. CART (Classification and Regression Trees)
● Purpose: Can be used for both classification and regression tasks.
● Splitting Criteria: Gini Impurity for classification, and squared error for regression.
● Tree Structure: Always produces binary trees (two branches per node).

How it works:

1. Calculate Splitting Criterion: For each attribute, calculate the Gini impurity (or squared
error for regression) for all possible splits.
2. Select Best Split: Choose the split that minimizes the impurity (or squared error).
3. Create Child Nodes: Split the dataset into two subsets based on the selected split.
4. Recursively Apply: Repeat steps 1-3 for each subset until a stopping criterion is met (e.g.,
minimum number of instances in a node).

CS1701A Machine Learning Department of CSE- SAEC Unit II 18

Overfitting in decision trees
● Decision tree construction proceeds until all leaves are “pure” , i.e. all examples are from
the same class.
● As the tree grows, the generalization performance can start to degrade, because the
algorithm is including irrelevant attributes/tests/outliers.
How can we avoid this?
● Early stopping: Stop growing the tree when further splitting the data does not improve
information gain of the validation set.
● Post pruning: Grow a full tree, then prune the tree by eliminating lower nodes that have
low information gain on the validation set..

CS1701A Machine Learning Department of CSE- SAEC Unit II 19

 2.7 SUPPORT VECTOR MACHINE

● Support Vector Machine (later generalized under the name kernel machine) is a relatively
new classification method for both linear and nonlinear data

● It uses a nonlinear mapping to transform the original training data into a higher dimension

● With the new dimension, it searches for the linear optimal separating hyperplane(i.e., “decision
boundary”)

● With an appropriate nonlinear mapping to a sufficiently high dimension, data from two classes
can always be separated by a hyperplane

● SVM finds this hyperplane using support vectors (“essential” training tuples) and
margins(defined by the support vectors)

● Linear classifiers: logistic regression, Gaussian classes with common covariance, now linear
SVMs.
Features: training can be slow but accuracy is high owing to their ability to model complex
nonlinear decision boundaries (margin maximization)
• Used both for classification and prediction
• Applications:

✔ handwritten digit recognition, object recognition, speaker identification, benchmarking

time-series prediction tests

✔ Face detection, image classification, text categorization

The support vector machine has been popular in recent years for a number of reasons:
1. It is a discriminant-based method and uses Vapnik’s principle to never solve a more complex
problem as a first step before the actual problem
2. After training, the parameter of the linear model, the weight vector, can be written down in
terms of a subset of the training set, which are the so-called support vectors.
3. The output is written as a sum of the influences of support vectors and these are given by kernel
functions that are application-specific measures of similarity between data instances.
4. Typically in most learning algorithms, data points are represented as vectors, and either dot
product (as in the multilayer perceptrons) or Euclidean distance (as in radial basis function
networks) is used. A kernel function allows us to go beyond that
5. Kernel-based algorithms are formulated as convex optimization problems, and there is a single
optimum that we can solve for analytically
CS1701A Machine Learning Department of CSE- SAEC Unit II 20
SVM can be of two types:
Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can
be classified into two classes by using a single straight line, then such data is termed as
linearly separable data, and classifier is used called as Linear SVM classifier.
Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if
a dataset cannot be classified by using a straight line, then such data is termed as non-linear
data and classifier used is called as Non-linear SVM classifier.

● The goal of the SVM algorithm is to create the best line or decision boundary that can
segregate n-dimensional space into classes so that we can easily put the new data point in
the correct category in the future.

● This best decision boundary is called a hyperplane.

● SVM chooses the extreme points/vectors that help in creating the hyperplane.

● These extreme cases are called as support vectors, and hence algorithm is termed as
Support Vector Machine

Consider the below diagram in which there are two different categories that are classified using a
decision boundary or hyperplane:

Linear Classifiers
Consider a two dimensional dataset with two classes

CS1701A Machine Learning Department of CSE- SAEC Unit II 21

Key Terminologies
Margin: The margin is the distance between the hyperplane and the nearest data points (support
vectors) on either side.
Classifier Margin: Define the margin of a linear classifier as the width that the boundary could
be increased by before hitting a data point.
Maximum Margin: The maximum margin linear classifier is the linear classifier with the
maximum margin. This is the simplest kind of SVM (Called Linear SVM)
Support Vectors: These are the data points that lie closest to the decision boundary (hyperplane)
of a SVM.
Hyperplane: A hyperplane is a decision boundary that separates data points of different classes

How they work together

● The SVM algorithm iteratively adjusts the hyperplane's position and orientation to
maximize the margin.
● The support vectors are the critical points that influence the hyperplane's position.
● Maximizing the margin helps to improve the model's generalization ability, reducing the
risk of overfitting.

Where

CS1701A Machine Learning Department of CSE- SAEC Unit II 22

SVM—Linearly Separable

Soft Margin and Hard Margin

● Soft margin is a technique used in Support Vector Machine (SVM) classification that
allows for some misclassification of data points in order to achieve a wider margin and a
more flexible decision boundary.
● In traditional SVM classification with a hard margin, the goal is to find a hyperplane that
completely separates the two classes of data with no misclassification. However, in real-
world datasets, this is often not possible due to noise, outliers, or other factors that make
the data non-separable.
● The objective is to find the optimal hyperplane that maximizes the margin while also
minimizing the amount of misclassification.
● Cross Validation is used to determine the best soft margin.

CS1701A Machine Learning Department of CSE- SAEC Unit II 23

Kernel Trick and Kernel Function

1. Kernel function: In SVM, a kernel function is used to transform the input data into a
higher dimensional space where it can be more easily separated. Popular kernel functions
include linear, polynomial, and radial basis function (RBF) kernels.
2. Kernel trick: Kernel trick is a technique used in machine learning, specifically in Support
Vector Machine (SVM) algorithms, to transform a low-dimensional input space to a higher
dimensional feature space without actually computing the coordinates of the data in the
higher dimensional space.
3. By using the kernel function to compute the dot product, the data points are effectively
mapped to a higher dimensional space without explicitly computing their coordinates.

/Non-linear SVMs
• Datasets that are linearly separable with some noise work out great:

• But what are we going to do if the dataset is just too hard?

• How about… mapping data to a higher-dimensional space Machine

Non-linear SVMs: Feature spaces

General idea: the original feature space can always be mapped to some higher dimensional feature
space where the training set is separable:

CS1701A Machine Learning Department of CSE- SAEC Unit II 24

Key Points
● SVMs are particularly effective for linearly separable data.
● For non-linearly separable data, kernel tricks can be used to map the data into a higher-
dimensional space where it becomes linearly separable.
● SVMs are known for their strong generalization performance and ability to handle high-
dimensional data.

2.8 NEURAL NETWORKS (GRADIENT DESCENT AND BACK PROPAGATION)

● The study of artificial neural networks (ANNs) has been inspired in part by the observation
that biological learning systems are built of very complex webs of interconnected neurons.

● Artificial neural networks (ANNs) provide a general, practical method for learning real-valued,
discrete-valued, and vector-valued functions from examples.

● Algorithms such as BACKPROPAGATION and gradient descent to tune network

parameters to best fit a training set of input-output pairs.

● ANN learning is robust to errors in the training data and has been successfully applied to
problems such as interpreting visual scenes, speech recognition, and learning robot control
strategies.

● A neural network model is trained using the stochastic gradient descent optimization
algorithm and weights are updated using the backpropagation of error algorithm
Types of Neural Networks
There are many different types of neural networks and each of the neural network types is specific
to certain business scenarios and data patterns.

CS1701A Machine Learning Department of CSE- SAEC Unit II 25

Following are some of the most widely used neural networks.
● Perceptron
● Feed-Forward Neural Network
● Radial Basis Function Neural Network (RBF):
● Convolutional Neural Network (CNN)
● Recurrent neural network (RNN)
● Auto Encoder (AE)
● Hopfield Network (HN)
● Boltzmann Machine (BM)
● Deep Belief Network(DBN)
● Generative Adversarial Network (GAN)
Properties of Artificial Neural Networks
• A large number of very simple, neuron-like processing elements called units,
• A large number of weighted, directed connections between pairs of units
– Weights may be positive or negative real values
• Local processing in that each unit computes a function based on the outputs of a limited
number of other units in the network
• Each unit computes a simple function of its input values, which are the weighted outputs from
other units.
– If there are n inputs to a unit, then the unit's output, or activation is defined by
a = g((w1 * x1) + (w2 * x2) + ... + (wn * xn)).
– Each unit computes a (simple) function g of the linear combination of its inputs.
• Learning by tuning the connection weights
1. PERCEPTRON

One type of ANN system is based on a unit called a perceptron. A perceptron takes a vector of
real-valued inputs, calculates a linear combination of these inputs, then outputs a 1 if the result is
greater than some threshold and -1 otherwise.

● Perceptron is a Linear Threshold Unit (LTU).

● A perceptron takes a vector of real-valued inputs, calculates a linear combination of these

inputs, then outputs 1 if the result is greater than some threshold and -1 otherwise.

● Given inputs xl through xn , the output o(x1, . . . , xn) computed by the perceptron is:

● each wi is a real-valued constant, or weight, that determines the contribution of input xi to

the perceptron output.
CS1701A Machine Learning Department of CSE- SAEC Unit II 26
 ● The quantity (-w0 ) is a threshold that the weighted combination of inputs must surpass in
order for the perceptron to output 1.
– To simplify notation, we imagine an additional constant input x0 = 1
Neuron consists of three basic components –weights, thresholds and a single activation
function

Figure: Perceptron

Perceptron - Learning

● Learning a perceptron involves choosing values for weights w0, …,wn

● The space H of candidate hypotheses considered in perceptron learning is the set of all
possible real-valued weight vectors
Representational Power of Perceptrons

● A perceptron represents a hyperplane decision surface in the n-dimensional space of

instances.

CS1701A Machine Learning Department of CSE- SAEC Unit II 27

 ● The perceptron outputs 1 for instances lying on one side of the hyperplane and outputs -1
for instances lying on the other side.

● The equation for this decision hyperplane is

● Some sets of positive and negative examples cannot be separated by any hyperplane. Those
that can be separated are called linearly separable sets of examples.

● A single perceptron can be used to represent many boolean functions.

o AND, OR, NAND, NOR are representable by a perceptron
o XOR cannot be representable by a perceptron

PERCEPTRON TRAINING RULE

To learn an acceptable weight vector
• Begin with random weights, then iteratively apply the perceptron to each training example,
modifying the perceptron weights whenever it misclassifies an example.

✔ If the training example classifies correctly, weights are not updated.

• This process is repeated, iterating through the training examples as many times as needed
until the perceptron classifies all training examples correctly.

✔ Each pass through all of the training examples is called one epoch

• Weights are modified at each step according to perceptron training rule which revises the
weight wi associated with input xi according to the rule

where

● If the output is correct (t=o) the weights wi are not changed

● If the output is incorrect (t≠ o) the weights wi are changed such that the output of the
perceptron for the new weights is closer to t.

● The algorithm converges to the correct classification

✔ if the training data is linearly separable

✔ and ⎜ is sufficiently small

CS1701A Machine Learning Department of CSE- SAEC Unit II 28

CS1701A Machine Learning Department of CSE- SAEC Unit II 29
2. TRAINING MULTI-LAYER NETWORKS: BACK-PROPAGATION
The simplest kind of feed-forward network is a multilayer perceptron (MLP), as shown in
Figure

CS1701A Machine Learning Department of CSE- SAEC Unit II 30

● Backpropagation is a fundamental algorithm used to train artificial neural networks,
especially in feed-forward neural networks.
● It is an efficient method for computing gradients needed to perform gradient-based
optimization of the weights in a multi-layer network

● It's an iterative process that helps minimize the cost function by adjusting the network's weights
and biases.
● Use gradient descent to learn the weights

The operations of the Backpropagation neural networks can be divided into two steps:
Feedforward and Backpropagation.
o In the feedforward step, an input pattern is applied to the input layer and its effect propagates,
layer by layer, through the network until an output is produced.
o The network's actual output value is then compared to the expected output, and an error signal
is computed for each of the output nodes.
o Since all the hidden nodes have, to some degree, contributed to the errors evident in the output
layer, the output error signals are transmitted backwards from the output layer to each node in
the hidden layer that immediately contributed to the output layer.
o This process is then repeated, layer by layer, until each node in the network has received an
error signal that describes its relative contribution to the overall error.
o Once the error signal for each node has been determined, the errors are then used by the nodes
to update the values for each connection weights until the network converges to a state that
allows all the training patterns to be encoded.
o The Backpropagation algorithm looks for the minimum value of the error function in weight
space using a technique called the delta rule or gradient descent. The weights that minimize
the error function is then considered to be a solution to the learning problem.
Back-propagation: Loop until convergence:

CS1701A Machine Learning Department of CSE- SAEC Unit II 31

How Does it Work?
1. Forward Pass:
○ Input data is fed into the neural network.
○ Information propagates through the layers, with each neuron performing
calculations based on its inputs and weights.
○ The output layer produces a prediction.
○ The error between the predicted output and the actual target is calculated.
2. Backward Pass:
○ The error is propagated backward through the network, layer by layer.
○ The chain rule from calculus is applied to compute the gradient of the error with
respect to each weight and bias.
○ These gradients indicate how much each weight and bias contributes to the overall
error.
○ The weights and biases are updated using an optimization algorithm like gradient
descent to minimize the error.
3. Iterative Refinement:
○ Steps 1 and 2 are repeated multiple times for different training examples (epochs).
○ With each iteration, the network's parameters are adjusted, improving its ability to
make accurate predictions.

Key Components
● Error Function: Measures the difference between the predicted output and the actual
target.
● Gradient Descent: Optimization algorithm used to update weights and biases.
● Chain Rule: Used to calculate gradients efficiently.
● Activation Functions: Introduce non-linearity into the network.

Activation Functions
Activation functions are an integral building block of neural networks that enable them to learn
complex patterns in data. They transform the input signal of a node in a neural network into an
output signal that is then passed on to the next layer . Types are
 Sigmoid
 Tanh
 ReLU (Rectified Linear Unit)

CS1701A Machine Learning Department of CSE- SAEC Unit II 32

Algorithm

Example

CS1701A Machine Learning Department of CSE- SAEC Unit II 33

Gradient Descent and the Delta Rule

Gradient descent is the most popular optimization technique for feedforward neural networks. The
term “gradient” refers to the quantity change of output obtained from a neural network when the
inputs change a little. Technically, it measures the updated weights concerning the change in error.
The gradient can also be defined as the slope of a function. The higher the angle, the steeper the
slope and the faster a model can learn.
Although the perceptron rule finds a successful weight vector when the training examples are
linearly separable, it can fail to converge if the examples are not linearly separable. A second
training rule, called the delta rule, is designed to overcome this difficulty. If the training examples
are not linearly separable, the delta rule converges toward a best-fit approximation to the target
concept.
The key idea behind the delta rule is to use gradient descent to search the hypothesis space of
possible weight vectors to find the weights that best fit the training examples. This rule is important
because gradient descent provides the basis for the BACKPROPAGATION algorithm, which can
learn networks with many interconnected units. It is also important because gradient descent can
serve as the basis for learning algorithms that must search through hypothesis spaces containing
many different types of continuously parameterized hypotheses

• The perceptron rule finds a successful weight vector when the training examples are linearly
separable, it can fail to converge if the examples are not linearly separable.
• The delta rule overcomes this difficulty.
• If the training examples are not linearly separable, the delta rule converges toward a best-fit
approximation to the target concept.
• The key idea behind the delta rule is to use gradient descent to search the hypothesis space
of possible weight vectors to find the weights that best fit the training examples.
• The delta rule is important because gradient descent provides the basis for the
BACKPROPAGATION Algorithm, which can learn networks with many interconnected
units.
• The gradient descent can serve as the basis for learning algorithms that must search through
hypothesis spaces containing many different types of continuously parameterized hypotheses

CS1701A Machine Learning Department of CSE- SAEC Unit II 34

Comparison of Perceptron and Gradient Descent Rule
Perceptron learning rule guaranteed to succeed if
• Training examples are linearly separable

• Sufficiently small learning rate ⎜

Linear unit training rules uses gradient descent

• Guaranteed to converge to hypothesis with minimum squared error

• Given sufficiently small learning rate ⎜

• Even when training data contains noise

• Even when training data not separable by H

CS1701A Machine Learning Department of CSE- SAEC Unit II 35

Gradient descent vs Backpropagation
o Gradient descent is the process of using gradients to find the minimum value of the cost
function, while backpropagation is calculating those gradients by moving in a backward
direction in the neural network
o Backpropagation refers only to the method for computing the gradient, while gradient
descent, is used to perform learning using this gradient.
Backpropagation, short for “backward propagation of errors”, is a mechanism used to update the
weights using gradient descent. It calculates the gradient of the error function with respect to the
neural network’s weights. The calculation proceeds backwards through the network.
Gradient descent is an iterative optimization algorithm for finding the minimum of a functionTo
find a local minimum of a function using gradient descent, one takes steps proportional to the
negative of the gradient of the function at the current point.

2.9 NAÏVE BAYES ALGORITHM

● The Naïve Bayes algorithm is based on the idea of Conditional Probability.

● Conditional probability is based on finding the probability that something will happen,
given that something else has already happened.

● The task of the algorithm then is to look at the evidence and to determine the likelihood of a
specific class and assign a label accordingly to each entity.

● Bayes theorem provides a way to calculate the probability of a hypothesis based on its prior
probability, the probabilities of observing various data given the hypothesis, and the observed
data itself.
Naïve Bayes is an example of a probabilistic classifier. We can do this using the Bayes rule
defined as

Bayes theorem provides a way to calculate the posterior probability P(A|B), from the prior
probability P(A), together with P(B) and P(B|A).

1. The importance of the result is that it helps us to “invert” conditional probabilities, that is, to
express the conditional probability P(A|B) in terms of the conditional probability P(B|A).
2. The following terminology is used in this context:

● A is called the proposition and B is called the evidence.

● P(A) is called the prior probability of proposition and P(B) is called the prior probability
CS1701A Machine Learning Department of CSE- SAEC Unit II 36
 of evidence.

● P(A|B) is called the posterior probability of A given B.

● P(B|A) is called the likelihood of B given A.

Naive Bayes algorithm

Assumption
The naive Bayes algorithm is based on the following assumptions:
• All the features are independent and are unrelated to each other. Presence or absence of a
feature does not influence the presence or absence of any other feature.
• The data has class-conditional independence, which means that events are independent so long
as they are conditioned on the same class value.
These assumptions are, in general, true in many real world problems. It is because of these
assumptions, the algorithm is called a naive algorithm.

CS1701A Machine Learning Department of CSE- SAEC Unit II 37

CS1701A Machine Learning Department of CSE- SAEC Unit II 38
 2.10 LINEAR REGRESSION AND LOGISTIC REGRESSION

1. Linear Regression:

● Linear regression is a statistical regression method which is used for predictive analysis.

● It is one of the very simple and easy algorithms which works on regression and shows the
relationship between the continuous variables.

● It is used for solving the regression problem in machine learning.

● Linear regression shows the linear relationship between the independent variable (X-axis)
and the dependent variable (Y-axis), hence called linear regression.
Types of Linear Regression
Linear regression can be further divided into two types of the algorithm:

● Simple Linear Regression:

If a single independent variable is used to predict the value of a numerical dependent variable, then
such a Linear Regression algorithm is called Simple Linear Regression.

CS1701A Machine Learning Department of CSE- SAEC Unit II 39

● Multiple Linear regression:
If more than one independent variable is used to predict the value of a numerical dependent
variable, then such a Linear Regression algorithm is called Multiple Linear Regression.
Linear Regression Line:
A linear line showing the relationship between the dependent and independent variables is called
a regression line. A regression line can show two types of relationship:

✔ Positive Linear Relationship:

If the dependent variable increases on the Y-axis and independent variable increases on X-
axis, then such a relationship is termed as a Positive linear relationship.

✔ Negative Linear Relationship:

If the dependent variable decreases on the Y-axis and independent variable increases on the X-
axis, then such a relationship is called a negative linear relationship.
The relationship between variables in the linear regression model can be explained using the below
image.
Here we are predicting the salary of an employee on the basis of the year of experience.

Below is the mathematical equation for Simple Linear regression:

Y= aX+b
Here, Y = dependent variables (target variables),
X= Independent variables (predictor variables),
a and b are the linear coefficients

CS1701A Machine Learning Department of CSE- SAEC Unit II 40

Some popular applications of linear regression are:

● Analyzing trends and sales estimates

● Salary forecasting

● Real estate prediction

● Arriving at ETAs in traffic.

Example: Find the linear regression equation for the given data:

X Y
3 8
9 6
5 4

3 2

Solution: Calculating intercept and slope value.

x y x2 xy
3 8 9 24

9 6 81 54

5 4 25 20

3 2 9 6

∑x = 20 ∑y = 20 ∑x2 = 124 ∑xy = 104

CS1701A Machine Learning Department of CSE- SAEC Unit II 41

Finding the best fit line:
When working with linear regression, our main goal is to find the best fit line that means the error
between predicted values and actual values should be minimized. The best fit line will have the
least error.
The different values for weights or the coefficient of lines (a0, a1) gives a different line of
regression, so we need to calculate the best values for a0 and a1 to find the best fit line, so to
calculate this we use cost function.
Cost function

● The different values for weights or coefficient of lines (a0, a1) gives the different line of
regression, and the cost function is used to estimate the values of the coefficient for the best
fit line.

● Cost function optimizes the regression coefficients or weights. It measures how a linear
regression model is performing.

● We can use the cost function to find the accuracy of the mapping function, which maps the
input variable to the output variable. This mapping function is also known as Hypothesis
function.
For Linear Regression, we use the Mean Squared Error (MSE) cost function, which is the
average of squared error occurred between the predicted values and actual values. It can be written
as:

CS1701A Machine Learning Department of CSE- SAEC Unit II 42

For the above linear equation, MSE can be calculated as:
Where,
N=Total number of observation
Yi = Actual value
(a1xi+a0)= Predicted value.
Residuals: The distance between the actual value and predicted values is called residual. If the
observed points are far from the regression line, then the residual will be high, and so cost function
will high. If the scatter points are lose to the regression line, then the residual will be small and
hence the cost function

2. Logistic Regression:

● Logistic regression is another supervised learning algorithm which is used to solve the
classification problems. In classification problems, we have dependent variables in a binary
or discrete format such as 0 or 1.

● Logistic regression algorithm works with the categorical variable such as 0 or 1, Yes or No,
True or False, Spam or not spam, etc.

● It is a predictive analysis algorithm which works on the concept of probability.

● Logistic regression is a type of regression, but it is different from the linear regression
algorithm in the term how they are used.

● Logistic regression uses sigmoid function or logistic function which is a complex cost
function. This sigmoid function is used to model the data in logistic regression. The function
can be represented as:

✔ f(x)= Output between the 0 and 1 value.

✔ x= input to the function

✔ e= base of natural logarithm.

When we provide the input values (data) to the function, it gives the S-curve as follows:

CS1701A Machine Learning Department of CSE- SAEC Unit II 43

● It uses the concept of threshold levels, values above the threshold level are rounded up to 1,
and values below the threshold level are rounded up to 0.
There are three types of logistic regression:

● Binary(0/1, pass/fail)

● Multi(cats, dogs, lions)

● Ordinal(low, medium, high)

2.11 RANDOM FOREST

● A random forest is an ensemble learning method where multiple decision trees are constructed
and then they are merged to get a more accurate prediction.

● Random forest is identified as a collection of decision trees. Each tree estimates a

classification, and this is called a “vote”.

● Ideally, we consider each vote from every tree and chose the most voted classification
(Majority-Voting).

● Random Forest follow the same bagging process as the decision trees but each time a split is
to be performed, the search for the split variable is limited to a random subset of m of the p
attributes (variables or features) aka Split-Attribute Randomization :

✔ classification trees: m = √p

✔ regression trees: m = p/3

m is commonly referred to as m try
Random Forests produce many unique trees.
CS1701A Machine Learning Department of CSE- SAEC Unit II 44
 Example: Suppose there is a dataset that contains multiple fruit images. So, this dataset is
given to the Random forest classifier. The dataset is divided into subsets and given to each decision
tree. During the training phase, each decision tree produces a prediction result, and when a new
data point occurs, then based on the majority of results, the Random Forest classifier predicts the
final decision. Consider the below image:

The following steps explain the working Random Forest Algorithm:

Step 1: Select random K data points from the training set.
Step 2:Build the decision trees associated with the selected data points(Subsets).
Step 3:Choose the number N for decision trees that we want to build.
Step 4:Repeat Step 1 and 2.
Step 5: For new data points, find the predictions of each decision tree, and assign the new
data points to the category that wins the majority votes.
This combination of multiple models is called Ensemble. Ensemble uses two methods:
[Link]: Creating a different training subset from sample training data with replacement is
called Bagging. The final output is based on majority voting.

CS1701A Machine Learning Department of CSE- SAEC Unit II 45

[Link]: Combing weak learners into strong learners by creating sequential models such that
the final model has the highest accuracy is called Boosting. Example: ADA BOOST, XG BOOST.

Algorithm
Here is an outline of the random forest algorithm.
1. The random forests algorithm generates many classification trees. Each tree is generated
as follows:

CS1701A Machine Learning Department of CSE- SAEC Unit II 46

 a. If the number of examples in the training set is N, take a sample of N examples
at random - but with replacement, from the original data. This sample will be the
training set for generating the tree.
b. If there are M input variables, a number m is specified such that at each node, m
variables are selected at random out of the M and the best split on these m is used
to split the node. The value of m is held constant during the generation of the
various trees in the forest.
c. Each tree is grown to the largest extent possible.
2. To classify a new object from an input vector, put the input vector down each of the trees
in the forest. Each tree gives a classification, and we say the tree “votes” for that class.
The forest chooses the classification

Strengths and weaknesses

Strengths
The following are some of the important strengths of random forests.
• It runs efficiently on large data bases.
• It can handle thousands of input variables without variable deletion.
• Random forest run times are quite fast, and they are able to deal with unbalanced and missing
data.
• They can handle binary features, categorical features, numerical features without any need for
scaling.
Weaknesses
• A weakness of random forest algorithms is that when used for regression they cannot predict
beyond the range in the training data, and that they may over-fit data sets that are particularly
noisy.
• The sizes of the models created by random forests may be very large. It may take hundreds of
megabytes of memory and may be slow to evaluate.
• Random forest models are black boxes that are very hard to interpret

2.12 ENSEMBLE LEARNING

• Ensemble Learning is a technique that create multiple models and then combine them to
produce improved results.
• Ensemble learning usually produces more accurate solutions than a single model would.
• Ensemble learning methods is applied to regression as well as classification.

✔ Ensemble learning for regression creates multiple repressors i.e. multiple regression
models such as linear, polynomial, etc.

CS1701A Machine Learning Department of CSE- SAEC Unit II 47

 ✔ Ensemble learning for classification creates multiple classifiers i.e. multiple classification
models such as logistic, decision tress, KNN, SVM, etc.
By suitably combining multiple learners the accuracy can be improved (but it need not to).
o How to generate base learners that complement each other?
o How to combine their outputs for maximum accuracy?
• Example: train an ensemble of L decision trees on L different subsets of the training set and
define the ensemble output for a test instance as the majority vote (for classification) or the
average (for regression) of the L trees.
• Ensembles of decision trees (random forest, boosted decision trees) are practically among the
most accurate models in machine learning.

Need for Ensemble Learning

No free lunch theorem states that there is no single learning algorithm that in any domain always
induces the most accurate learner.
By suitably combining multiple base learners accuracy can be improved.
How to generate diversified learners?
• Different learning algorithms/models
• Different hyper parameters
• Different optimization algorithm or initialization:
• Different input representations / features
• Different training set

CS1701A Machine Learning Department of CSE- SAEC Unit II 48

Bias and Variance
 Bias of a model is the error in predicting the training dataset, while variance is the change in
model with respect to change in training dataset.
 A model with high bias tends to underfit the training data, i.e. the model is too simple to capture
the underlying patterns in the data.
 In contrast, a model with high variance is prone to memorize the noise in the training data
rather than the underlying patterns. In other words, it overfit the training data

Types of Ensemble Models

The most common types are

1. Bagging (Bootstrap Aggregation)

In bagging, multiple models are trained on different subsets of the training data, which reduces the
variance by averaging the predictions of the individual models.
2. Boosting
Boosting, on the other hand, focuses on reducing bias by sequentially training models on
misclassified instances.
3. Stacking
Stacking combines multiple models by training a meta-model on their predictions. The meta-model
learns to combine the strengths of the individual models and generate a final prediction.

1. Bagging (Bootstrap Aggregation)

Basic idea: Given a dataset D with N examples.

– Randomly draw (with replacement) B datasets of size N from D.
– Estimate the measure of interest on each of the B datasets.
– Take the mean of the estimates.

● Bagging: generate different bootstrapped training data sets

✔ bootstrap: sampling with replacement

✔ each bagged tree makes use of around 2/3 of the observations

✔ remaining 1/3 of the observations are referred to as the out-of-bag (OOB) observations
• Each individual tree has high variance, but low bias, averaging these trees reduces the
variance
• Reduce overfitting; reduce bias; break the bias-variance trade-off
CS1701A Machine Learning Department of CSE- SAEC Unit II 49
Vanilla Bagging
Vanilla Bagging is widely used in real-world applications including credit risk assessment, fraud
detection, stock price prediction etc.
Algorithm
• In Vanilla Bagging, we use bootstrap method (randomly choosing an element with replacement)
to make subset datasets of equal length as the initial dataset.
• Even though the number of elements in subsets are equal to the number of elements in the
original dataset, they are not the same dataset as the draw has been done with replacements.
• The number of subsets, N, is a hyperparameter.

2. Boosting
 Boosting is an ensemble learning method that combines a set of weak learners into a strong
learner to minimize training errors.
 In boosting, a random sample of data is chosen, fitted with a model, and then sequentially
trained. In other words, each model aims to make up for the shortcomings of the one before
it.
 Intuitively, we do not have a super learner, but many bad learners. These bad learners are
combined to obtain a strong learner with lower bias.
 So, Boosting is used to shift the models from high bias −→ low bias

There are three popular types of boosting:

a) AdaBoost, b) Gradient Boosting and c)XGBoost.
CS1701A Machine Learning Department of CSE- SAEC Unit II 50
 GRADIENT BOOSTING
 Boosting algorithms can differ in how they create and aggregate weak learners during the
sequential process.
 It works by sequentially adding predictors to an ensemble, each correcting for its
predecessor’s errors. Each predictor is trained on theresidual errors of the previous
predictor.
 When the target column is continuous, we use Gradient Boosting Regressor; when it is a
 classification problem, we use Gradient Boosting Classifier. The difference between the
two is
 the Loss function used.

Steps in Boosting:
– Use the training set to train a simple predictor.
– Re-weight the training examples, putting more weight on examples that were not properly
classified in the previous predictor.
– Repeat n times.
– Combine the simple hypotheses into a single, accurate predictor

Bagging Vs Boosting

CS1701A Machine Learning Department of CSE- SAEC Unit II 51

 3. Stacking

Disadvantages of Ensemble Learning

– An ensemble of learners is computationally more costly in time and space than a single learner,
both at training and test time.

CS1701A Machine Learning Department of CSE- SAEC Unit II 52