Module 3: Supervised Learning- II

Contents

Logistic Regression
KNN Classifier
Single layer Perceptron
Multi layer Perceptron
Support Vector Machines
Error Bound-Performance Metrics
Logistic Regression

 Name is somewhat misleading. Really a technique for classification, not

regression.
– “Regression” comes from fact that we fit a
linear model to the feature space.

 Involves a more probabilistic view of classification.

 In Logistic regression, instead of fitting a regression line, we fit an "S"

shaped logistic function, which predicts two maximum values (0 or 1).

 The independent variable should not have multi-collinearity.

Logistic Regression
Logistic regression is for predicting the categorical dependent variable
using a given set of independent variables.

Dependent Variable(y):
The response categorical variable holding
values like 0 or 1, yes or no etc.

Independent Variable(x):
The predictor variable used to predict the
response variable.
Math behind Logistic Regression

Odds are useful to quantify the probability of an event.

Math behind Logistic Regression
In equation(1), if the values of probability (y) changes from 0 to 1, then
values of odd range from 0 to +inifinity.

But we need range between -[infinity] to +[infinity], then take logarithm of

the equation (1) it will become,
Finding Sigmoid Function: Step by step
Logistic Regression: Example

To predict if a student will be admitted to a degree

program based on his CGPA.

Admissi CGPA
on
0 4.2
0 5.1
0 5.5
1 8.2
1 9.0
1 9.1
Estimated Regression Equation
Example:
Consider the following training examples:
Marks scored: X = [81 42 61 59 78 49]
Grade (Pass/Fail): Y = [Pass Fail Pass Fail Pass Fail]
Assume we want to model the probability of Y of the form
which is parameterized by (β0, β1).
i) Which of the following parameters would you use to model p(x).
(a) (-119, 2) (b) (-120, 2) (c) (-121, 2)
(ii) With the chosen parameters, what should be the minimum mark to ensure the
student gets a ‘Pass’ grade with 95% probability?
Sample Calculation for p(x) for
 Among three, the maximum likelihood value is for β0 = -
120 , β1 = 2.
 Therefore, we have to use these values to model p(x)
ii) With the chosen parameters, what should be the minimum
mark to ensure the student gets a ‘Pass’ grade with 95%
probability?
Substituting p(x) = 0.95, 0 = -120 and 1 = 2, we will get
K Nearest Neighbor
How does KNN algorithm work”
How does KNN algorithm work”
How does KNN algorithm work”
At k=3, we classify the new variable as square
At k=7, we classify the new variable as triangle
KNN Distance Metrics
Performance Evaluation
Example: To classify the given image is dog or not

Understanding TP, TN, FP and FN

Performance Evaluation

Predicted Predicted
No Yes

Actual No 1-TN 3-FP

Dog
Actual 2-FN 4-TP
Dog
Performance Evaluation

1. Confusion Matrix
Confusion Matrix is a performance measurement for the machine learning classification
problems where the output can be two or more classes. It is a table with combinations of
predicted and actual values.
Performance Evaluation

Confusion Matrix

Assume for the given problem out of 150 samples the classifier predicted
“Yes” 100 times and “No” 50 times.
Performance Evaluation

Confusion Matrix

•True Positive: We predicted positive and it’s true.

•True Negative: We predicted negative and it’s true.

•False Positive (Type 1 Error): We predicted positive and it’s false.

•False Negative (Type 2 Error): We predicted negative and it’s false.

Performance Evaluation
2. Accuracy

It simply measures how often the classifier

correctly predicts.

It is the ratio of correct predictions to the

total number of input Samples.
Performance Evaluation

3. Misclassification Rate:
Performance Evaluation
4. True Negative Rate/ Specificity

It explains how many of the actual negative cases

we were able to predict correctly with our model.
Performance Evaluation
5. Precision
Performance Evaluation
6. True Positive Rate/ Sensitivity/
Recall:

It explains how many of the actual positive

cases we were able to predict correctly with
our model.

Recall for a label is defined as the number of true positives divided

by the total number of actual positives.

=95%
Performance Evaluation
7. Prevalence
Performance Evaluation
8. F1 Score

F1 Score is the harmonic mean of precision

and recall.
Performance Evaluation
9. AUC-ROC Curve

AUC-ROC stands for the Area Under the Receiver Operating Characteristic Curve
The ROC curve is drawn by calculating the true positive rate (TPR) and false positive rate
(FPR) at every possible threshold (in practice, at selected intervals), then graphing TPR
over FPR. A perfect model, which at some threshold has a TPR of 1.0 and a FPR of 0.0,
can be represented by either a point at (0, 1) if all other thresholds are ignored, or by the
following:
Performance Evaluation
9. AUC-ROC Curve

The Area Under the Curve (AUC) is the measure of the ability of a classifier to
distinguish between classes and is used as a summary of the ROC curve.

The higher the AUC, the better the performance of the model at distinguishing
between the positive and negative classes.

When AUC = 1, then the classifier is able to perfectly distinguish between all
the Positive and the Negative class points correctly.

If, however, the AUC had been 0, then the classifier would be predicting all
Negatives as Positives, and all Positives as Negatives.
Performance Evaluation
9. AUC-ROC Curve

When 0.5<AUC<1, there is a high chance that the classifier will be able to
distinguish the positive class values from the negative class values.

This is so because the classifier is able to detect more numbers of True

positives and True negatives than False negatives and False positives.

When AUC=0.5, then the classifier is not able to distinguish between Positive
and Negative class points. Meaning either the classifier is predicting random
class or constant class for all the data points.
Performance Evaluation
9. AUC-ROC Curve

A typical AUC-ROC curve looks like:

 The Perceptron
By Resenblatt in 1959
Initial proposal of connectionist networks
Computational model of the retina of the eye

Processing Elements in NN
An ANN consists of perceptrons. Each of the perceptrons receives inputs,
processes inputs and delivers a single output.
 Perceptron

Summation Unit Thresholding Unit

x0=1
x1 w1
w0
w2
x2  o
. n
I= wi xi
. 1 if I> θ
. wn i=0
O=f(I)=
0 otherwise

xn
xi = input
wi =weight
O= Output

47
The Perceptron: Threshold Activation Function
• Threshold activation function

Step Threshold

1 if I>θ
O=f(I)=
0 otherwise
Nonlinear Activation Functions
• Sigmoid Neuron unit function

1
O f(I) 
1  e  I
Sigmoid
Perceptron and Linearly Separable
Problem
Perceptron can handle task which are linearly separable

Example: If the two different sets in 2D are linearly separable

Class1 Class2
Single layer Perceptron

Output Unit Response

Single layer Perceptron Cntd…
Weight and Bias update
Perceptron for AND function with
bipolar inputs and targets
Solution

Initialize w1=w2=b=0, α=1, θ=0

For input pair (1,1), x1=1 and x2=1

Perceptron for AND function with bipolar
inputs and targets
Perceptron for AND function with bipolar
inputs and targets
Multilayer Neural Network
 Multilayer Neural Network
-To solve XoR problem

Generate output for ANDNOT and NOTAND and combine the results using OR function

Threshold is 1
Combine the solution of ANDNOT and NOTAND using OR
Multi layer feed forward network
Support Vector Machine
SVM
 Support Vector Machine” (SVM) is a supervised machine learning
algorithm that can be used for both classification or regression
challenges. However, it is mostly used in classification problems. In
the SVM algorithm, we plot each data item as a point in n-
dimensional space (where n is a number of features) with the value
of each feature being the value of a particular coordinate. Then, we
perform classification by finding the hyper-plane that differentiates
the two classes very well.
SVM
 It is a supervised machine learning problem where we try to find a
hyperplane that best separates the two classes.
 Support Vectors are simply the coordinates of individual observation. The
SVM classifier is a frontier that best segregates the two classes (hyper-
plane/ line).
Types of SVM

•Linear SVM: Linear SVM is used for linearly separable data,

which means if a dataset can be classified into two classes by
using a single straight line, then such data is termed as linearly
separable data, and classifier is used called as Linear SVM
classifier.

•Non-linear SVM: Non-Linear SVM is used for non-linearly

separated data, which means if a dataset cannot be classified by
using a straight line, then such data is termed as non-linear data
and classifier used is called as Non-linear SVM classifier.
Hyperplane and Support Vectors in the SVM algorithm:
Hyperplane: There can be multiple lines/decision boundaries to
segregate the classes in n-dimensional space, but we need to find
out the best decision boundary that helps to classify the data
points. This best boundary is known as the hyperplane of SVM. The
dimension of the hyperplane depends upon the number of features.
If the number of input features is 2, then the hyperplane is just a
line. If the number of input features is 3, then the hyperplane
becomes a two-dimensional plane. It becomes difficult to imagine
when the number of features exceeds 3.
Hyperplane and Support Vectors in the SVM algorithm:

Support Vectors:
The data points or vectors that are the closest to the hyperplane
and which affect the position of the hyperplane are termed as
Support Vector. Since these vectors support the hyperplane,
hence called a Support vector.
Selecting the best hyper-
plane:
The distance between the support vectors and the hyperplane is
called as margin. And the goal of SVM is to maximize this
margin. The hyperplane with maximum margin is called
the optimal hyperplane.
How SVM works in different
scenario?
Identify the right hyper-plane
(Scenario-1)
Here, we have three hyper-planes (A, B, and C). Now, identify the right
hyper-plane to classify stars and circles.

“Select the hyper-plane which segregates the two classes better.” In

this scenario, hyper-plane “B” has excellently performed this job.
How SVM works in different
scenario?
Identify the right hyper-plane
(Scenario-2)
we have three hyper-planes (A, B, and C), and all segregate the classes well.
Now, How can we identify the right hyper-plane?

Maximizing the distances between the nearest data point (either

class) and the hyper-plane will help us to decide the right hyper-
plane. This distance is called a Margin. We name the right hyper-
plane as C
How SVM works in different
scenario?
Identify the right hyper-plane
(Scenario-3)
Use the rules as discussed in the previous section to identify the right hyper-
plane.

You may have selected hyper-plane B as it has a higher margin

compared to A. But, here is the catch, SVM selects the hyper-plane
which classifies the classes accurately prior to maximizing the margin.
Here, hyper-plane B has a classification error, and A has classified all
correctly. Therefore, the right hyper-plane is A.
How SVM works in different
scenario?
Can we classify two
classes (Scenario-4)?

One star at the other end is like an outlier for the star class.
The SVM algorithm has a feature to ignore outliers and find
the hyper-plane that has the maximum margin. Hence, we
can say SVM classification is robust to outliers.
How SVM works in different
scenario?
Find the hyper-plane to segregate to classes (Scenario-5)

In the scenario below, we can’t have a linear hyper-plane

between the two classes, so how does SVM classify these two
classes? This could be possible by Non-linear SVM.
How SVM works in different
scenario?
Find the hyper-plane to segregate to classes (Scenario-5)

SVM can solve this problem. It solves this problem by

introducing additional features. Here, we will add a new feature,
z=x2+y2
By adding the third dimension, the sample space will
become as below image:
How SVM works in different
scenario?
Find the hyper-plane to segregate to classes (Scenario-5)

So now, SVM will divide the datasets into classes in the following
way.
How SVM works in different
scenario?
Find the hyper-plane to segregate to classes (Scenario-5)

How to add this feature manually to have a hyper-plane. This

process in SVM has a technique called the kernel trick.

The SVM kernel is a function that takes low dimensional

input space and transforms it to a higher dimensional space,
i.e., it converts not separable problem to a separable
problem. It is mostly useful in non-linear data separation
problems.
Different Kernel functions
Linear SVM Problem

x1 X2 Output

1 1 1
1 -1 1
2 1 1
2 -1 1
4 0 -1
5 1 -1
5 -1 -1
6 0 -1
Linear SVM Problem
Linear SVM Problem
Linear SVM Problem
Linear SVM Problem
Linear SVM Problem
Note: If w=(1,0) then draw a vertical line as hyperplane
If w=(0,1) then draw a horizontal line as hyperplane
If w=(1,1) then draw a diagonal line(45 degree) connecting x and y
axes as hyperplane
Non-linear SVM Problem

Example 1: For the given problem there is no clear separating hyperplane

between the red class and blue.
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem

Note: Consider this as a Linear problem and apply linear SVM

Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem
Non-linear SVM Problem

Normalizing the weights and the bias

Non-linear SVM Problem
 Non-linear SVM Problem
Let us consider an unknown point (-1,2). Check can we classify this point with
the above non-linear classified model
 Non-linear SVM Problem
We could transform the point (-1,2 to (16,13) in the below graph and could be
separable by the same hyperplane that we identified.
Non-linear SVM Problem
Advantages of SVM
•Support vector machine works comparably well when there is an understandable margin of dissociation between classes.
•It is more productive in high-dimensional spaces.
•It is effective in instances where the number of dimensions is larger than the number of specimens.
•Support vector machine is comparably memory systematic. Support Vector Machine (SVM) is a powerful supervised
machine learning algorithm with several advantages. Some of the main advantages of SVM include:
•Handling high-dimensional data: SVMs are effective in handling high-dimensional data, which is common in many
applications such as image and text classification.
•Handling small datasets: SVMs can perform well with small datasets, as they only require a small number of support
vectors to define the boundary.
•Modeling non-linear decision boundaries: SVMs can model non-linear decision boundaries by using the kernel trick, which
maps the data into a higher-dimensional space where the data becomes linearly separable.
•Robustness to noise: SVMs are robust to noise in the data, as the decision boundary is determined by the support vectors,
which are the closest data points to the boundary.
•Generalization: SVMs have good generalization performance, which means that they are able to classify new, unseen data
well.
•Versatility: SVMs can be used for both classification and regression tasks, and it can be applied to a wide range of
applications such as natural language processing, computer vision and bioinformatics.
•Sparse solution: SVMs have sparse solutions, which means that they only use a subset of the training data to make
predictions. This makes the algorithm more efficient and less prone to overfitting.
•Regularization: SVMs can be regularized, which means that the algorithm can be modified to avoid overfitting.
Disadvantages of support vector machine:

•Computationally expensive: SVMs can be computationally expensive for large

datasets, as the algorithm requires solving a quadratic optimization problem.
•Choice of kernel: The choice of kernel can greatly affect the performance of an
SVM, and it can be difficult to determine the best kernel for a given dataset.
•Sensitivity to the choice of parameters: SVMs can be sensitive to the choice of
parameters, such as the regularization parameter, and it can be difficult to
determine the optimal parameter values for a given dataset.
•Memory-intensive: SVMs can be memory-intensive, as the algorithm requires
storing the kernel matrix, which can be large for large datasets.
•Limited to two-class problems: SVMs are primarily used for two-class problems,
although multi-class problems can be solved by using one-versus-one or one-
versus-all strategies.
•Lack of probabilistic interpretation: SVMs do not provide a probabilistic
interpretation of the decision boundary, which can be a disadvantage in some
applications.
•Not suitable for large datasets with many features: SVMs can be very slow and
can consume a lot of memory when the dataset has many features.
•Not suitable for datasets with missing values: SVMs requires complete datasets,
with no missing values, it can not handle missing values.