1
CLASSIFICATION
2
DEFINITION
• Categorized output can have the form such as
• Yes/No
• Spam/ No spam
• Black/White
3
CLASSIFICATION
• Process of predicting class or category from observed values or given data points
• Mathematically,
• The task of approximating a mapping function (f) from input variables (X) to output
variables (Y).
• Basically belongs to the supervised machine learning in which targets are also
provided along with the input data set.
• Example
• Spam detection in emails. There can be only two categories of output, “spam” and
“no spam”; hence this is a binary type classification.
• To implement this classification, we first need to train the classifier
• “spam” and “no spam” emails would be used as the training data
• After successful training of the classifier, it can be used to classify an unknown email
as spam or no spam
4
TYPES OF LEARNERS IN
CLASSIFICATION
1) Lazy Learners
Waits for the testing data to be appeared after storing the training data.
Classification is done only after getting the testing data
Spend less time on training but more time on predicting
Examples: K-nearest neighbor and case-based reasoning
2) Eager Learners
Opposite to lazy learners
Construct classification model without waiting for the testing data to be
appeared after storing the training data
Spend more time on training but less time on predicting
Examples: Decision Trees, Naïve Bayes and Artificial Neural Networks (ANN)
5
VARIOUS ML CLASSIFICATION
ALGORITHMS
Some important Machine Learning classification algorithms are:
• Logistic Regression
• Support Vector Machine (SVM)
• Decision Tree
• Naïve Bayes
• Random Forest
Applications of Classification Algorithm
• Speech Recognition
• Handwriting Recognition
• Biometric Identification
• Document Classification
6
K-NEAREST NEIGHBORS (KNN)
ALGORITHM
• Supervised Machine Learning Algorithm
• Can be used for both classification as well as regression predictive problems
• Mainly used for classification predictive problems in industry
• Properties of KNN
• Lazy learning algorithm −
• Does not have a specialized training phase and uses all the data for training while
classification
• Non-parametric learning algorithm − it doesn’t assume anything about the
underlying data.
7
WORKING OF KNN ALGORITHM
• uses ‘feature similarity’ to predict the values of new datapoints
• the new data point will be assigned a value based on how closely it
matches the points in the training set
8
Suppose we have a dataset which can be plotted as EXAMPLE
follows −
Now, we need to classify new data point with black dot (at
point 60,60) into blue or red class. We are assuming K = 3
i.e. it would find three nearest data points. It is shown in
the next diagram −
How KNN algorithm works 9
Suppose we have height, weight and T-shirt size of
Height (in cms) Weight (in kgs) T Shirt Size
some customers and we need to predict the T-shirt size 158 58 M
of a new customer given only height and weight 158 59 M
information we have. Data including height, weight and
158 63 M
T-shirt size information is shown below -
160 59 M
160 60 M
163 60 M
163 61 M
Step 1 : Calculate Similarity based on distance function 160 64 L
163 64 L
There are many distance functions but Euclidean is the 165 61 L
most commonly used measure. It is mainly used when 165 62 L
data is continuous. Manhattan distance is also very 165 65 L
common for continuous variables. 168 62 L
168 63 L
168 66 L
170 63 L
170 64 L
170 68 L
10
KNN
• The idea to use distance measure is to find the distance (similarity)
between new sample and training cases and then finds the k-closest
customers to new customer in terms of height and weight.
We have to identify the size of customer having height 161cm and weight 61kg.
Euclidean distance between first observation and new observation is as follows -
=SQRT((161-158)^2+(61-58)^2)
Similarly, we will calculate distance of all the training cases with new case and calculates the rank in terms of
distance. The smallest distance value will be ranked 1 and considered as nearest neighbor.
11
• Step 2 : Find K-Nearest Neighbors
Let k be 5. Then the algorithm
searches for the 5 customers closest
to New customer i.e. most similar to
new customer in terms of attributes,
and see what categories those 5
customers were in. If 4 of them had
‘Medium T shirt sizes’ and 1 had
‘Large T shirt size’ then your best
guess for new customer is ‘Medium
T shirt.
12
KNN VISUAL REPRESENTATION
• In the graph below, binary dependent
variable (T-shirt size) is displayed in blue
and orange color. 'Medium T-shirt size' is
in blue color and 'Large T-shirt size' in
orange color. New customer
information is exhibited in yellow circle.
Four blue highlighted data points and
one orange highlighted data point are
close to yellow circle. so the prediction
for the new case is blue highlighted
data point which is Medium T-shirt size.
13
1. Standardization
ASSUMPTIONS OF KNN
When independent variables in training data are
measured in different units, it is important to
standardize variables before calculating distance
Example: if one variable is based on height in cms,
and the other is based on weight in kgs then
height will influence more on the distance
calculation.
In order to make them comparable we need to
standardize them which can be done by any of
the given methods :
14
KNN AFTER STANDARDIZATION
• After standardization, 5th closest value got changed
as height was dominating earlier before
standardization. Hence, it is important to standardize
predictors before running K-nearest neighbor
algorithm.
15
ASSUMPTIONS OF KNN
• 2. Outlier
Low k-value is sensitive to outliers and a higher K-value is more resilient to
outliers as it considers more voters to decide prediction.
16
ADVANTAGES AND
DISADVANTAGES OF KNN
Advantages
• It is very simple algorithm to understand and interpret.
• It is very useful for nonlinear data because there is no assumption about data in this
algorithm.
• It is a versatile algorithm as we can use it for classification as well as regression.
• It has relatively high accuracy but there are much better supervised learning models
than KNN.
Disadvantages
• It is computationally a bit expensive algorithm because it stores all the training data.
• High memory storage required as compared to other supervised learning algorithms.
• Prediction is slow in case of big N.
• It is very sensitive to the scale of data as well as irrelevant features.
17
APPLICATIONS OF KNN
Banking System
• KNN can be used in banking system to predict weather an individual is fit for loan
approval? Does that individual have the characteristics similar to the defaulters
one?
Calculating Credit Ratings
• KNN algorithms can be used to find an individual’s credit rating by comparing with
the persons having similar traits.
Politics
• With the help of KNN algorithms, we can classify a potential voter into various classes
like “Will Vote”, “Will not Vote”, “Will Vote to Party ‘Congress’, “Will Vote to Party
‘BJP’.
Other areas in which KNN algorithm can be used are Speech Recognition,
Handwriting Detection, Image Recognition and Video Recognition.
18
IMPLEMENTATION OF KNN ALGORITHM
Steps to Implement KNN KNN ALgorithm
• Data Pre-processing step • Following are the steps −
• Fitting the K-NN algorithm to the Training set • Step 1 − Load the training as well as test data.
• Predicting the test result • Step 2 − Choose the value of K i.e. the nearest data points. K can
be any integer.
• Test accuracy of the result(Creation of Confusion matrix)
• Step 3 − For each point in the test data do the following −
• Visualizing the test set result. Calculate the distance between test data and each row of
training data with the help of any of the method namely: Euclidean,
Manhattan or Hamming distance. Commonly used method to
calculate distance is Euclidean.
Now, based on the distance value, sort them in ascending
order.
Next, it will choose the top K rows from the sorted array.
Now, it will assign a class to the test point based on most
frequent class of these rows.
• Step 4 − End
19
DATA PRE-PROCESSING STEP
# importing libraries
import numpy as nm
import [Link] as mtp
import pandas as pd
#importing datasets
data_set= pd.read_csv('user_data.csv')
#Extracting Independent and dependent Variable
x= data_set.iloc[:, [2,3]].values
y= data_set.iloc[:, 4].values
20
DATA SPILTTING AND FEATURE
SCALING
# Splitting the dataset into training and test set.
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test= train_test_split(x, y, test_size= 0.25, random_state=0)
#feature Scaling
from [Link] import StandardScaler
st_x= StandardScaler()
x_train= st_x.fit_transform(x_train)
x_test= st_x.transform(x_test)
21
FITTING K-NN CLASSIFIER TO THE
TRAINING DATA
• :
Now we will fit the K-NN classifier to the training data. To do this we will import
the KNeighborsClassifier class of Sklearn Neighbors library. After importing the class, we
will create the Classifier object of the class. The Parameter of this class will
ben_neighbors: To define the required neighbors of the algorithm. Usually, it takes 5.
• metric='minkowski': This is the default parameter and it decides the distance between the
points.
• p=2: It is equivalent to the standard Euclidean metric.
22
FITTING K-NN CLASSIFIER TO THE
TRAINING DATA
#Fitting K-NN classifier to the training set
from [Link] import KNeighborsClassifier
classifier= KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2 )
[Link](x_train, y_train)
23
RESULT PREDICTION
#Predicting the test set result
y_pred= [Link](x_test)
24
CONFUSION MATRIX CREATION
#Creating the Confusion matrix
from [Link] import confusion_matrix
cm= confusion_matrix(y_test, y_pred)
25
TRAINING SET RESULT VISUALIZATION
#Visulaizing the trianing set result
from [Link] import ListedColormap
x_set, y_set = x_train, y_train
x1, x2 = [Link]([Link](start = x_set[:, 0].min() - 1, stop = x_set[:, 0].max() + 1, step =0.01),
[Link](start = x_set[:, 1].min() - 1, stop = x_set[:, 1].max() + 1, step = 0.01))
[Link](x1, x2, [Link]([Link]([[Link](), [Link]()]).T).reshape([Link]),
alpha = 0.75, cmap = ListedColormap(('red','green' )))
[Link]([Link](), [Link]())
[Link]([Link](), [Link]())
for i, j in enumerate([Link](y_set)):
[Link](x_set[y_set == j, 0], x_set[y_set == j, 1],
c = ListedColormap(('red', 'green'))(i), label = j)
[Link]('K-NN Algorithm (Training set)')
[Link]('Age')
[Link]('Estimated Salary')
[Link]()
[Link]()
26
TEST SET RESULT VISUALIZATION
#Visualizing the test set result
from [Link] import ListedColormap
x_set, y_set = x_test, y_test
x1, x2 = [Link]([Link](start = x_set[:, 0].min() - 1, stop = x_set[:, 0].max() + 1, step =0.01),
[Link](start = x_set[:, 1].min() - 1, stop = x_set[:, 1].max() + 1, step = 0.01))
[Link](x1, x2, [Link]([Link]([[Link](), [Link]()]).T).reshape([Link]),
alpha = 0.75, cmap = ListedColormap(('red','green' )))
[Link]([Link](), [Link]())
[Link]([Link](), [Link]())
for i, j in enumerate([Link](y_set)):
[Link](x_set[y_set == j, 0], x_set[y_set == j, 1],
c = ListedColormap(('red', 'green'))(i), label = j)
[Link]('K-NN algorithm(Test set)')
[Link]('Age')
[Link]('Estimated Salary')
[Link]()
[Link]()