Data Mining:
Concepts and Techniques
Chapter 7 —ML Model Evaluation Techniques
Dr. Yatendra Sahu
Assistant Professor, Department of CSE
Indian Institute of Information Technology Bhopal
1
Training To build
Dataset model
Dataset
To
Test Evaluate
Dataset Model
Training set:
Training set is used to build a model.
It is used find relevant information on how to associate input data with
output decision. The system is trained by applying these algorithms on the
dataset, all the relevant information is extracted from the data and results are
obtained.
Generally, 70% of the data of the dataset is taken for training data.
Testing set:
Testing data is used to test model. It is the set of data which is used to verify
whether the system is producing the correct output after being trained or
not. Generally, 30% of the data of the dataset is used for testing.
2
Learning Algorithm
Machine Learning is a concept which provides ability to the machine to
automatically learn and improve from experience without being explicitly
programmed.
The process of learning begins with observations in order to find patterns in
data and make better decisions in the future based on the examples that we
provide.
The primary aim of learning algorithm is to allow the computers learn
automatically without human intervention
Machine Learning
Algorithm
Supervised Un-Supervised Reinforcement
Learning Learning Learning
Algorithm Algorithm Algorithm 3
Types of Supervised Learning
Machine Learning
Algorithm
Supervised Un-Supervised Reinforcement
Learning Learning Learning
Algorithm Algorithm Algorithm
Regression Classification
4
10
Num-1 Num-2 Sum
5 5 10
8 2 10 5
Model Logic
10 3 13
5
15 6 21
20 4 21 Training Phase
30 40 70
30
Trained Model 70
40
Testing Phase
5
Evaluation Metric for Regression and
Classification
6
Binary Classification Multi-Class Classification
Source::
[Link]
DISEASE_PREDICTION_SYSTEM_HDPS/figures?lo=1
7
Classification Evaluation Metric
• True Positives (TP): Actual TRUE, which was predicted as TRUE
• True Negatives (TN): Actual FALSE, which was predicted as FALSE
• False Positives (FP): Actual FALSE, which was predicted as TRUE (Type I
error)
• False Negatives (FN): Actual TRUE, which was predicted as FALSE (Type
II error)
1) Confusion Matrix
A confusion matrix is created by comparing the predicted class label of a
data point with its actual class label.
A confusion matrix can be created for a binary classification as well as a
multi-class classification model. 8
Actual Predicted
Output (y) Output (y’) 1 0
1 3 1
1 1
1 1 1 1
0
0 1
0 0
1 0
1 1
9
It is a n*n matrix, where n is a number of classes.
10
[Link]
11
12
Why Confusion Matrix?
13
Accuracy
It is the proportion of the total number of predictions that are correct.
Accuracy
= (45+30)/(45+20+5+30) = 75%
The 75% of examples are correctly
classified by the classifier.
Normal Fraud
Normal
940 10
40 10
Fraud
14
Accuracy/Precision/ Recall
Source: [Link]
15
Precision
It is a ratio of total number of correctly classified positive examples and the
total number of predicted positive examples. It shows correctness achieved
in positive prediction.
Precision becomes important in cases where we are more concerned about
finding the maximum number of positive class even if the total accuracy
reduces.
Precision = TP/(TP+FP)
Precision = 45/(45+5)= 90%
The 90% of examples are classified as spam
are actually spam.
16
Recall / Sensitivity / True Positive Rate
It is measure of positive examples labelled as positive by classifier. It
should be higher.
Recall= TP/(TP+FN) = TP/P
Sensitivity = 45/(45+20) = 69.23% .
The 69.23% spam emails are correctly
Classify
17
Specificity / True Negative Rate
It is measure of negative examples labeled as negative by classifier.
There should be high specificity.
Specificity = TN/(TN+FP) = TN/N
specificity = 30/(30+5) = 85.71%
The 85.71% non-spam emails
are accurately classified
18
F1 Score
It is a weighted average of the recall and precision.
F1 score might be good choice when you seek to balance between
Precision and Recall.
19
20
How to Calculate Accuracy, Precision
and Recall for More than Two Classes
21
Source: [Link] 22
Source: [Link] 23
24
Source: [Link] 25
Education – Partnership – Solutions
26