Machine Learning
Introduction
Introduction
• Machine learning enables a machine to automatically learn from
data, improve performance from experiences, and predict things
without being explicitly programmed.
• In simple words, an ML algorithm requires a data set and some
learning strategy.
• A Machine Learning Algorithm may use Mathematics, Statistics,
Psychology, Biology or such other concepts to learn from data.
• Most Machine Learning Algorithms are AI algorithms but not all.
• Machine Learning algorithms learn iteratively.
What types of Problems may be solved by ML algorithms?
In general, ML algorithms may be used to solve problems related to:
• Classification
• Regression
• Clustering
Some machine learning algorithms are only used for classification.
Some are only used for Regression and some of them are used for both
classification as well as Regression.
Classification
• Classification refers to those tasks that have two or more class labels.
There are four main types of Classification:
• Binary Classification
• Email: spam detection (Spam or not)
• Multi-Class Classification (Performance of this person is Excellent or
Very Good, Good or Satisfactory.
• Multi-Label Classification
• Example: whether this fruit is orange, apple, mango or tomato
• Imbalanced Classification
• examples in each class are unequally distributed
Regression
Regression analysis is used to predict relationship between dependent
and independent parameters.
Examples:
Parameter Estimation
Forecasting
Prediction
Optimization
Clustering
Clustering is the task of dividing the population or data points into a number of groups
(a) (b) (c)
Difference b/w Regression and Classification
Source: [Link]
[Link]
vs-classification-
classification-in-
in-machine-
machine-learning
Regression Classification
• In Regression, the output variable must be of • In Classification, the output variable must be
continuous natural or real value. a discrete value.
• The task of the regression algorithm is to map • The task of the classification algorithm is to
the input value x with the continuous output map the input value x with the discrete
variable y. output variable y.
• Regression Algorithms are used with • Classification Algorithms are used with
continuous data. discrete data.
• In Regression, we try to find the best fit line, • In Classification, we try to find the decision
which can predict the output more boundary, which can divide the dataset into
accurately. different classes.
• Regression algorithms can be used to solve • Classification Algorithms can be used to solve
the regression problems such as Weather classification problems such as Identification
Prediction, House price prediction, etc. of spam emails, Speech Recognition,
Identification of cancer cells, etc.
• The Regression Algorithm can be further
divided into Linear and Non-linear • The Classification algorithms can be divided
Regression. into Binary Classifier and Multi-class Classifier.
Machine Learning Tasks
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning
Supervised Learning
input d(n)
+
y(n) -
e(n)
e(n) = d(n) – y(n)
• An ML algorithm trained in this way optimizes its
performance so that a cost function based on e(n) is
minimum.
• Supervised learning may be used to solve both Classification
and Regression problems.
• Unsupervised or Self Organizing Learning
No external teacher or supervisor available during training.
• clustering: group data according to "distance"
• association: find frequent co-occurrences
• link prediction: discover relationships in data
• data reduction: project features to fewer features
• Reinforcement Learning
• Learning with a Critic
Cost Function or Fitness Function or Objective Function
• It is a function that measures the performance of a
Machine Learning Algorithm from given data set.
• In its most basic form, cost function is the error
between predicted values and expected values, which
is to be minimized.
• The cost function is not always the error function. It can be
anything which is to be minimized or maximized.
Cost Function
Examples:
• Design a system so that energy consumption is
minimum. In this case, energy consumption is the
cost function.
• Design a system such that efficiency is maximum.
• Design a car with maximum fuel efficiency.
• Design an algorithm with minimum Computational
Cost.
• Design a communication system so that signal-to-
noise ratio is maximum.
• Design a controller so that the error between the
actual and desired output is minimum.
Cost Function
• Cost function is always minimized or maximized.
• Machine Learning Algorithms are trained by either
minimizing or maximizing a function.
• Cost function may be a function of a single variable or
more than one variable.
Example:
(i) Cost = min(energy) + max(efficiency)
(ii) Cost = min(weight) + max(stability)
More about Cost Function
e3(t) e4(t)
e1(t) e2(t)
Cost Function = e(t) = e1(t) – e2(t) + e3(t) – e4(t)
What will happen if e(t) = 0?
The system will think that no error is present
so further improvement is not necessary but
actually four individual errors are present.
• It is not always good to minimize error itself.
For better performance, squares of the error
e2(t) or absolute values of error |e(t)| or MSE
(Mean Squares Error) or any other function of
error should be used.
• Most of the Machine Learning Algorithms use MSE as
a cost function.
Machine learning algorithms minimizes or maximizes
a cost function iteratively.
Cost Function
Number of Iterations
Data Sets to be used in ML algorithms
• Training Data
A set of actual dataset to train the required model. A model is trained epoch
by epoch (i.e. iteratively). The model is said to be trained if it minimizes or
maximizes the chosen cost function.
A graph between cost function and number of epochs is plotted which
determines the training accuracy of the model.
• Testing Data
A given data set is usually divided into training and testing data. The size of
training set is usually larger than the testing data. (e.g. you may use 75% data
for training and 25% for testing). The testing data is used to determine the
performance of the model after training. This is called Testing Accuracy.
• Validation Data: This is a subset of training data. This data set is used to
ensure that the over-fitting does not occur. It is used to fine tune the model
parameters.
A successful ML algorithm should have good training, testing and validation
accuracy.
Evaluation of Classification Algorithms
• The Confusion Matrix is widely used for evaluation of a Classification
Algorithm.
• Several metrics can be computed from the Confusion matrix which
may determine the quality of a classifier. The metrics include:
• Precision
• Recall
• F1-Score
• Accuracy
• Weightage Average Precision/Recall
• Others
Confusion Matrix
• A confusion matrix is a technique for summarizing the performance of a
classification algorithm.
• The name stems from the fact that it makes it easy to see whether the
system is confusing two classes (i.e. commonly mislabeling one as
another).
• A Confusion matrix is an N x N matrix used for evaluating the performance
of a classification model, where N is the number of target.
• It is a square matrix where the column represents the actual values and
the row depicts the predicted value of the model and vice versa.
Predicted Condition
Actual Condition
TP FN
PF TN
where TP = True positive, FP = False Positive, FN = False Negative, TN = True Negative
In the above Confusion Matrix:
• True positive (TP) is the total number of correct results or predictions
when the actual class was positive.
• False positive (FP) is the total number of wrong results or predictions
when the actual class was positive.
• True negative (TN) is the total number of correct results or
predictions when the actual class was negative
• False negative (FN) is the total number of wrong results or
predictions when the actual class was negative.
Example 2: Following Table contains10 records with expected outcomes and a set of predictions from our
classification algorithm. Develop the Confusion matrix.
Actual man man woman man woman woman woman man man woman
Predicted woman man woman man man woman woman man woman woman
Solution:
Predicted
man woman man woman
Actual
man 3 2 man TP FN
woman 1 4 woman FP TN
TP = 3 FN = 2 FP = 1 TN = 4
Matlab v.16 Code:
actual=["man", "man", "woman", "man", "woman", "woman", "woman", "man", "man", "woman"]
predicted =["woman", "man", "woman", "man", "man", "woman", "woman", "man", "woman", "woman"]
confusionmat(actual, predicted)
Python Code:
from [Link] import confusion_matrix
yt=["man", "man", "woman", "man", "woman", "woman", "woman", "man", "man", "woman"]
yp =["woman", "man", "woman", "man", "man", "woman", "woman", "man", "woman", "woman"]
a=confusion_matrix(yt, yp)
print(a)
Example: Construct the Confusion Matrix from the following input
output data.
y 0 1 1 1 0 0 0
(Predicted)
y (Actual) 0 1 0 1 1 0 1
Solution:
Class 0 1
2 1
0
2 2
1
Class 0 1
TP FN
0
FP TN
1
TP = 2, FP = 2, FN = 1, TN = 2
Example: Construct the Confusion Matrix from the following data.
y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
Solution: y_pred
Class 0 1 2
y_actu
0 3 0 0
1 0 1 2
2 2 1 3
For Class 0: TP = 3, FP = 2, FN = 0, TN = 7
For Class 1: TP = 1, FP = 1, FN = 2, TN = 8
For Class 2: TP = 3, FP = 2, FN = 3, TN = 4
Example: Computer TP, FP, FN and TN from the following Confusion Matrix?
Class a b c
a 16 0 0
b 0 17 1
c 0 0 11
Solution:
Class a: TP = 16, FP = 0 +0=0, FN = 0 + 0 = 0, TN =17 + 1+0+11 = 29
Class b: TP = 17, FP = 0 + 0 = 0, FN = 0 + 1 = 1, TN = 16 + 0 + 0 + 11 = 27
Class c: TP = 11, FP = 0 + 1 = 1, FN = 0 + 0 = 0, TN = 16+0+0+17 = 33
Evaluation of Classifiers
Classifiers are evaluated based on some or all of the following parameters:
Misclassification = Error Rate =
Precision = =
( )=
Specificity =
F1-Score =
Example: Construct the confusion matrix from the following input-
output data and hence compute Recall, Precision and Accuracy.
Y 0 1 0 1 1 0 1
(Actual)
y (Pred.) 0 1 1 1 0 0 0
Predicted
Class 0 1
Actual
0 2 1
1 2 2
! "
Solution: TP = 2, FP = 1, TN = 2, FN = 2. Recall = =! =
! !
! ! *
Precision = = = Accuracy = =
! " # +
Example: Calculate the Accuracy, Precision, Recall and F1-Score of the
following confusion matric
Class A B C
A 15 2 3
B 7 15 8
C 2 3 45
Solution:
Class A: TP = 15, FP = 7+2 = 9, FN = 2 + 3 = 5, TN = 15+8+3+45=71
,- 4, 5,
= = = = 0.81
,- . - 4, ,00
,- ,-
Precision = = = = 0. 2 -, ( )= =
,-
=
,-
= 0.75
,- . / ,- - 0
!×7.8!9×7.+9
F1-Score = = = 0.68
7.8!9 7.+9
Class A B C
A 15 2 3
B 7 15 8
C 2 3 45
Class B:
TP = 15, FP = 2 + 3 = 5, FN = 7 + 8 =15, TN = 15 + 3 + 2 + 45 = 65
,- 2- 50
= = = = 0.80
,- - ,- 2- ,00
,- ,- ,- ,-
Precision = = = = 0. 4-, ( )= = = = 0. -
,- - 0 ,- ,- :0
!×7.+9×7.9
F1-Score = = = 0.6
7.+9 7.9
Class A B C
A 15 2 3
B 7 15 8
C 2 3 45
Class C:
TP = 45, FP = 3 + 8 = 11, FN = 2 + 3 = 5, TN = 15 +2 + 7 + 15 = 39
/- :. 5/
= = /- = ,00 = 0.84
:. ,, -
*9 *9 /- /-
Precision = = = = 0.80, ( )= = = = 0. .
*9 "" 98 /- - -0
!×7.;7×7.<
F1-Score = = = 0.847
7.;7 7.<
Compute the Recall, Precision and F1-Score from the following
confusion matrix
C1 C2 C3 C4 C5
C1 0.885 0.045 0.010 0.007 0.054
C2 0.182 0.796 0.022 0.000 0.000
C3 0.069 0.000 0.800 0.068 0.064
C4 0.007 0.028 0.059 0.854 0.053
C5 0.034 0.000 0.046 0.048 0.872
Class C1:
TP = 0.885, FP = 0.182+0.069+0.007+0.034 = 0.292,
FN = 0.045 + 0.010+0.007+0.054 = 0.116
TN = 0.796 + 0.022 + 0.0 + 0.0 + 0.0+0.8 + 0.068 + 0.064+.028+0.059+0.854+0.053+
0+0.046+0.048+0.872 = 3.71
Precision of C1 = TP/(TP+FP) = 0.75, Recall of C1 = TP/(TP+FN) = 0.884
Short Cut Method for calculation of classification Matrices
Compute the Precision, Recall, Accuracy, weightage average Precision
and weightage average Recall from the following confusion matrix.
Class A B C
A 15 2 3
B 7 15 8
C 2 3 45
Solution:
Class A B C Total Precision =
A 15 2 3 20 ,-
Class A Precision = = 0. 2 -
B 7 15 8 30 /
,-
C 2 3 45 50 Class B Precision = = 0. 4-
0
24 20 56 100
/-
Class C Precision = = 0. 5
-2
Class A B C Total =
Recall =
A 15 2 3 20
,-
B 7 15 8 30 Class A Recall = = 0. 4-
0
C 2 3 45 50
,-
Class B Recall = = 0. -
24 20 56 100 :0
/-
Class C Recall = = 0. .
-0
>?@A B>CCDE?AF BA@GGHIHDJ "9 "9 *9 +9
Accuracy = KE?L@A
= "77
= "77
= 0.75
Repeat the previous example for the following confusion matrix
C1 C2 C3 C4 C5
C1 0.885 0.045 0.010 0.007 0.054
C2 0.182 0.796 0.022 0.000 0.000
C3 0.069 0.000 0.800 0.068 0.064
C4 0.007 0.028 0.059 0.854 0.053
C5 0.034 0.000 0.046 0.048 0.872
C1 C2 C3 C4 C5 Total
Solution:
C1 0.885 0.045 0.010 0.007 0.054 1.001
C2 0.182 0.796 0.022 0.000 0.000 1.00
C3 0.069 0.000 0.800 0.068 0.064 1.001
C4 0.007 0.028 0.059 0.854 0.053 1.001
C5 0.034 0.000 0.046 0.048 0.872 1
1.177 0.869 0.937 0.977 1.043 5.003
C1 C2 C3 C4 C5 Total
C1 0.885 0.045 0.010 0.007 0.054 1.001 Precision =
C2 0.182 0.796 0.022 0.000 0.000 1.00
=
C3 0.069 0.000 0.800 0.068 0.064 1.001 Recall =
C4 0.007 0.028 0.059 0.854 0.053 1.001
C5 0.034 0.000 0.046 0.048 0.872 1
1.177 0.869 0.937 0.977 1.043 5.003
0.55- 0.55-
Precision of Class C1 = = 0.75 Recall of Class C1 = = 0.884
,.,44 ,.00,
0.4.2 0.4.2
Precision of Class C2 = = 0.926 Recall of Class C2 = = 0.796
0.52. ,
0.5 0.5
Precision of Class C3 = = 0.854 Recall of Class C3 = = 0.799
0..:4 ,.00,
0.5-/
Precision of Class C4 =
0.5-/
= 0.874 Recall of Class C4 = = 0.853
0..44 ,.00,
0.54
0.54
= 0.836 Recall of Class C5 = = 0.872
Precision of Class C4 = ,
,.0/:
C1 C2 C3 C4 C5 Total
>?@A B>CCDE?AF BA@GGHIHDJ
Accuracy =
C1 0.885 0.045 0.010 0.007 0.054 1.001 KE?L@A
C2 0.182 0.796 0.022 0.000 0.000 1.00
7.;;9 7.+<8 7.; 7.9;* 7.;+!
=
9.77#
C3 0.069 0.000 0.800 0.068 0.064 1.001
C4 0.007 0.028 0.059 0.854 0.053 1.001 = 0.7869
C5 0.034 0.000 0.046 0.048 0.872 1
1.177 0.869 0.937 0.977 1.043 5.003
Evaluation of Regression Algorithms
There are several metrics which can be used to evaluate the
performance of a regression algorithm. Some of these are:
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)
• R2 (Coefficient of Determination)
• Adjusted R2.
Mean Squared Error (MSE):
Mean Square Error is an absolute measure of the goodness for the fit.
It is calculated as:
j
1
MSE = e fg − fig !
N
gk"
where yi = actual output, fig = predicted output
Root Mean Square Error (RMSE) = MSE
"
Mean Absolute Error (MAE) = ∑Hk" yH − yiH
Coefficient of Determination (R2)
• R2 shows how well the data fit the regression model (the goodness
of fit).
• Its range is from 0 to 1. For example, an R2 of 0.6 reveals that 60% of
the data fit the regression model.
• A Regression Model should not be evaluated with R2 only. Other
metrics should also e used.
m ∑ nF o∑(n) ∑(F)
•R=
m ∑ np o(∑ n)p m ∑(Fp )o(∑ F)p
("oqp )(ro")
R2
• Adjusted =1 - roso"
where R2 = sample R2, n = Total sample
size, p = Number of independent variables (predictors)
The Coefficient of Determination is also calculated as follows:
ttu
R! = tt
where SSR = Sum of Squared Regression = ∑g (fig − fvg )!
SST = Sum of Squares Total = ∑g fg − fvg !
Determine Coefficient of Determination from the following data:
x y
1.0 2.6
2.3 2.8
3.1 3.1
4.8 4.7
5.6 5.1
6.3 5.3
Solution: x y xy x2 y2
1 2.6 2.6 1.0 6.8
2.3 2.8 6.44 5.3 7.8
3.1 3.1 9.61 9.6 9.6
4.8 4.7 22.56 23 22.0R9
5.6 5.1 28.56 31.4 26
6.3 5.3 33.39 39.7 28.1
e w = :. ,
∑ x =23.6 ∑(wx) = ∑(w ) = ∑ x =
103.16 110 100.4
m ∑ nF o∑(n) ∑(F)
R= = 0.9741, R2 = 0.9488
m∑ np o(∑ n)p m ∑(Fp )o(∑ F)p
Determine Coefficient of determination from the following data:
x y
0 0.01
0.1 0.116
0.2 0.23
0.4 0.476
0.6 0.661
0.8 0.811
1.00 1.123
Solution:
x y xy x2 y2
0.0 0.01 0 0 0.0001
0.1 0.116 0.0116 0.01 0.013456
0.2 0.23 0.046 0.04 0.0529
0.4 0.476 0.1904 0.16 0.226576
0.6 0.661 0.3966 0.36 0.436921
0.8 0.811 0.6488 0.64 0.657721
1.00 1.123 1.123 1 1.261129
∑ w = 3.1 ∑ x = :. / 4 ∑(wx) = ∑(w ) = ∑ x =
. /,2/ . , :. .,
m ∑ nF o∑(n) ∑(F)
R= = 0.9984, R2 = 0.9968
m∑ np o(∑ n)p m ∑(Fp )o(∑ F)p
suppose that the Human Resources department of a major corporation
wants to determine whether the salaries of its employees are related
to the employees' years of work experience and their level of graduate
education. To test this idea, the HR department picks a sample of eight
employees randomly and records their annual salaries (measured in
thousands of dollars per year), years of experience, and years of
graduate education. Find the adjusted coefficient of determination if R2
= 0.944346527.
Solution: n = 8, p = 2, R2 = 0.944346527
("oqp )(ro")
R2 =1 - roso"
= 0.922085138
Example: Calculate the Coefficient of Determination and Adjusted
Coefficient of Determination