MACHINE LEARNING FOR REAL

WORLD APPLICATION
PCC-CSD601

Evolutionary Matrices in Linear Regression

Residual = actual value — predicted value
e=y—ŷ

Residual plots show the residual values on the y-axis and predicted values
on the x-axis. If your model is biased you cannot trust the results.

Residual plot showing the errors corresponding to the predicted values

must be randomly distributed. However, if there are any signs of a
systematic pattern, then your model is biased.

Evaluation metrics are a measure of how good a model performs and how
well it approximates the relationship. Let us look at MSE, MAE, R-
squared, Adjusted R-squared, and RMSE.

Mean Squared Error (MSE)

The most common metric for regression tasks is MSE. It has a convex
shape. It is the average of the squared difference between the predicted
and actual value. Since it is differentiable and has a convex shape, it is
easier to optimize.

Mean Absolute Error (MAE)

This is simply the average of the absolute difference between the target
value and the value predicted by the model. Not preferred in cases where
outliers are prominent.

R-squared or Coefficient of Determination

This metric represents the part of the variance of the dependent variable
explained by the independent variables of the model. It measures the
strength of the relationship between your model and the dependent
variable.

We know that residual is the difference between actual and predicted

value. Thus, RSS (Residual sum of squares) can be calculated as follows.

The total variation in Y can be given as a sum of squared differences of

the distance between every point and the arithmetic mean of Y values.
This can be termed as TSS (Total sum of squares).

Image by the author.

To summarize, the ratio of the residual error (RSS) against the total error
(TSS) tells you how much of the total error remains in your regression
model. Subtracting that ratio from 1 gives how much error you removed
using the regression analysis. That is the R-squared error.

If R² is high (say 1), then the model represents the variance of the
dependent variable.

If R² is very low, then the model does not represent the variance of the
dependent variable and regression is no better than taking the mean value,
i.e. you are not using any information from the other variables.

A Negative R² means you are doing worse than the mean value. It can
have a negative value if the predictors do not explain the dependent
variables at all such that RSS ~ TSS.

Thus R² evaluates the scattered data points about the regression line.

It is not possible to see a model with an R² of 1. In that case, all predicted

values are the same as actual values and this essentially means that all
values fall on the regression line.

Root Mean Squared Error (RMSE)

This is the square root of the average of the squared difference of the
predicted and actual value.
R-squared error is better than RMSE. This is because R-squared is a
relative measure while RMSE is an absolute measure of fit (highly
dependent on the variables — not a normalized measure).

Basically, RMSE is just the root of the average of squared residuals. We

know that residuals are a measure of how distant the points are from the
regression line. Thus, RMSE measures the scatter of these residuals.

Evaluation Matrix for Classification Problem:

What is a Confusion Matrix?

A confusion matrix is a matrix that summarizes the performance of a machine
learning model on a set of test data. It is a means of displaying the number of
accurate and inaccurate instances based on the model’s predictions. It is often
used to measure the performance of classification models, which aim to predict
a categorical label for each input instance.
The matrix displays the number of instances produced by the model on the test
data.
• True positives (TP): occur when the model accurately predicts a positive
data point.
• True negatives (TN): occur when the model accurately predicts a
negative data point.
• False positives (FP): occur when the model predicts a positive data point
incorrectly.
• False negatives (FN): occur when the model mispredicts a negative data
point.
Why do we need a Confusion Matrix?
When assessing a classification model’s performance, a confusion matrix is
essential. It offers a thorough analysis of true positive, true negative, false
positive, and false negative predictions, facilitating a more profound
comprehension of a model’s recall, accuracy, precision, and overall
effectiveness in class distinction. When there is an uneven class distribution in
a dataset, this matrix is especially helpful in evaluating a model’s performance
beyond basic accuracy metrics.
Let’s understand the confusion matrix with the examples:
Confusion Matrix For binary classification
A 2X2 Confusion matrix is shown below for the image recognition having a Dog
image or Not Dog image.

• True Positive (TP): It is the total counts having both predicted

and actual values are Dog.
• True Negative (TN): It is the total counts having both predicted
and actual values are Not Dog.
• False Positive (FP): It is the total counts having prediction is Dog
while actually Not Dog.
• False Negative (FN): It is the total counts having prediction is
Not Dog while actually, it is Dog.

Example for binary classification problems

• Actual Dog Counts = 6

• Actual Not Dog Counts = 4
• True Positive Counts = 5
• False Positive Counts = 1
• True Negative Counts = 3
• False Negative Counts = 1
Metrics based on Confusion Matrix Data
1. Accuracy
Accuracy is used to measure the performance of the model. It is the ratio of Total correct
instances to the total instances.

For the above case:

Accuracy = (5+3)/(5+3+1+1) = 8/10 = 0.8

2. Precision
Precision is a measure of how accurate a model’s positive predictions
are. It is defined as the ratio of true positive predictions to the total
number of positive predictions made by the model.

3. Recall
Recall measures the effectiveness of a classification model in identifying all relevant
instances from a dataset. It is the ratio of the number of true positive (TP) instances to the
sum of true positive and false negative (FN) instances.

For the above case:

Recall = 5/(5+1) =5/6 = 0.8333

Note: We use precision when we want to minimize false positives, crucial in scenarios like
spam email detection where misclassifying a non-spam message as spam is costly. And we
use recall when minimizing false negatives is essential, as in medical diagnoses, where
identifying all actual positive cases is critical, even if it results in some false positives.

Specificity:
Specificity is another important metric in the evaluation of classification models,
particularly in binary classification. It measures the ability of a model to correctly identify
negative instances. Specificity is also known as the True Negative Rate.

Specificity=3/(1+3)=3/4=0.75

Type 1 and Type 2 error

Type 1 error
Type 1 error occurs when the model predicts a positive instance, but it is actually negative.
Precision is affected by false positives, as it is the ratio of true positives to the sum of true
positives and false positives.

For example, in a courtroom scenario, a Type 1 Error, often referred to as a false positive,
occurs when the court mistakenly convicts an individual as guilty when, in truth, they are
innocent of the alleged crime. This grave error can have profound consequences, leading to
the wrongful punishment of an innocent person who did not commit the offense in question.
Preventing Type 1 Errors in legal proceedings is paramount to ensuring that justice is
accurately served and innocent individuals are protected from unwarranted harm and
punishment.

Type 2 error
Type 2 error occurs when the model fails to predict a positive instance. Recall is directly
affected by false negatives, as it is the ratio of true positives to the sum of true positives and
false negatives.
In the context of medical testing, a Type 2 Error, often known as a false negative, occurs
when a diagnostic test fails to detect the presence of a disease in a patient who genuinely
has it. The consequences of such an error are significant, as it may result in a delayed
diagnosis and subsequent treatment.
Precision emphasizes minimizing false positives, while recall focuses on minimizing false
negatives.

How to interpret a confusion matrix?

A confusion matrix summarizes a classification model’s performance,
with entries representing true positive, true negative, false positive, and
false negative instances, providing insights into model accuracy and
errors.

2. What are the advantages of using Confusion matrix?

The confusion matrix provides a comprehensive evaluation of a
classification model’s performance, offering insights into true positives,
true negatives, false positives, and false negatives, aiding nuanced
analysis beyond basic accuracy.

[Link] are some examples of confusion matrix applications?

Confusion matrices find applications in various fields, including medical
diagnosis (identifying true/false positives/negatives for diseases), fraud
detection, sentiment analysis, and image recognition accuracy
assessment.

4. What is the confusion matrix diagram?

A confusion matrix diagram visually represents the performance of a
classification model. It displays true positive, true negative, false
positive, and false negative values in a structured matrix format.

[Link] are the three values of the confusion matrix?

The three values of the confusion matrix are true positive (correctly
predicted positive instances), true negative (correctly predicted negative
instances), and false positive (incorrectly predicted positive instances).
Till the homogeneity ‘H’ is less than the threshold, you need to continue splitting the node. The process of splitting
needs to be continued until homogeneity exceeds the threshold value and the majority data points in the node are
of the same class.

Now, how do you handle best split for different attributes in a decision tree using the CART algorithm?
CART stands for classification and regression decision trees. They are quite similar to the human decision-making
process and are, hence, easy to understand. However, the most attractive feature of this algorithm is the flexibility of
input variables, i.e., it handles both categorical and continuous variables.

A tree can be split based on different rules of an attribute and these attributes can be categorical or continuous in

nature. If an attribute is nominal categorical, then there are possible splits for this attribute, where k is
the number of classes. In this case, each possible subset of categories is examined to determine the best split.

If an attribute is ordinal categorical or continuous in nature with n different values, there are n - 1 different possible
splits for it. Each value of the attribute is sorted from the smallest to the largest and candidate splits based on the
individual values are examined to determine the best split point which maximizes the homogeneity at a node.

There are various other techniques like calculating percentiles and midpoints of the sorted values for handling and
getting rules from continuous features for splitting in different algorithms and this process is known as discretization.

There are various methods used to quantify homogeneity, such as the classification error, Gini index and entropy.

The classification error is calculated as follows:

In practice, classification error does not perform well. So, we generally prefer using either the Gini index or entropy
over it.

While using CART, the measure used for calculating the information gain for categorical target variables is the ​Gini
Index​. The Gini index measures the number of times a random variable is incorrectly identified. It is calculated using
the following formula:

where pi is the probability of finding a point with the label i, and k is the number of classes.

Gini index of 0 indicates that all the data points belong to a single class. Gini index of 0.5 indicates that the data
points are equally distributed among the different classes.

Suppose you have a data set with two class labels. If the data set is completely homogeneous, i.e., all the data points
belong to label 1, then the probability of finding a data point corresponding to label 2 will be 0 and that of label 1 will

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

be 1. So, p1 = 1 and p2 = 0. The Gini index, which is equal to 0, will be the lowest in such a case. Hence, the higher
the homogeneity, the lower the Gini index.

CART is an algorithm for building decision trees with the Gini index as a splitting criterion. It is a binary tree that can
be built by splitting nodes into two child nodes repeatedly. You can refer to the following steps for building a
decision tree using the CART algorithm:

1. Calculate the Gini index before splitting on the whole data set.

2. Consider any one of the available attributes.

3. Calculate the Gini index after splitting on that particular attribute for each of the levels of the attribute.

4. Combine the Gini index of all the levels to obtain the Gini index of the overall attribute.

5. Repeat steps 2​–​5 with another attribute until you have exhausted all of them.

6. Compare the Gini index across all the attributes and select the one that has the minimum Gini index.

Let's go ahead and see how Gini index can be used to decide where to make the split on the data point using an
example.

Suppose you have a dataset for 100 patients and the target variable consists of two classes: class 0 having 60 people
with no heart disease and class 1 having 40 people with a heart disease with two attributes i.e. ‘sex’ and
‘cholesterol’. While making your first split, you need to choose an attribute such that the purity gain is maximum.
You can calculate the Gini index of the split on ‘sex’ (gender) and compare that with the Gini index of the split based
on ‘cholesterol’.

Expressing this in terms of probabilities you get:

Now, you can calculate the gini index for the data before making any splits as follows:

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

Let's now evaluate which split gives the maximum reduction in impurity among the possible choices. You have the
following information about the target variable and the two attributes.

As you can see, the table above shows the number of diseased/non-diseased people w.r.t. the levels in the two
attributes - 'Sex' and 'Cholesterol'. Let's calculate the homogeneity reduction on each attribute individually, starting
with 'Sex'.

Split based on Sex

Let's consider the first candidate split based on sex/gender. As you can see from the first table, of the 100 people,
you have 70 males and 30 females. Among the 70 males i.e. the child node containing males, ​50 belong to class 0 i.e,
they do not have a heart disease and the rest ​20 males belong to class 1 having a heart disease. So basically for the
split on "Sex", you have something like this —

[Note that (x, y) on any node means (# Label 0, # Label 1)]

Now the probabilities of the two classes within the male subset comes out to be:

p0=50/70=0.714 and p1=20/70=0.286

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

Now using the same formula, Gini impurity for males becomes:

0.714(1−0.714)+0.286(1−0.286)=0.41

Let's now take the other case i.e. the child node containing females, where there are 30 females out of which 10
belong to class 0 having no heart disease and 20 belong to class 1 having a heart disease. The probabilities of the two
classes within the female subset comes out to be:

p0=10/30=0.333 and p1=20/30=0.667

Now using the formula, Gini impurity for females becomes:

0.333(1−0.333)+0.667(1−0.667)=0.44

Now how do you get the overall impurity for the attribute 'sex' after the split? You can aggregate the Gini impurity of
these two nodes by taking a weighted average of the impurities of the male and female nodes. So, you have -

pmale=70/100=0.7 and pfemale=30/100=0.3

This gives the Gini impurity after the split based on gender as:

0.7×0.41+0.3×0.44=0.42

Thus, the split based on gender gives the following insights:

• Gini impurity before split = 0.48

• Gini impurity after split = 0.42

• Reduction in Gini impurity = 0.48 - 0.42 = 0.06

Hence, you get the following tree after splitting on 'Sex' —

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

Split based on Cholesterol

Let's now take another candidate split based on cholesterol. You divide the dataset into two subsets: Low
Cholesterol (Cholesterol < 250) and High Cholesterol (Cholesterol > 250). There are 60 people belonging to the low
cholesterol group and 40 people belonging to the high cholesterol group.

If you see the second table given above, you will notice that among the 60 low cholesterol people, 50 belong to class
0, i.e, they do not have a heart disease and the rest 10 belong to class 1 having a heart disease. So basically for the
split on "Cholesterol", you have something like this —

Now the probabilities of the two classes within the low cholesterol subset comes out to be:

p0=50/60=0.833 and p1=10/60=0.167

Now using the formula, Gini impurity for low cholesterol subset becomes:

0.833(1−0.833)+0.167(1−0.167)≈0.27

Let's now take the other case where there are 40 high cholesterol (Cholesterol > 250) people out of which 10 belong
to class 0 having no heart disease and 30 belong to class 1 having a heart disease. The probabilities of the two classes
within the high cholesterol subset comes out to be:

p0=10/40=0.25 and p1=30/40=0.75

Now using the formula, Gini impurity for high cholesterol subset becomes:

0.25(1−0.25)+0.75(1−0.75)≈0.37

The overall impurity for the data after the split based on cholesterol can be computed by taking a weighted average
of the impurities of the high and low cholesterol nodes. So, you have -

plow−cholesterol=60/100=0.6 and phigh−cholesterol=40/100=0.4

This gives the Gini impurity after the split based on cholesterol as:

0.6×0.27+0.4×0.37≈0.3

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

Thus, the split based on cholesterol gives the following insights:

• Gini impurity before split = 0.48

• Gini impurity after split = 0.3

• Reduction in Gini impurity = 0.48 - 0.3 = 0.18

Hence, you get the following tree after splitting on 'Cholesterol' —

Hence, from the above example, it is evident that we get a significantly higher reduction in Gini impurity when you
split the dataset on cholesterol as compared to when you split on gender.

While dealing with CART, the Gini index is the splitting criterion for classification trees or categorical variables.
However, in the case of continuous variables, the splitting criterion is the MSE (mean squared error). The MSE is not
different from variance, which is nothing but the square of standard deviation.

Some additional points and summary of key learnings so far have been included below:

1. A categorical tree is one that has class labels as its leaf nodes, whereas a continuous tree has numerical
values as its leaf nodes.

2. CART can only deal with binary trees, whereas ID3 can deal with non-binary trees (trees that have more than
two splits) as well.

3. ID3 uses entropy/information gain to measure the impurity of the variable, whereas CART uses the Gini
index for categorical target variables.

4. Both the entropy and the Gini index measure the impurity of the variable or the number of times a variable
is incorrectly classified. Therefore, your main aim while building the tree is to achieve a reduction in these
© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved
 values.

5. CART uses MSE/variance for continuous variables.

6. Standard deviation is nothing but the square root of the MSE value or the variance of a variable.

Entropy: ​Entropy is the measure of the randomness of a variable. Thus, a higher entropy means more randomness,
which leads to less purity. Entropy quantifies the degree of disorder in the given data. Its value varies from 0 to 1.

In simple words, a pure data set means that all the data belongs to the same class and the entropy is zero, whereas
an impure one means data is divided over different classes and the entropy is greater than zero but less than one. If
a data set contains an equal distribution of both the classes, then the entropy of that data set will be 1, i.e., there is
complete disorder in the data. Hence, like the Gini index, the higher the homogeneity, the lower the entropy. The
entropy of a variable in a given class can be calculated using the following formula:

where pi is the probability of finding a point with the label i, and k is the number of classes.

The graphical representation given below will help you understand the outcomes of the formula:

Some important facts about entropy that can be derived from the formula and the image given above are as follows:
1. If all the variables belong to the same class, the probability is one or zero, the entropy is zero and the purity
is maximum.

2. If the variables are equally divided over two classes (as in the case of binary classification), the probability is
0.5 for both the classes, the entropy is 1 and the purity is minimum.

Information gain: ​This is the main deciding parameter in determining the feature that is to be split at each node.
While splitting, the main factor to be kept in mind is purity. The end solution should have maximum purity as
compared with other trees that can be created in its place. The measure that you use for purity is information gain.

As the name suggests, information gain calculates the measure of information that you can obtain from a variable.
You need to continue the splitting process until the value of information gain is zero or very small. However, in order
to do so, you should keep in mind the following assumptions while constructing a tree:

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

 1. When you begin creating the tree, the whole set of training data is considered a root node.

2. A recursive approach is followed while creating the tree.

3. You will use a statistical approach while finding the root node and the internal nodes.

The change in impurity or the purity gain is given by the difference of impurity post-split from impurity pre-split, i.e.,
Δ Impurity = Impurity (pre-split) – Impurity (post-split)

The post-split impurity is calculated by finding the weighted average of two child nodes. The split that results in
maximum gain is chosen as the best split.

To summarise, the information gain is calculated by:

Where D is the entropy of the parent set (data before splitting),DA is the entropy of the partitions obtained after
splitting on attribute A. Note that reduction in entropy implies information gain.

Let's understand how we compute information gain with an example. Suppose you have four data points out of
which two belong to the class label '1', and the other two belong to the class label '2'. You split the points such that
the left partition has two data points belonging to label '1', and the right partition has the other two data points that
belong to label '2'. Now let's assume that you split on some attribute called 'A'.

So, the information gain after splitting the original data set on attribute 'A' is 1.0. You always try to maximise
information gain by achieving maximum homogeneity and this is possible only when the value of entropy decreases
from the parent set after splitting.

In case of a classification problem, you always try to maximise purity gain or reduce the impurity at a node after

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved

every split and this process is repeated till you reach the leaf node for the final prediction.

Look at the figure given below to understand the range of values each of the impurity measures can take.

The scaled version of the entropy in the figure is nothing but entropy/2. It has been used to emphasize that the Gini
index is an intermediate measure between entropy and the classification error.

In regression problems, a decision tree splits the data into multiple subsets. The difference between decision tree
classification and decision tree regression is that in regression, each leaf represents the average of all the values as
the prediction as opposed to a class label in classification trees. For classification problems, the prediction is assigned
to a leaf node using majority voting but for regression, it is done by taking the average value. This average is
calculated using the following formula:

Where yi’s represent the observations in a node, Nt refers to the total number of observations and Dt refers to the
dataset.

For example, suppose you are predicting the sales your company will have based on various factors such as
marketing, no. of products, etc. Now, if you use a decision tree to solve this problem (the process of actually building
a regression tree is covered in the next session), and if one of the leaf nodes has, say, 5 data points, 1 Cr, 1.3 Cr, 0.97
Cr, 1.22 Cr, 0.79 Cr. Now, you will just take the average of these five values which comes out to be 1.07 Cr, and that
becomes your final prediction.

Decision tree classification is what you’ll most commonly work on. However, remember that if you get a data set
where you want to perform regression, decision tree regression is also a good idea.

© Copyright 2020. upGrad Education Pvt. Ltd. All rights reserved