TDB3123

Universiti Teknologi PETRONAS

Recap Chapter 4: Feature Scaling and
Selection
✓ To understand the process of feature scaling

✓ To understand the process of feature selection

 Chapter 5:
Supervised Machine Learning
 Learning Outcomes
At the end of this chapter, you should be able
1. To explain the supervised learning process
▪ Classification
▪ Regression
2. To implement the application of supervised machine learning
models:
▪ Regression Model
▪ Decision Trees
▪ Support Vector Machine
▪ Neural Networks
▪ Naives Bayes
▪ Random Forest
Cross-Industry Standard Process for Data Mining (CRISP-DM)
Standard Machine Learning Pipeline
 Supervised Learning
Supervised learning methods or algorithms include learning algorithms that take in data
samples (known as training data) and associated outputs (known as labels or responses)
with each data sample during the model training process.

The main objective is to learn a mapping or association between input data samples x and
their corresponding outputs y based on multiple training data instances.

This learned knowledge can then be used in the future to predict an output y′ for any new
input data sample x′ which was previously unknown or unseen during the model training
process.

These methods are termed as supervised because the model learns on data samples where
the desired output responses/labels are already known beforehand in the training phase.
Supervised Machine Learning Pipeline
 Supervised Learning
Supervised learning methods are of two major classes based on the type of ML tasks they
aim to solve:

1. Classification - objective is to predict output labels or responses that are categorical

in nature for input data based on what the model has learned in the training phase.
Output labels here are also known as classes or class labels are these are categorical in
nature meaning they are unordered and discrete values. Thus, each output response
belongs to a specific discrete class or category.

2. Regression - objective is value estimation can be termed as regression tasks.

Regression based methods are trained on input data samples having output responses
that are continuous numeric values unlike classification, where we have discrete
categories or classes.
Supervised Learning – Regression Example
Supervised Learning – Classification Example
1. Regression Model
 Linear Regression Model

Linear Regression establishes a

relationship between
dependent variable (Y) and one
or more independent variables
(X) using a best fit straight line
(also known as regression line).

It is represented by an equation
Y= mX + c.
 Linear Regression
Let’s generate some linear-looking data to test this equation on (refer to code provided):
 Polynomial Regression
Let’s generate some non-linear looking data to test this equation on (refer to code provided):
 Multiple Regression
Multiple regression is like linear regression, but with more than one independent
value, meaning that we try to predict a value based on two or more variables.

Car Model Volume Weight CO2

Toyota Aygo 1000 790 99
We can predict the CO2 emission
Mitsubis Space 1200 1160 95 of a car based on the size of the
hi Star engine, but with multiple regression
Skoda Citigo 1000 929 95 we can throw in more variables, like
Fiat 500 900 865 90 the weight of the car, to make the
Mini Cooper 1500 1140 105 prediction more accurate.
VW Up! 1000 929 105

Let us try multiple regression example!

 Logistic Regression
Logistic regression is a statistical method for predicting binary classes. The
outcome or target variable is dichotomous in nature. Dichotomous means there are
only two possible classes.

For example, it can be used for cancer detection problems. It computes the
probability of an event occurrence.
 Regression Models

Pros:
• Linear Regression is simple to implement and easier to interpret the output
coefficients.

Cons:
• Linear regression technique, outliers can have huge effects on the
regression and boundaries are linear in this technique.
2. Decision Trees
 Decision Tree
Decision tree is a tree-like structure where internal nodes represent a test on an
attribute, each branch represents outcome of a test, and each leaf node
represents class label, and the decision is made after computing all attributes.

A path from root to leaf represents classification rules. Thus, a decision tree consists
of Root node, Branch node and Leaf node (class label).

Use training data to build a tree generator model, which will determine which variable
to split at a node and the value of the split.

A decision to stop or split again assigns leaf nodes to a class. An advantage of a

decision tree is that there is no need for the exclusive creation of dummy
variables.
 Decision Tree

Decision tree model output

is easy to interpret, and it
provides the rules that drive
a decision or event; in the
above use case we can get
the rules that lead to a don’t
play scenario, that is 1)
sunny and temperature
>30°c 2)rainy and windy is
true
 Decision Tree
Classification and Regression
Trees (CART) is a term used to
refer to the Decision Tree algorithm.

Root (brown) and decision (blue)

nodes contain questions which split
into subnodes. The root node is just
the topmost decision node.

The leaf nodes (green), also called

terminal nodes, are nodes that don’t
split into more nodes. Leaf nodes
are where classes are assigned by
majority vote.
 Decision Tree
Pros:
• Compared to other algorithms decision trees requires less effort for data
preparation during pre-processing.
• A decision tree does not require normalization of data and scaling.

Cons:
• For a Decision tree sometimes calculation can go far more complex compared
to other algorithms.
• Decision tree often involves higher time to train the model.
• Decision tree training is relatively expensive as the complexity and time have
taken are more.
• The Decision Tree algorithm is inadequate for applying regression and
predicting continuous values.
3. Ensemble Learning
 Ensemble Learning

Ensemble learning is a type of learning where you join different types of

algorithms or same algorithm multiple times to form a more powerful
prediction model.

Types of Ensemble Methods

1. BAGGing, or Bootstrap AGGregating.
• Learners’ model that learns from each other independently in parallel and combines
them for determining the model average.
• BAGGing combines Bootstrapping and Aggregation to form one ensemble model. Given
a sample of data, multiple bootstrapped subsamples are pulled
2. Boosting
• Learners learn sequentially and adaptively to improve model predictions of a learning
algorithm.
BAGGing, or Bootstrap AGGregating

Source: [Link]
 Boosting
Algorithm:
1. Initialise the dataset and assign equal
weight to each of the data point.
2. Provide this as input to the model and
identify the wrongly classified data
points.
3. Increase the weight of the wrongly
classified data points and decrease the
weights of correctly classified data
points. And then normalize the weights
of all data points.
4. if (got required results)
Goto step 5
else
Goto step 2
5. End

Source: [Link]
 Differences Between Bagging and Boosting
Bagging Boosting
The simplest way of combining predictions that belong to A way of combining predictions that belong to the different types.
the same type.
Aim to decrease variance, not bias. Aim to decrease bias, not variance.

Each model receives equal weight. Models are weighted according to their performance.
Each model is built independently. New models are influenced by the performance of previously
built models.
Different training data subsets are selected using row Every new subset contains the elements that were misclassified
sampling with replacement and random sampling methods by previous models.
from the entire training dataset.
Bagging tries to solve the over-fitting problem. Boosting tries to reduce bias.
If the classifier is unstable (high variance), then apply If the classifier is stable and simple (high bias) the apply
bagging. boosting.
In this base classifiers are trained parallelly. In this base classifiers are trained sequentially.
Example: The Random forest model uses Bagging. Example: The AdaBoost uses Boosting techniques

Source: [Link]
 Random Forest – Ensemble Learning
Random forest is a type of supervised machine learning algorithm based on
ensemble learning.

The random forest algorithm combines multiple algorithm of the same type i.e.
multiple decision trees, resulting in a forest of trees, hence the name "Random
Forest".

Random Forest Models can be thought of as BAGGing, with a slight tweak. When
deciding where to split and how to make decisions, BAGGed Decision Trees have
the full disposal of features to choose from.

The random forest algorithm can be used for both regression and classification
tasks.
Random Forest – Ensemble Learning
Rather than just relying on one
Decision Tree and hoping we
made the right decision at each
split, Ensemble Methods allow
us to take a sample of Decision
Trees into account, calculate
which features to use or
questions to ask at each split,
and make a final predictor based
on the aggregated results of the
sampled Decision Trees.
 Random Forest – Ensemble Learning
The following are the basic steps involved in performing the random forest algorithm:

1. Pick N random records from the dataset.

2. Build a decision tree based on these N records.
3. Choose the number of trees you want in your algorithm and repeat steps 1 and
2.
4. In case of a regression problem, for a new record, each tree in the forest predicts
a value for Y (output). The final value can be calculated by taking the average
of all the values predicted by all the trees in forest. Or, in case of a
classification problem, each tree in the forest predicts the category to which
the new record belongs. Finally, the new record is assigned to the category
that wins the majority vote.

Let us test two case studies for regression and classification!

 Random Forest – Ensemble Learning
Pros:
• Random Forest can be used to solve both classification as well as regression
problems.
• Random Forest works well with both categorical and continuous variables.
• Random Forest can automatically handle missing values.
• No feature scaling required.

Cons:
• Complexity: Random Forest creates a lot of trees (unlike only one tree in case
of decision tree) and combines their outputs. It decreases the variance and helps
to avoid overfitting.
• Longer Training Period: Random Forest require much more time to train as
compared to decision trees as it generates a lot of trees (instead of one tree in
case of decision tree) and makes decision on the majority of votes.
4. Support Vector Machine
 Support Vector Machine

“Support Vector Machine” (SVM) is a

supervised machine learning algorithm which
can be used for both classification or
regression challenges. However, it is
mostly used in classification problems.

In the SVM algorithm, we plot each data

item as a point in n-dimensional space
(where n is number of features you have)
with the value of each feature being the value
of a particular coordinate.
 How Does SVM Work

Let’s imagine we have two tags: red and blue,

and our data has two features: x and y.

We want a classifier that, given a pair of (x,y)

coordinates, outputs if it’s either red or blue.

We plot our already labelled training data on a

plane:
 How Does SVM Work
A support vector machine takes these data
points and outputs the hyperplane (which
in two dimensions it’s simply a line) that
best separates the tags.

This line is the decision boundary: anything

that falls to one side of it we will classify as
blue, and anything that falls to the other as
red.
 How Does SVM Work

But, what exactly is the best hyperplane?

For SVM, it’s the one that maximizes the

margins from both tags.

In other words: the hyperplane (remember

it's a line in this case) whose distance to
the nearest element of each tag is the
largest
 Identify the right hyperplane
Identify the right hyper-plane (Scenario-1):
Here, we have three hyper-planes (A, B and C).

Now, identify the right hyper-plane to classify

star and circle.

You need to remember a thumb rule to identify

the right hyper-plane: “Select the hyper-plane
which segregates the two classes better”.

In this scenario, hyper-plane “B” has

excellently performed this job.
 Identify the right hyperplane
Identify the right hyper-plane (Scenario-2):
Here, we have three hyper-planes (A, B and C)
and all are segregating the classes well.

Now, How can we identify the right hyper-

plane?

The margin for hyper-plane C is high as compared to both A and B.

Another lightning reason for selecting the hyper-plane with higher
margin is robustness. If we select a hyper-plane having low margin,
then there is high chance of miss-classification.
 Identify the right hyperplane
Identify the right hyper-plane (Scenario-3):
Hint: Use the rules as discussed in
previous section to identify the right hyper-
plane

Some of you may have selected the hyper-plane B as it has higher margin compared
to A. But, here is the catch, SVM selects the hyper-plane which classifies the classes
accurately prior to maximizing margin. Here, hyper-plane B has a classification error
and A has classified all correctly. Therefore, the right hyper-plane is A.
 Identify the right hyperplane
Can we classify two classes (Scenario-4)?:

Below, I am unable to segregate the two

classes using a straight line, as one of the
stars lies in the territory of other(circle)
class as an outlier.
 Non-linear SVM
Find the hyper-plane to segregate to classes (Scenario-5): In the scenario below,
we can’t have linear hyper-plane between the two classes, so how does SVM
classify these two classes? Till now, we have only looked at the linear hyper-
plane.
 SVM Kernel
SVM algorithms use a set of mathematical functions that are defined as the kernel.
The function of kernel is to take data as input and transform it into the required
form. Different SVM algorithms use different types of kernel functions. These
functions can be different types. For example, linear, nonlinear, polynomial, radial
basis function (RBF), and sigmoid. The most used type of kernel function is RBF.
Because it has localized and finite response along the entire x-axis.

The kernel functions return the inner product between two points in a suitable feature
space. Thus, by defining a notion of similarity, with little computational cost even in
very high-dimensional spaces.

Let us try linear and non-linear examples!

 SVM Kernel
Pros:
• It works really well with a clear margin of separation, and It is effective in high
dimensional spaces.
• It uses a subset of training points in the decision function (called support
vectors), so it is also memory efficient.

Cons:
• It doesn’t perform well when we have large data set because the required
training time is higher.
• It also doesn’t perform very well, when the data set has more noise i.e.
target classes are overlapping.
5. Naïve Bayes
 Naive Bayes
Naive Bayes is a probabilistic machine learning algorithm based on the
Bayes Theorem, used in a wide variety of classification tasks. Typical
applications include filtering spam, classifying documents, sentiment
prediction etc.

It is based on the works of Rev. Thomas Bayes and hence the name.

But why is it called ‘Naive’?

The name naive is used because it assumes the features that go into the
model is independent of each other. That is changing the value of one feature,
does not directly influence or change the value of any of the other features used in
the algorithm.
 Naive Bayes
Bayes theorem provides a way
of calculating the posterior
probability, P(c|x), from P(c),
P(x), and P(x|c).

Naive Bayes classifier assume

that the effect of the value of a
predictor (x) on a given class (c)
is independent of the values of
other predictors. This
assumption is called class
conditional independence.
Naive Bayes
Let us test it on a new set of features (let us call it today):

today = (Sunny, Hot, Normal, False)

So, prediction that golf would be played is ‘Yes’.

 Naive Bayes
In the snapshot of the data below, notice that the data frame has two columns, x
and y. Here, x is the feature and y is the label. We’re going to predict y using x
as an independent variable.

x = glucose and blood pressure

y = diabetes
 Naive Bayes
Pros:
• It is easy and fast to predict class of test data set. It also perform well in multi
class prediction.
• When assumption of independence holds, a Naive Bayes classifier performs
better compared to other models like logistic regression and you need less
training data.

Cons:
• If categorical variable has a category (in test data set), which was not observed
in training data set, then model will assign a 0 (zero) probability and will be
unable to make a prediction. This is often known as “Zero Frequency”.
• Another limitation of Naive Bayes is the assumption of independent
predictors. In real life, it is almost impossible that we get a set of predictors
which are completely independent.
6. K-Nearest Neighbor
 K-Nearest Neighbor
KNN is a non-parametric and lazy learning algorithm. Non-parametric means
there is no assumption for underlying data distribution. In other words, the
model structure determined from the dataset. This will be very helpful in practice
where most of the real-world datasets do not follow mathematical theoretical
assumptions.

Lazy algorithm means it does not need any training data points for model
generation. All training data used in the testing phase. This makes training faster
and testing phase slower and costlier. Costly testing phase means time and
memory.

In the worst case, KNN needs more time to scan all data points and scanning all
data points will require more memory for storing training data.
 How does the KNN algorithm work?

In KNN, K is the number of nearest neighbors.

The number of neighbors is the core deciding
factor.

K is generally an odd number if the number of

classes is 2. When K=1, then the algorithm is
known as the nearest neighbor algorithm. This
is the simplest case.
 How does the KNN algorithm work?
Suppose P1 is the point, for which label
needs to predict. First, you find the k
closest point to P1 and then classify
points by majority vote of its k neighbors.

Each object votes for their class and the

class with the most votes is taken as the
prediction. For finding closest similar points,
you find the distance between points.
KNN has the following basic steps:

1. Calculate distance
2. Find closest neighbors
3. Vote for labels
 KNN Classifier Building
In this dataset, you have two features (weather and temperature) and one
label(play).
# Assigning features and label variables
# First Feature

weather =['Sunny','Sunny','Overcast','Rainy','Rainy','Rainy','Overcast','Sunny','Sunny',
'Rainy','Sunny','Overcast','Overcast','Rainy’]

# Second Feature
temp=['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool','Mild','Mild','Mild','Hot','Mild']

# Label or target variable

play=['No','No','Yes','Yes','Yes','No','Yes','No','Yes','Yes','Yes','Yes','Yes','No']

Let us create KNN classifier in python!

 KNN Classifier
Pros:
• The training phase of K-nearest neighbor classification is much faster compared to
other classification algorithms. There is no need to train a model for generalization, That is
why KNN is known as the simple and instance-based learning algorithm.
• KNN can be useful in case of nonlinear data. It can be used with the regression
problem.

Cons:
• The testing phase of K-nearest neighbor classification is slower and costlier in terms of
time and memory. It requires large memory for storing the entire training dataset for
prediction.
• KNN requires scaling of data because KNN uses the Euclidean distance between two
data points to find nearest neighbors. Euclidean distance is sensitive to magnitudes. The
features with high magnitudes will weight more than features with low magnitudes.
• KNN also not suitable for large dimensional data.
7. Neural Networks
 Artificial Neural Network
• An Artificial Neural Network (ANN) models the relationship
between a set of input signals and an output signal using a model
derived from our understanding of how a biological brain
responds to stimuli from sensory inputs.

• Just as a brain uses a network of interconnected cells called

neurons to create a massive parallel processor, ANN uses a
network of artificial neurons or nodes to solve learning problems.
 Artificial Neural Network
▪ Very loose inspiration: human neurons
 Learning Process
▪ Start with weights = 0
▪ For each training instance:
▪ Classify with current weights

▪ If correct (i.e., y=y*), no change!

▪ If wrong: adjust the weight vector

General Structure of an Artificial Neural Network
 Activation Functions
• The activation function is the mechanism by which the artificial
neuron processes incoming information and passes it
throughout the network.
• Just as the artificial neuron is modeled after the biological version,
so is the activation function modeled after nature's design.
 Hidden Layers
• A neural network with multiple hidden layers is called a Deep
Neural Network (DNN) and the practice of training such network
is sometimes referred to as deep learning.
 Backpropagation Process
• Backpropagation algorithm iterates through many cycles of two
processes.

• Each cycle is known as an epoch. Because the network contains

no a priori (existing) knowledge, the starting weights are
typically set at random.

• Then, the algorithm iterates through the processes, until a

stopping criterion is reached. Each epoch in the backpropagation
algorithm includes two phases:
 Backpropagation Process
• Each epoch in the backpropagation algorithm includes:

1. A forward phase in which the neurons are activated in sequence

from the input layer to the output layer, applying each neuron's
weights and activation function along the way.

2. A backward phase in which the network's output signal resulting

from the forward phase is compared to the true target value in
the training data. The difference between the network's output
signal and the true value results in an error that is propagated
backwards in the network to reduce future errors.
 Low Accuracy in outputs

▪ Noise: if the data isn’t separable,

weights might thrash
▪ Averaging weight vectors over time
can help (averaged perceptron)

▪ Mediocre generalization: finds a

“barely” separating solution

▪ Overtraining: test / held-out

accuracy usually rises, then falls
▪ Overtraining is a kind of overfitting
ANN Pros and Cons
Example: Logical AND
Let us consider the operation AND.
After completing the initialization step, the
perceptron is activated by the sequence of
four input patterns representing an epoch.
The perceptron weights are updated after
each activation.
This process is repeated until all the weights
converge to a uniform set of values.

Forward Propagate (Epoch 1) y

(0)(0.3) + (0)(- 0.1) – 0.2 = 0 + 0 – 0.2 = - 0.2 0
(0)(0.3) + (1)(- 0.1) – 0.2 = 0 + (- 0.1) – 0.2 = - 0.3 0
(1)(0.3) + (0)(- 0.1) – 0.2 = 0.3 + 0 – 0.2 = 0.1 1
(1)(0.2) + (1)(- 0.1) – 0.2 = 0.2 – 0.1 – 0.2 = - 0.1 0
Example: Logical AND
Error = Desired (Yd) – Actual (Y)

If the error, is positive, we need to increase perceptron output Y

(p), but if it is negative, we need to decrease Y(p). If input value
X(p) is positive, an increase in its weight W(p) tends to increase
perceptron output Y(p), whereas if X(p) is negative, an increase
in W(p) tends to decrease Y(p). Thus, Learning rate 0.1 is a
positive constant less than unity.

Adjusted Weights = 0.3 + (0.1 x 1 x (-1)) = 0.3 – 0.1 = 0.2

Epoch 2 y
Example Calculation
Determine hidden layer outputs
Determine hidden layer outputs
Determine hidden layer outputs
 Example: Logical AND
Let Also, the steps in this method are very similar to how Neural Networks learn, which is as
follows;

1. Initialize weight values and bias

2. Forward Propagate
3. Check the error
4. Backpropagate and Adjust weights and bias
5. Repeat for all training examples consider the operation AND.
6. After completing the initialization step, the perceptron is activated by the sequence of
four input patterns representing an epoch.

The perceptron weights are updated after each activation.

This process is repeated until all the weights converge to a uniform set of values.
Example: Python Code for AND Gate
 Chapter 5: Summary
✓ To understand the supervised learning process

✓ To implement the application of supervised machine learning

models:
✓ Regression Model
✓ Decision Trees
✓ Support Vector Machine
✓ Neural Networks
✓ Naive Bayes
✓ Random Forest