Machine Learning

An introduction
Dr Yeo Wee Kiang

Unauthorized distribution or sharing of these materials is strictly prohibited.

1
 IS5126 Hands-on with Applied Analytics

Overview
• What is machine learning? • Supervised Learning workflow
• Applications of ML • Steps involved
• Model Validation or Evaluation
• The ML Process
• Train, validation, test split
• The different stages • Train, test split
• Types of ML • k-fold Cross Validation
• Supervised
• Confusion Matrix
• Unsupervised
• Semi-supervised • Types of errors
• Supervised Learning • Metrics for evaluating classification models
• Classification • Common issues in ML
• Regression • Imbalanced datasets
• Unsupervised Learning • ‘Dirty’ data
• Applications • Irrelevant data
• Clustering techniques • Lack of relevant data
Unauthorized distribution or sharing of these materials is strictly prohibited.
2
Machine Learning
What is it?

Machine Learning is the science of getting computers to learn

patterns and trends and improve their learning over time in an
autonomous and iterative fashion, by providing them with data
and information in the form of observations and real-world
interactions.
• It is a method of analysing data that automates model building.
• It allows computers to find hidden insights without being explicitly
programmed step-by-step.

Unauthorized distribution or sharing of these materials is strictly prohibited.

3
Applications of Machine Learning
• Fraud detection
• Product recommendations
• Natural Language Processing
• Predicting customer or employee churn
• Customer segmentation
• Image recognition and object detection
• New Pricing Models
• Financial Modelling
Unauthorized distribution or sharing of these materials is strictly prohibited.
4
Machine Learning Process
The different stages

Unauthorized distribution or sharing of these materials is strictly prohibited.

5
Types of Machine Learning
• Supervised Learning
Supervised • Supervised learning is the machine learning
Learning
All data are labelled Model
task of learning a function that maps an input
to an output based on examples of input-
output pairs.
Small portion of data • Semi-supervised Learning
are labelled • Semi-supervised learning is a type of machine
Semi-supervised Model learning that uses both labeled and unlabeled
Learning data to train a model. It can be used to
Large portion of data improve the performance of a machine
learning model when there is limited labeled
are unlabelled data available.
• Unsupervised Learning
Unsupervised All data are • Unsupervised learning is a type of machine
Model learning algorithm used to draw inferences
Learning unlabelled from datasets consisting of input data without
labelled responses.

Unauthorized distribution or sharing of these materials is strictly prohibited.

6
Supervised Learning
Two types

Supervised Learning

Classification Regression
Unauthorized distribution or sharing of these materials is strictly prohibited.
7
Supervised Learning
What is the supervisor?

• Datasets suitable for

supervised learning contain
known class labels.
• Datasets without known
class labels can only be
used for unsupervised
learning.

Unauthorized distribution or sharing of these materials is strictly prohibited.

8
Classification versus Regression
Supervised Learning

• Classification is used when

the prediction outcome is
categorical.
• e.g. cats or dogs.
• Regression is used when the
prediction outcome is
numerical.
• e.g. stock prices, temperature.

Unauthorized distribution or sharing of these materials is strictly prohibited.

9
Unsupervised Learning
For data without class labels

• Unsupervised learning does Supervised Learning

Unsupervised
Learning
not require labelled data.
Use of
Relies on labelled data Lacks predefined
• It discovers patterns and Predefined
with predefined target target labels; operates
structures in data Target
outcomes. solely with the data.
Labels
independently.
Algorithms
Algorithms are guided
• Key differences from Learning
by labeled examples
autonomously identify
supervised learning: Guidance
during training.
patterns and
structures in data.
• No predefined target labels.
Primarily focuses on
• Learns data patterns without Focus and specific prediction More exploratory and
explicit supervision. open-ended, revealing
Objective tasks with known
insights within data.
labels.
Unauthorized distribution or sharing of these materials is strictly prohibited. 10
Unsupervised Learning
For data without class labels

• Applications of unsupervised learning:

• Clustering Customer Segmentation:
• Grouping customers based on their purchasing behaviour for targeted
marketing.
• Anomaly Detection in Network Security:
• Identifying unusual patterns in network traffic to detect cyber threats.
• Healthcare Data Analysis:
• Identifying patient groups with similar medical histories for personalized
treatment plans.

Unauthorized distribution or sharing of these materials is strictly prohibited. 11

Unsupervised Learning
For data without class labels

Clustering is a fundamental unsupervised learning technique.

• Clustering groups similar data points together.

Unauthorized distribution or sharing of these materials is strictly prohibited. 12

Unsupervised Learning
For data without class labels

• Clustering techniques:
• K-Means Clustering
• Partitions data into K clusters based on similarity.
• Minimizes intra-cluster distance and maximizes inter-
cluster distance.
• Good for initial clustering exploration.
• Hierarchical Clustering
• Builds a dendrogram to represent the hierarchy of
clusters.
• Starts with individual data points and merges them.
• Reveals relationships between clusters at different
levels.
[Link]
[Link]
Unauthorized distribution or sharing of these materials is strictly prohibited. 13
Supervised Learning Workflow
The different stages

Unauthorized distribution or sharing of these materials is strictly prohibited.

14
Model Validation or Evaluation
Supervised Learning

Model Validation

Train, Validation, k-fold cross

Train, Test split
Test split validation

Model validation or evaluation is the process of assessing the performance of a

trained machine learning model on a held-out dataset. This is important
because it helps us to understand how well the model will generalize to new
data.
Unauthorized distribution or sharing of these materials is strictly prohibited.
15
Model Validation or Evaluation
Supervised Learning

Train, Validation, Test Split Train Validation Test

• One common way to validate a model is to split the dataset into three parts: train,
validation, and test. The train set is used to train the model, the validation set is used
to tune the model hyperparameters, and the test set is used to evaluate the final
model.
• We need to randomly split the rows in the original dataset into subsets. Typically, the
train set comprises about approximately 70% of the original number of rows. The
remainder rows are split randomly between the validation set and test set.
Unauthorized distribution or sharing of these materials is strictly prohibited.
16
 Model Validation or Evaluation
Supervised Learning

Train,
Validation,
Test Split

Unauthorized distribution or sharing of these materials is strictly prohibited.

17
Model Validation or Evaluation
Supervised Learning Train Test

Train, Test Split

• Another common way to validate a model is to split the dataset into two parts:
train and test. The train set is used to train the model, and the test set is used
to evaluate the final model. This approach is simpler than the train, validation,
test split, but it can be less reliable, especially for small datasets.
• We randomly split the rows in the original dataset into train set and test set.
Typically, the training set comprises approximately 70% of the original number
of rows. The remainder rows will become the test set.
Unauthorized distribution or sharing of these materials is strictly prohibited.
18
k-fold Cross Validation
Supervised Learning
K-fold cross-validation is a more
robust method for model
validation.
k=5
• In k-fold cross-validation, the
dataset is split into k folds.
• For each fold, the model is
trained on the k-1 remaining
folds and evaluated on the held-
out fold.
• This process is repeated for all k
folds.
• The average performance of the
model across all folds is used as
the final evaluation metric. [Link]

Unauthorized distribution or sharing of these materials is strictly prohibited.

19
Confusion Matrix
Model Validation or Evaluation for Supervised Learning
Prediction: Class 1 Prediction: Class 2
True Positive (TP) False Negative (FN)
Actual: Class 1 👍👍 Type 2 Error

Actual: Class 2 False Positive (FP) True Negative (TN)

Type 1 Error 👍👍
A confusion matrix is a table that summarizes the performance of a classification
model on a test set.
• Each column of the confusion matrix represents a predicted class, and each row
represents a true class.
• The diagonal elements of the confusion matrix represent the number of correctly
classified samples,
• The off-diagonal elements represent the number of incorrectly classified samples.
Unauthorized distribution or sharing of these materials is strictly prohibited.
20
Confusion Matrix
Model Validation or Evaluation for Supervised Learning
Prediction: Class 1 Prediction: Class 2
True Positive (TP) False Negative (FN)
Actual: Class 1 👍👍 Type 2 Error

Actual: Class 2 False Positive (FP) True Negative (TN)

Type 1 Error 👍👍

There are two main types of error in classification models:

• False positives (Type 1 error) are samples that are predicted to belong to a class
but do not actually belong to that class.
• False negatives (Type 2 error) are samples that are not predicted to belong to a
class but actually do belong to that class.

Unauthorized distribution or sharing of these materials is strictly prohibited.

21
Types of Errors
Model Validation or Evaluation for Supervised Learning

[Link]

Unauthorized distribution or sharing of these materials is strictly prohibited.

22
Metrics
Model Validation or Evaluation for Supervised Learning

Metrics for evaluating Metrics for evaluating Regression

Classification models models
• Accuracy • Mean squared error (MSE)
• Precision • Root mean squared error (RMSE)
• Recall • Mean absolute error (MAE)
• F1 score • R-squared

Unauthorized distribution or sharing of these materials is strictly prohibited.

23
Metrics
Model Validation or Evaluation for Supervised Learning

There are many different metrics that can be used to evaluate classification
models. Some common metrics include:
• Accuracy is the percentage of samples that are correctly classified.
• Precision is the percentage of predicted positive samples that are actually
positive. Precision is a good metric for evaluating models when the cost of
false positives is high.
• Recall is the percentage of actual positive samples that are predicted to be
positive. Recall is a good metric for evaluating models when the cost of false
negatives is high.
• The F1 score is a harmonic mean of precision and recall. It is a good metric for
evaluating models when both precision and recall are important.
Unauthorized distribution or sharing of these materials is strictly prohibited.
24
Metrics
Model Validation or Evaluation for Supervised Learning

More about precision and recall:

• Precision is the fraction of predicted positives that are actually positive.
• High precision means that the model is very accurate at predicting positive
samples. However, it may not be very complete at identifying all positive samples.
• For example, a spam filter with high precision will correctly identify most spam
emails, but it may also miss some spam emails.
• Recall is the fraction of actual positives that are predicted positive.
• High recall means that the model is very complete at identifying all positive
samples. However, it may not be very accurate at predicting positive samples.
• For example, a spam filter with high recall will identify all spam emails, but it may
also flag some legitimate emails as spam.

Unauthorized distribution or sharing of these materials is strictly prohibited.

25
Metrics
Model Validation or Evaluation for Supervised Learning

• Accuracy is the most intuitive performance measure, and it is simply a

ratio of correctly predicted observation to the total observations.
• One may think that, if we have high accuracy then our model is best. For example,
if we got 0.932, it means that our model is approximately 93% accurate. Yes,
accuracy is a great measure but only when you have balanced datasets where
you have an even class distribution.
• Most of the time, you must look at other metrics to evaluate the performance of
your model since class distribution is usually uneven or imbalanced.
• Accuracy works best if false positives and false negatives have similar cost. If the
cost of false positives and false negatives are very different, it’s better to look at
both Precision and Recall.

Unauthorized distribution or sharing of these materials is strictly prohibited.

26
Metrics
Model Validation or Evaluation for Supervised Learning

• F1 score: 2 * Precision * Recall/(Precision + Recall)

• F1 Score is the harmonic mean of the precision and recall of a model.
• Therefore, this score takes both false positives and false negatives into account.
Intuitively it is not as easy to understand as accuracy, but F1 is usually more
useful than accuracy, especially if you have an uneven class distribution.
• A high F1 score indicates that the model is performing well on both precision and
recall.
• A low F1 score indicates that the model is
• not performing well on either precision or recall,
• or that it is not performing well on both precision and recall.

[Link]
Unauthorized distribution or sharing of these materials is strictly prohibited.
27
Metrics
Model Validation or Evaluation for Supervised Learning

Prediction: Class 1 Prediction: Class 2

Actual: Class 1 True Positive (TP) False Negative (FN)
Actual: Class 2 False Positive (FP) True Negative (TN)

We have the following formulae:

• Accuracy = (TP + TN)/(TP+TN+FP+FN)
• Precision = TP/(TP+FP)
• Recall or Sensitivity = TP/(TP+FN)
• F1 score = 2 * Precision * Recall/(Precision + Recall)

Unauthorized distribution or sharing of these materials is strictly prohibited.

28
Confusion Matrix
Example: 2-class classification

Prediction: Class 1 Prediction: Class 2

Actual: Class 1 320 43
Actual: Class 2 20 538

• Accuracy = (TP + TN)/(TP+TN+FP+FN) = (320+538)/(320+538+20+43) = 0.932

• Precision = TP/(TP+FP) = 320/(320+20) = 0.941
• Recall or Sensitivity = TP/(TP+FN) = 320/(320+43) = 0.882
• F1 score = 2 * Precision * Recall/(Precision + Recall) = 2*0.941*0.882/(0.941+0.882) = 0.911

Unauthorized distribution or sharing of these materials is strictly prohibited.

29
Common issues in Machine Learning
• Imbalanced datasets
• This happens when the number of data points for a particular class label is very
much larger than the number of data points for other class labels.
• Class distribution is uneven.
• ‘Dirty’ data
• Any dataset probably needs some level of cleaning and pre-processing.
• Using raw data without any prior cleaning is not advisable.
• Irrelevant data or lack of relevant data
• The data points (rows) and features (columns) in your dataset do not support the
problem statement that you are trying to answer or solve.

Unauthorized distribution or sharing of these materials is strictly prohibited.

30