Applied Machine Learning
Topic 1: Introduction to Machine Learning
AP Engelbrecht
Department of Industrial Engineering, and
Division of Computer Science
Stellenbosch University
South Africa
engel@[Link], [Link]
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 1 / 17
Topic 1: Introduction to Machine Learning
Outline
What is machine learning?
Generic machine learning system
Use cases for machine learning
Machine learning paradigms
Machine learning approaches
Machine learning pipeline
Performance versus complexity
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 2 / 17
Topic 1: Introduction to Machine Learning
Reading Material
Read chapter 1 of Fundamentals of Machine Learning for Predictive
Data Analytics: Algorithms, Worked Examples, and Case Studies,
John D. Kelleher, Brian Mac Namee, Aoife D’Arcy, The MIT Press,
2015
In addition, google the terms used in this topic and read what you get
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 3 / 17
Introduction to Machine Learning
What is Machine Learning?
Machine learning (ML) is an application of artificial intelligence (AI) that
provides systems the ability to automatically learn and improve from
experience without being explicitly programmed
Machine learning algorithms build a mathematical model based on
sample data, known as training data, in order to make predictions or
decisions without being explicitly programmed to perform the task
The most important ingredient of machine learning algorithms: Data,
and enough of it, and of sufficient quality
NB: Garbage in, garbage out
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 4 / 17
Introduction to Machine Learning
Generic Supervised Machine Learning System
Input variables:
x = (x1 , x2 , . . . , xN )
Hidden variables:
h = (h1 , h2 , . . . , hK )
Output variables:
y = (y1 , y2 , . . . , yM )
f : XN → YM
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 5 / 17
Introduction to Machine Learning
Use Cases for Machine Learning
Problem Description Example Use Cases
Type
Classification Determine discrete class Spam filtering, sentiment
of data pattern analysis, diagnosis, fraud
detection, risk prediction
Regression Predict real-valued value Stock-market prediction,
Forecasting of output variable forecasting, price estima-
tion, life-time value predic-
tion
Recommen- Predicting alternatives Product recommendation,
dation online dating, customer
profiling
Imputation Infer values of missing in- Deactivated sensors,
put data missing customer data
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 6 / 17
Introduction to Machine Learning
Machine Learning Paradigms
Different classes of learning algorithms exist, depending on the
availability of data labels (targets) and feedback signals
Supervised learning:
Learning by being taught
Labeled data: target output values are provided for each data
pattern
Problem types: Classification, regression, forecasting
Unsupervised learning
Unlabelled data: No target information provided
Discover hidden patterns and relationships among input variables
Problem types: Recommendation, imputation, clustering,
classification (?)
Reinforcement learning
Learning from consequences of actions
Also called trial and error learning
Learning via rewards and penalties
Hybrid learning models
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 7 / 17
Introduction to Machine Learning
Machine Learning Paradigms (cont)
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 8 / 17
Introduction to Machine Learning
Machine Learning Approaches
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 9 / 17
Introduction to Machine Learning
How to Choose Which Machine Learning Algorithm to Use?
Machine learning algorithm selection is very much based on answers
to the following questions:
How much data do you have?
What are the data types of the input variables?
Are there output variables/targets?
What are the data types of the output variables?
Is it classification or regression?
How many classes do you have?
Do you have skew class distributions?
Accuracy versus training speed?
Results interpretation, easy or hard?
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 10 / 17
Introduction to Machine Learning
Machine Learning Pipeline
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 11 / 17
Introduction to Machine Learning
Performance versus Complexity
Underfitting:
Model is too simple to learn the complex mapping/relationships
Poor predictive accuracy on both the training and test sets, bad
generalization
Overfitting:
Model has too many free parameters, is too complex
Learns noise in the training data
Good training predictive accuracy, but poor generalization
Need to achieve a good balance between model complexity and
generalization performance
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 12 / 17
Introduction to Machine Learning
Performance versus Complexity (cont)
Regression
Classification
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 13 / 17
Introduction to Machine Learning
Bias-Variance Dilemma
Bias Variance
Error from erroneous assump- Error from sensitivity to small
tions in the learning algorithm fluctuations in the training set
Model predicted values are far How scattered predicted outputs
from target values are for similar inputs
High bias can cause relevant re- High variance can cause an al-
lations between features and tar- gorithm to model the random
gets to be missed noise in the training data
Underfitting Overfitting
Goal: minimize Error = Bias + Variance
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 14 / 17
Introduction to Machine Learning
Bias-Variance Dilemma (cont)
High bias/Low variance Low bias/High variance
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 15 / 17
Introduction to Machine Learning
Data Sets Construction
To construct a machine learning system, a number of data sets are
needed:
Training set (DT ), containing the bulk of the data patterns, used to
construct the machine learning model
Validation set (DV ), used during training to validate the model
Test/Generalization set (DG ), used to quantify generalization
ability after training
Important to note:
If data set is stationary, original data set randomly divided into
these three sets such that
DT ∩ DV = ∅
DT ∩ DG = ∅
DV ∩ DG = ∅
|DT | : |DV | : |DG | ratio? Usually, 60%:20%:20%
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 16 / 17
Introduction to Machine Learning
Data Sets Construction (cont)
If the data set is temporal, then
Split D into two sets in, e.g. a ratio of 80%:20%
From the large set, randomly select 10% to be in DV and 10% to be
in DG
Split the small set randomly in equal parts to be added to DV and
DG
The patterns from the large set in DG is used to quantify
interpolation ability
The patterns from the small set in DG is used to quantify
extrapolation ability
Engelbrecht Applied Machine Learning Stellenbosch University, 2024 17 / 17