Predictive Analytics

Week 1: Introduction to Predictive Modelling

Semester 2, 2021
Discipline of Business Analytics, The University of Sydney Business School
Week 1: Introduction to Predictive Modelling

1. Introduction

2. What is QBUS2820 about?

3. Admin stuff

4. Some key concepts and notations

2
Introduction
A big picture of business analytics process

• ”The world’s most valuable resource is no longer oil, but

data” - The Economist
• It’s perhaps not too exaggerated to say that if a business has
access to the necessary data and is able to convert the data
into business insights, it can conquer the business world.
• Below is Google search trend of the term “Data analytics”.
What does this tell you?

3
A big picture of business analytics process

Start with a business problem thought to be solvable using data

analytics, e.g.,
• Why have the sales decreased in the last two quarters?
• A number of the bank’s ATMs were cash-overloaded, can we
predict the ATM cash-demand better?
• What is the prediction of the volatility of a stock return
tomorrow?
• What are annual predictions of international tourists to NSW
in the next 5 years?

how and where should you start?

Note that not all business problem can be soved using data
analytics, but some can.
4
A big picture of business analytics process

1
1
[Link]
process-overview
5
Business Analytics process

• Business understanding: Know the business context and

what you’re dealing with. Raising the right business question
is as important as finding the solution.
• Data acquisition and understanding: understand the
appropriate data needed for the given business objectives,
how/where to get the proper data
• Modelling: Develop a mathematically sound mechanism
based on which the business objectives set out in the business
understanding phase can be obtained. This is a main, and the
most mathematically challenging step.
• Deployment: puts the data analytic findings to use, the
findings are now handed over to software engineers and
relevant business teams for implementation.
6
Business Analytics process

Some useful tools for Business understanding: The Double

Diamond process, the Five Why technique.

Figure 1: [Link]

7
Business Analytics process

• The Double Diamond process: provides a teamwork platform

for exploring and understanding the business problem in a
creative and innovative way. Each diamond has two phases:
diverging and converging.
• Diverging phase: you try to think out-of-the-box, keep an open
mind, consider anything and everything
• Converging phase: narrow down the ideas and focuses on the
most plausible findings.
• The Five Why technique: ask question ‘Why’ five times:
• Why 1: Why did we underload the bank’s ATM network? A:
We followed the predictive model.
• Why 2: Why did the prediction go wrong? A: We did not
update the model in the last two years.
• Why 3: Why wasn’t the model updated? A: We did not get
new data.
• etc.
8
Business Analytics process

This is a recursive process of learning and discovery. The entire

process might need to be repeated and updated regularly as the
business world is always changing. 9
What are covered in this unit?

• This unit focuses mainly on the Modelling step, with some

demonstration of the entire BA process as much as possible
• HUGE number of data analysis techniques can be useful in
Modelling: inference, prediction, regression, classification,
deep learning, Bayesian computation, etc. This unit only
covers a tiny part of this. Yes, knowledge is infinite!
• More precisely, this unit focuses on Predictive modelling

Predictive modelling is a set of methods for detecting patterns in

data and using these patterns for predicting future data and
informing decision making. It draws on methods from the fields of
statistics, econometrics, and machine learning.

10
What are covered in this unit?

You might become a data-based fortune teller after studying this

course2

2
Picture credit:
[Link]
11
What is QBUS2820 about?
QBUS2820 content structure

1. Statistical Machine Learning foundations and applications:

key concepts in predictive modelling, statistical thinking,
K-nearest neighbours, model evaluation, model selection, and
model inference, etc.

2. Regression: subset selection, ridge regression, LASSO, etc.

3. Classification: key concepts, evaluating classification models,

logistic regression, regularised logistic regression, etc.

4. Forecasting: key concepts, time series, exponential smoothing

and ARIMA models.

12
Learning outcomes

By successfully completing this unit, you are expected to:

1. Understand the conceptual and theoretical foundations of

predictive modelling.

2. Develop an in-depth knowledge of basic methods for

regression, classification, and forecasting methods for business
applications.

3. Be able to conduct a complete data analysis project based on

these foundations and methods.

4. Know how to use Python for your practical workflow under

realistic data complexity (including tasks such as data
manipulation and visualisation).

5. Effectively communicate your results to guide decision making. 13

Comments

• This unit is designed as training for real-world predictive

analytics, which requires a range of skills.

• Practical work in this area involves more than knowing the

methods in the lectures: professionals typically spend a
substantial amount of time on tasks such as data
management, exploratory data analysis, feature engineering,
and implementing methods.

• All of this generally done through coding. Therefore, Python

is your bridge between knowledge and practice.

• For these reasons, please note that this unit requires

independent work and higher than average workload (within
the university guidelines).
14
Admin stuff
Suggested textbooks

ISL An introduction to statistical learning: With applications in R.

by G James et al. Springer-Verlag: easy to read, sloppy
sometimes, come with R/Python code for practice.
ESL The Elements of Statistical Learning by Hastie et al.
[Link]
theory-oriented version of ISL, well-written, deep in theory,
suitable for students with a sound maths background.
FPP Forecasting: principles and practice. Rob J Hyndman and
George Athanasopoulos. Open access book available at
[Link]/fpp2/

15
Teaching Material

• Comprehensive lecture slides will be given weekly

• Information on the Canvas site
[Link]
• Tutorial material
• Some materials I will give during the course which may be of
interest to you

Check the Canvas site at least twice a week! Lectures,

tutorials, related info are constantly posted there.

16
Notes

• Statistical terminologies are in bold or blue when first

introduced, e.g., autoregression, recurrent neural network
• When there are something needed your attention, they are put
in italic or red, e.g., This part will be tested in the exam
• This is an applied course - we focus more on applications,
explaining how the methods work, how to use them, their
advantages and limits, etc. Theoretical part and maths are
also sometimes discussed briefly.
• Slides with “*” are rather technical. You are strongly
encouraged to go through them, but they are not tested in the
exams.

17
The teaching team

• Lecturer: Minh-Ngoc Tran

If you find it difficult to say my name, check my webpage:
[Link]
Email: [Link]@[Link].
• Tutors:
• Mr. Wai Yat Wong
• Miss Emma Nguyen
• See Canvas for the tutors’ details, tutorial timetable,
consultation times, etc

18
Assessments

Note: the due date and time are communicated via Canvas.
19
Plagiarism

• Be careful to avoid Plagiarism

• What, Why and How: [Link]
[Link]
• Plagiarism means presenting work that is not your own
without acknowledging the original source of the work. It
doesn’t matter whether you do this on purpose or accidentally.
• Never copy other stuff
• Turnitin system compares your assignment reports with other
assignments, reports, journals etc (almost openly available
sources)
• Do not use ghost writing! Expecting a severe punishment if
you hire a ghostwriter to do your assignments

20
Communication

• The best option is to talk to the teaching team during or after

classes, at consultation times
• Using emails is also an option
• Only send emails from your USyd official email address, don’t
send emails from your work email addresses or other personal
email addresses
• Always mention your full name and student ID

21
The Software for the Unit

• Python
• Free and Works on PCs, Mac, Unix/Linux
• Does statistical modelling, visualisation and programming
environment
• Can be used for almost all models to be discussed in this class
• Matlab and R are also very good for data analysis, but not
used in this unit.
• Excel/SPSS are not enough to complete most data analysis
tasks in this course

22
Some Tips

• Learning by doing
• You become a better problem solver by solving problems.
Focus your efforts on the assignments and tutorials.
• Ask questions in the lectures, consultation times, and by
e-mail.
• Discuss the materials with your classmates/colleagues. Ask
for help.
• Look for answers and extra readings on the Internet, especially
about programming issues.
• We focus on technical materials, but remember the profile of
a good business analyst: common sense, creation,
communication and data visualisation skills are very
important.
23
Let’s start our predictive analytics
journey...

24
Some key concepts and notations
Predictive modelling

Two trends bring predictive modelling to the forefront of successful

business decision making:

• We are in the era of big data. The Internet and increasing

presence of data capturing devices (such as mobile phones,
cameras, sensors, card readers, etc), combined with large
reductions in the cost of storage, brought an unprecedented
availability of data, and continued dramatic growth in the size
of data sets.

• Advancing computing power increases the scope for exploring

complex patterns in data. Quantum computers are about to
be around - would change our life forever!

25
Types of prediction

Different types of data lead to different types of prediction

problems:

• In cross-sectional prediction, we work with data all collected

at a particular time point. E.g., for house value evaluation, we
observe several variables such as selling price, location,
number of rooms, size, etc. for many sold houses across
Sydney, at a particular time point (such as 2021). Our task is
to predict the value of a house to be put on the market.

• In time series forecasting, we want to predict the value of a

variable at a specific time point in the future, based on past
and current information. Here, time at which we observed the
past data, and at which we want to forecast, is important.

26
Types of learning

• Supervised learning
• Unsupervised learning

27
Supervised learning

In the context of statistical learning, supervised learning is the

task of learning a function to predict an output variable Y based
on observed input variables x1 , . . . , xp . We develop methods that
learn this function based on labelled data {(xi , yi )}Ni=1 , which we
call the training data.

28
Supervised learning

In supervised learning, the output or response variable can be of

any type. We will study methods that address two main classes of
supervised learning problems:

• In regression, the response is a quantitative scalar (such as

the income of a worker).

• In classification, the response is nominal or categorical

variable Y ∈ {1, . . . , C}, where C is the number of classes.
When C = 2, this is called binary classification; if C > 2, this
is called multiclass classification. The response can be ordinal
- but not covered in this unit.

29
Example: handwritten digit recognition

A view of the MNIST dataset.

30
Example: Customer relationship management

• Customer relationship management (CRM) is a set of

practices that involve collecting and studying customer
information with the objective of maximising customer
lifetime value (CLV), the net value of a customer to a firm
over his/her entire lifetime.

• CRM may be part of a customer-centric (as opposed to

brand-centric) business strategy, which focuses on customer
satisfaction and loyalty towards the acquisition and retention
of profitable customers.

• CRM has four main areas: customer acquisition, retention,

churn, and win-back. Statistical models and machine learning
algorithms play a central role in in each of these areas.

31
Customer relationship management

The data is from Kumar and Petersen (2012), and refers to

corporate clients.

32
Customer relationship management

Kumar and Petersen (2012) estimate a model to predict the

response

1 if the customer was acquired,
Y =
0 if the customer was not acquired,

based on predictors such as the dollar spent on marketing efforts to

acquire the prospect, and characteristics of the prospect’s firm
such as industry, revenue, and number employees.

This is a binary classification problem.

33
Unsupervised learning

Unsupervised Learning: There are no input or output variables.

The data are unlabelled. Learning tasks are to uncover hidden
patterns, clusters, relationships between the variables or
distribution of the data. Examples of unsupervised learning
techniques:

• Clustering (also known as segmentation): cluster data into

groups with similar characteristics
• Principle Component Analysis: find a few key factors that
determine the data patterns

34
Notation

• We use upper case letters such as Y to denote random

variables, regardless of dimension.

• Lower case letters denote observed values. For example, y

denotes the realised value of the random variable Y .

• We use i to index the observations, j to index the inputs. For

example, yi is the observed response for sample i, while xij is
the value of predictor j for observation i.

• We use the hat notation (e.g. β)

b for estimators and
estimates. The notation may not distinguish between the two
(refer to context).

• Vectors are in lower case bold letters. Matrices are in upper

case bold letters.
35
Vector and matrix notation

Response vector:

 
y1
 
 y2 
y= . 
 
 .. 
 
yN

Review the provided materials of liner algebra.

36
Vector and matrix notation

Vector of predictor (features, attributes, covariates, regressors,

independent variables) values for observation i:
 
xi1
 
xi2 
xi =  . 
 
 .. 
 
xip

Vector of observed values for predictor j:

 
x1j
 
 x2j 
j
x = . 
 
 .. 
 
xN j

37
Vector and matrix notation

Design matrix:
 
x11 x12 ... x1p
 
 x21 x22 ... x2p 
X= .. .. .. ..
 
. . . .

 
 
xN 1 xN 2 . . . xN p

38
Prediction

We define prediction as follows:

1. Train a predictive function fb(x) using data D = {(yi , xi )}N

i=1 .

2. Upon observing a new input point x0 , make the prediction

fb(x0 ), the predictive function evaluated at x0 .

How should we perform this prediction task? How do we define our

objective? How do we measure success in achieving this objective?
To answer these questions, we turn to decision theory. We mostly
focus on regression problems for simplicity.

39
Loss function

A loss function or cost function L(y, yb) measures the cost of

prediction yb when the truth is y. The most common loss function
for regression is the squared loss:

L(y, yb) = (y − yb)2

For binary classification, a typical loss function is the 0-1 loss:


1 if y ̸= yb
L(y, yb) =
0 if y = yb.

40
Expected loss

Let Y and X have a joint probability distribution P (X, Y ), and

f (X) be a prediction of Y . The idea of decision theory is that we
take the action that minimises our expected loss or expected
error/risk:
R(f ) = E [L(Y, f (X))] ,

where the expectation is over P (X, Y ). Here, the risk is for a

given function f (·).

We can use the law of iterated expectations to rewrite the

expected loss as
h i
R(f ) = E E (Y − f (X))2 |X .

41
Optimal prediction

The optimal action is to choose the prediction function δ(·) that

minimises the expected loss. This is equivalent to minimising the
expected loss at every input point x:

δ(x) = argmin E(L(Y, f (x))|X = x)

f (·)

For the squared loss, the solution is the conditional expectation:

δ(x) = E(Y |X = x)

Concept: under the squared error loss, the optimal prediction of Y

at any point X = x is the conditional mean E(Y |X = x).
42
Statistical modelling

• Our regression problem reduces to the estimation of the

conditional expectation function E(Y |X = x) (often
unknown). In order to learn this function, we need to
introduce assumptions.

• Assumptions lead to statistical models.

• For example, the linear regression model assumes that

E(Y |X = x) is linear in x:

E(Y |X = x) = xT β

43
Additive error model

The additive error model is our basic general model for

regression. It assumes that the relationship between Y and X is
described as
Y = f (X) + ε,

where f (·) is an unknown regression function, and ε is a random

error with mean zero (E(ε) = 0).

Under this model,

E(Y |X = x) = E(f (x) + ε) = f (x),

since E(ε) = 0.

44
Example: linear regression

In the special case of the linear regression model, we assume that

f (X) = β0 + β1 X1 + . . . + βp Xp ,

leading to the model

Y = β0 + β1 X1 + . . . + βp Xp + ε,

and predictions

fb(x) = βb0 + βb1 x1 + . . . + βbp xp ,

where βb = (βb0 , βb1 , . . . , βbp ) is the vector of least squares estimates

of the model parameters.

45
Statistical decision theory

Our discussion of statistical decision theory lays the foundation for

the model selection and model assessment discussed later in the
course:

• Evaluating model performance: estimating the expected loss

of a trained model.

• Choosing a learning method/model: finding and estimating an

appropriate model such that we minimise our expected loss.

46
Other key concepts and themes

• Underfitting and Overfitting.

• Parametric vs non-parametric models.

• No-free lunch theorem.

• Accuracy vs interpretability.

47
Overfitting

• We say that there is overfitting when an estimated model is

excessively flexible, incorporating minor variations in the
training data that are likely to be noise rather than underlying
patterns.

• An overfit model has small training errors, but may predict

poorly. In essence, it has memorised the training set and
cannot generalise well to the unseen data.

• Overfitting can explained theoretically using bias-variance

decomposition, which will be discussed later.

• Techniques dealing with overfitting will be discussed

throughout the course.

48
Parametric vs nonparametric methods

• In parametric methods: we first assume a parametric form

for the underlying model that generated the data, then
estimate the parameters. The underlying model that
generated the data is described by a functional form that
depends on a vector of unknown parameters θ.

• E.g., simple linear regression

yi = β0 + β1 xi + ϵi , ϵi ∼ N (0, σ 2 )

is a parametric model as we assume the model that generated

data yi , given xi , is normal distribution with mean β0 + β1 xi
and variance σ 2 . The set of unknown parameters is
θ = (β0 , β1 , σ 2 ).

49
Parametric vs nonparametric methods

• In nonparametric methods, we don’t assume a functional

form for the underlying model that generated the data. An
example is the K-nearest neighbours method, which we will
study later.

• Parametric methods are faster to use, and more interpretable,

but have the disadvantage of making stronger assumptions
about the data.

• Nonparametric methods are more flexible, but have larger

variance and can be computationally infeasible for large
datasets.

50
No free lunch theorem

All models are wrong, but some are useful. – George Box

• The field of machine learning proposes a large range of models

and algorithms to solve supervised and unsupervised learning
problems.

• However, there is no single model or approach that works

optimally for all problems. This is sometimes called the no
free lunch theorem.

• Therefore, applied statistical learning requires awareness of

speed-accuracy-complexity trade-offs and data-driven
consideration of different approaches for every problem.

51
Prediction accuracy vs interpretability

Particularly in business/economic applications, interpretability is an

important consideration in addition to prediction accuracy. Highly
flexible, nonparametric methods, tend to be less interpretable than
simpler methods. High

Subset Selection
Lasso

Least Squares
Interpretability

Generalized Additive Models

Trees

Bagging, Boosting

Support Vector Machines

Low

Low High

Flexibility

52
Study guide

• Recall three important concepts from these slides, and explain

them in your own words.

• Use the review questions in the next slide to self-test on key

concepts.

• Study the mathematical details in the module notes.

• Study (or revise) Chapters 1 and 2 of ISL. Reader Chapter 3

before the next lecture.

53
Review questions (1/2)

• What is predictive modelling?

• What is the difference between cross-sectional prediction and

time-series forecasting?

• What is supervised learning?

• What is a loss function?

• What do we learn from statistical decision theory for

regression problems?

54
Review questions (2/2)

• How do we evaluate model performance with data?

• What is overfitting?

• What is the difference between parametric and nonparametric

models? What are the advantages and disadvantages of each
approach?

55