Introduction to Data Science: Data preprocessing

Part 3: A primer on statistical learning

Daniël Linders

University of Amsterdam
0 – Outline 2/67

1. What is Statistical Learning?

2. Types of learning problems

3. A primer to statistical learning

4. Training, validation and testing of a model

5. Statistical learning, predictive analytics and data science

1 – What is Statistical Learning? 3/67

Statistical learning:
I use statistical methods to learn about relationships between variables;
I and to improve predictions;
I using a data set with (historical) observations.
Examples:
I Use budgets spend on TV, radio, newspaper to predict future sales.
I Use information about an insured to predict future claims.
Data is employed to learn about existing relationships:
I a clever statistical method can discover hidden relationships;
I and can learn and improve when new data is available.
1 – What is statistical learning? 4/67
Response and predictor values

Response variable Y:
I the variable of interest we want to predict or investigate.
I Example: number of claims, risk class, price, churn, etc.
Predictor variables X1 , X2 , . . . , Xp :
I the predictor values may1 help to estimate Y.
I Notation: X = X1 , X2 , . . . , Xp .

I Example: age, sex, location, health status, past experience, volatility,

etc.

1 hopefully...
1 – What is statistical learning? 5/67
Example: risk classification

Car insurance data.

Response variable: risk class of the driver.
I Assume drivers are classified in different risk classes.
I Two risk classes: Good and bad drivers
Questions: how to determine the risk class...
I of an existing policyholder?
F The behavior of the driver can change over time.
F We may have assigned the wrong class in the past and new data
suggest we have to update.
I of a new policyholder?
F Little information is available.
1 – What is statistical learning? 6/67
Example: risk classification

Problem:
I the risk class is random and unpredictable;
I we have to guess.
Good news: we can try to improve the guessing...
I by using predictor variables like age, sex, claims history, etc;
I by investigating if and how these predictor variables are related with
the response;
I by using available data.
1 – What is statistical learning? 7/67
Example: risk classification

Goal:
I Make the guessing more scientific.
Remaining questions:
I how reliable is my outcome?
I what happens if my guess is wrong?
1 – What is statistical learning? 8/67
The regression function f

The relationship between the response Y and the predictors X:

Y = f (X) + e.

f : Regression function:
I captures the systematic information that is contained in the predictor
variables about the response variable.
e is an error term:
I unsystematic information;
I not predictable.
I E[e] = 0.
1 – What is statistical learning? 9/67
The regression function f

Statistical learning:
I approaches for estimating the function f ;
I using available data.
Why estimate f ?
I Prediction:
F Predict values of the response variable for new explanatory variables.
I Inference:
F Investigate relations between explanatory variables and response
variables.
2 – Types of learning problems 10/67

There are different types of learning problems:

I we group types of problems and types of learning algorithms;
I and consider solutions for the different groups.
Examples of learning problems:
I Predict the future claim amounts.
I Is this company likely to default?
I Do our customers form natural groups?
I Do we need to use all 100 variables in our data set?
2 – Types of learning problems 11/67
Supervised vs. Unsupervised

Supervised learning problems:

I The response variable Y supervises/teaches the algorithm.
F We can measure the error of the algorithm.
F We can optimize the algorithm by minimizing the error.
I The predictor variables xi are used to investigate the response variable.
I Popular algorithms: Regression, decision trees.
Unsupervised learning problems;
I No response variable to supervise/teach the algorithm.
I Learn from the data and find structure in the data, without predicting
a variable.
I Popular algorithm: Clustering, PCA.
2 – Types of learning problems 12/67
Supervised vs. Unsupervised

Dimensionality reduction:
I Consider a large data set with a large amount of different variables.
I Problem: Find a subset of the data which can represent the original
data.
Supervised dimensionality reduction:
I Use less predictor variables to estimate the response.
Unsupervised dimensionality reduction:
I Reduce the number of variables in the data set.
2 – Types of learning problems 13/67
Regression vs. Classification

Regression problems:
I The response variable is quantitative.
I Quantitative variables: The outcomes can be ordered from small to
large.
I Example: Number of claims, volatility of a stocks.
Classification problems:
I The response variable belongs to a class.
I There is no ordering relation between the different classes.
I Example: good/bad driver, Default/no default.
Several learning techniques can be used for both problems:
I K-nearest neighbor, decision trees, etc.
2 – Types of learning problems 14/67
Schematic overview of the main types of learning problems

Learning Problem

Supervised Unsupervised

Dimensionality
Regression Reduction Clustering

Classification
3 – A primer to statistical learning 15/67
The problem

Estimated regression function f̂ .

I real (unknown) regression function f ;
I we hope that f̂ ≈ f .
Assume for the moment:
I Y is a real-valued random variable.
I X is a random vector.
Goal: estimate the unknown response Y by ŷ:

ŷ = f̂ (X) .

I If X is known, ŷ is a real number: ŷ ∈ R.

3 – A primer to statistical learning 16/67
Optimal approximation of f

The approximation f̂ is accurate if ....

f̂ (X) is ‘close’ to Y.

In order to assess the quality of our prediction f̂ (X) ...

we need a measure for ‘closeness’.
3 – A primer to statistical learning 17/67
Optimal approximation of f

Examples for regression problems:

I Squared Error:
 2
Y − f̂ (X) .
I Absolute Error:
Y − f̂ (X) .

Examples for classification problems:

I 0 − 1 loss function: 
0, if Y = f̂ (X)
1, if Y 6= f̂ (X)
3 – Optimal approximation of f : Regression problem with
squared loss function 18/67
The theory

The loss function L:

   2
L Y, f̂ (X) = Y − f̂ (X)

The Expected Prediction Error:

 2 
EPE = E Y − f̂ (X) .

We determine f by solving the following minimization problem:

 2 
min E Y − f̂ (X) .
f̂
3 – Optimal approximation of f : Regression problem with
squared loss function 19/67
The theory

Rewrite the expectation:

 2    2 
E Y − f̂ (X) = E E Y − f̂ (X) | X

I Law of total expectation2

Pointwise optimization:
I We assume we know the predictor variables: X = x.
I Then: we optimize only the inner expectation.
New problem to be solved:
 2 
min E Y − f̂ (X) | X = x .
f̂

2 Check Wikipedia here!

3 – Optimal approximation of f : Regression problem with
squared loss function 20/67
The theory

Define:  
  2
g f̂ (x) = E Y − f̂ (X) | X = x .

Solving the minimization problem:

I First order condition:  
dg f̂ (x)
= 0.
df̂ (x)
I Second order condition:
 
d2 g f̂ (x)
> 0.
df̂ (x)2
3 – Optimal approximation of f : Regression problem with
squared loss function 21/67
The theory

Solution:
f̂ (x) = E [Y| X = x] .
Interpretation:
I f̂ (x) is the best estimate of the response;
I taking into account the information of the predictor variables.
Problem when implementing this solution:
I in order to determine the conditional expectation, the joint distribution
of (X, Y) is required.
Data is used to:
I estimate the joint distribution;
I approximate the expectation using a sample mean.
3 – Optimal approximation of f : Regression problem with
squared loss function 22/67
The real world: using the data

Question:
I We consider a one-dimensional predictor variable: X.
I Take: x = 0.5.
I Determine: f̂ (x) = E [Y| X = x].
Data set:
(xi , yi ) for i = 1, 2, . . . , 1000.
3 – Optimal approximation of f : Regression problem with
squared loss function 23/67
The real world: using the data

2.0
1.5
1.0
Y

0.5
0.0

−1.0 −0.5 0.0 0.5 1.0

X
3 – Optimal approximation of f : Regression problem with
squared loss function 24/67
The real world: using the data

Approximation 1:

1000
1
f̂ (x) = ∑ yi I ( xi = x ) ,
i = 1 I ( xi = x ) i = 1
∑1000

I I(xi = x) = 1 if xi = x and 0 otherwise.

Problem:
> x=0.5
> sum(X==5)
[1] 0
3 – Optimal approximation of f : Regression problem with
squared loss function 25/67
The real world: using the data
New (good) idea:
I Finding observations with x = 0.5 is too restrictive.
I Find observations (xi , yi ) where xi is in the neighborhood of x = 0.5.
Euclidian distance: q
d(x, xi ) = ( x − xi ) 2 .

I Distance between x and the observation xi .

K-nearest neighbor (KNN):
I We look for the K observations closest to x = 0.5.
I K is a tuning parameter.
I NK is the set with the K closest observations.
Approximation 2:
f̂ (x) ≈ Average of NK .
3 – Optimal approximation of f : Regression problem with
squared loss function 26/67
The real world: using the data

> ## k-nearest neighbor

> k=50
> DD=sqrt((X-x)^2)
> S=sort(DD, [Link]=TRUE)
> I=S$ix[1:k]
> y_hat=mean(Y[I])
> y_hat
[1] 0.4891559
3 – Optimal approximation of f : Regression problem with
squared loss function 27/67
The real world: numerical optimization

Numerical optimization:
I Instead of deriving the optimal (theoretical) solution, we can use KNN
to approximate the joint distribution and
I let R solve the problem.
Problem: Solve the minimization problem
1  2
min
f̂ (x) #NK ∑ yi − f̂ (x)
xi ∈NK

In R:
I Use the function optimize.
I We can easily change the loss function!
3 – Optimal approximation of f : Regression problem with
squared loss function 28/67
The real world: numerical optimization

> ### Numerical optimization ####

> SqLossFunction=function(yyhat,yy){
+ L=mean((yy-yyhat)^2)
+ return(L)}
> k=50
> DD=sqrt((X-x)^2)
> S=sort(DD, [Link]=TRUE)
> I=S$ix[1:k]
> optimize(SqLossFunction,yy=Y[I], interval = c(0,2))
$minimum
[1] 0.4891559

$objective
[1] 0.01136514

> y_hat
[1] 0.4891559
3 – Optimal approximation of f : Regression problem with
squared loss function 29/67
Conclusion

Numerical optimization = theoretical solution

3 – Optimal approximation of f : Regression problem with
squared loss function 30/67
The real world: Outliers

2.0
1.5
1.0
YY

0.5
0.0

−1.0 −0.5 0.0 0.5 1.0

XX

Red dot: f̂ (0.5) with a squared loss function.

Black dot: f̂ (0.5) with an absolute value loss function.
3 – Optimal approximation of f : Regression problem with
absolute value loss function 31/67
Again some theory

The absolute value loss function is:

I not very popular because of its lack of differentiability.
However:
I when considering numerical optimization, changing to the absolute
value loss function does not add a lot of complexity at first sight;
I the absolute value loss function seems to perform better when faced
with outliers.
Let’s figure out the mathematical details.
3 – Optimal approximation of f : Regression problem with
absolute value loss function 32/67
Again some theory

Loss function:  
L Y, f̂ (X) = Y − f̂ (X) .

We assume that X is known and we solve the following pointwise

minimization problem:
h i
min E Y − f̂ (X) | X = x .
f̂ (X)

Solution:
f̂ (X) = Median [Y| X = x] .
3 – Optimal approximation of f : Regression problem with
absolute value loss function 33/67
Using the data

Method 1: Determine f̂ (0.5) using numerical optimization.

> ## The absolute value
> MLossFunction=function(yyhat,yy){
+ L=mean(abs(yy-yyhat))
+ return(L)
+ }
> k=50
> DD=sqrt((XX-x)^2)
> S=sort(DD, [Link]=TRUE)
> I=S$ix[1:k]
> optimize(MLossFunction,yy=YY[I], interval = c(0,2))
$minimum
[1] 0.4838249531

$objective
[1] 0.1183076066
3 – Optimal approximation of f : Regression problem with
absolute value loss function 34/67
Using the data

Method 2: Determine f̂ (0.5) using the closed form solution.

> y_hat_M=median(YY[I])
> y_hat_M
[1] 0.4835166591
Numerical optimization gives a (slightly) different result.
I Investigate and understand where this difference comes from;
I and use numerical optimization with care.
3 – Why estimate f ? 35/67
Prediction

Prediction: Estimate Y as good as possible.

I Predicted response: ŷ.
Black box approach:
I important: How good is the prediction?
I Error: (Y − ŷ)2 .
Where does the error come from?
I f is approximated by f̂ .
I Irreducible error e.
3 – Why estimate f ? 36/67
Prediction

Theorem
Consider the function f̂ and the predictor values X. We assume that X is
non-randoom. If the predicted value is denoted by ŷ, then:
h i  2
E (Y − ŷ)2 = f (X) − f̂ (X) + Var [e] .
3 – Why estimate f ? 37/67
Proof of the result

Note that Y = f (X) + e.

And also: ŷ = f̂ (X).
Then:
h i  2 
2
E (Y − ŷ) = E f (X) + e − f̂ (X)
 2   
= E f (X) − f̂ (X) + e2 + 2 f (X) − f̂ (X) e .

 
If f and f̂ are fixed, f (X) − f̂ (X) is a constant.

Note also that E [e] = 0 and E e2 = Var [e].

 
3 – Why estimate f ? 38/67
Proof of the result: Continued

Then we find:
h i  2 
2
E (Y − ŷ) = E f (X) − f̂ (X) + E e2
 

h   i
+ 2E 2 f (X) − f̂ (X) e
 2
= f (X) − f̂ (X) + Var [e]
 
+ 2 f (X) − f̂ (X) E[e]
|{z}
=0
 2
= f (X) − f̂ (X) + Var [e] ,

which ends the proof.

3 – Why estimate f ? 39/67
Prediction

 2
f (X) − f̂ (X) : squared error of the best estimate.
I This error can be controlled.
I the error is small if f̂ ≈ f .
Var [e]: variance of the error term / irreducible error
I This error cannot be controlled.
I Var [e] = σ2 : independent from f̂ .
h i
E (Y − ŷ)2 : expected squared error of the prediction Ŷ.
I Small is good,
h i
I but: E (Y − ŷ)2 ≥ Var [e] .
3 – Why estimate f ? 40/67
Inference

Assume a car insurer, having observations about its policyholders:

I X1 , X2 , . . . , Xp : explanatory variables/predictors;
I response variable Y: number of claims.
Prediction problem:
I Assume values for the explanatory variables;
I Predict, as good as possible, the number of claims.
Complex models can be used, provided they significantly improve the
prediction.
3 – Why estimate f ? 41/67
Inference

Inference problem:
I Which variables are most important to estimate the future number of
claims?
I Do we need all p predictors, or can we obtain a desired accuracy with
much less predictors?
I How does age, miles per year, etc. influence the predicted number of
claims?
I If one policyholder drives twice as many miles per year than another
policyholder, does the first one produce the double amount of
accidents, ceteris paribus?
I ....
Simple models are more suited for inference problems:
I they are easier to understand than complex models.
I Example: linear models vs. non-linear models.
4 – Assessing Model Accuracy 42/67

Different statistical learning techniques can be applied on the same

data set.
Question: which method is the best?
Answer: compare the different models according to
I quality of fit;
I quality of the prediction;
I bias;
I etc.
4 – Measuring the quality of fit 43/67
The Mean Squared Error function

Assume: we fit a model using a dataset with n observations:

I Data: (xi , yi ), i = 1, 2, 3, . . . , n.
I We use a model to determine the function f̂ using the observations.
Question: How good can we predict the response?
I real response (coming from the data): yi ;
I prediction of the response (coming from the model): f̂ (xi ).
Mean Squared Error:

1 n  2
MSE = ∑ yi − f̂ (xi ) .
n i=1

I Measure to assess the quality of fit.

I Compare the data (the yi ) with the model (the f̂ (xi )).
4 – Measuring the quality of fit 44/67
The Mean Squared Error function

2.0
1.5
1.0 Data and Fitted values
y

0.5
0.0

Data
Fitted values

−1.0 −0.5 0.0 0.5 1.0

x
4 – Measuring the quality of fit 45/67
The Mean Squared Error function

MSE = 1.099%.
I The MSE quantifies how the model deviates from the data.
I MSE’s can be compared across models.
Can your MSE be too good to be true?
I In our example3 : Var[e] = (0.1)2 = 1%.
I There is still room for improvement!

d
3 We know this since we generated the data using Y = 2X2 + e, with e = N (0, 0.12 )
4 – Measuring the quality of fit 46/67
Can your MSE be too good to be true?

MSE for different values of k

1.10
1.05
1.00
MSE

0.95
0.90
0.85

10 20 30 40 50
4 – Measuring the quality of fit 47/67
The Mean Squared Error function

The MSE captures the model fit in a single number.

I Easy to understand and to compare across models;
I but might hide regions where the model does not perform well.
Solution: Make a residual plot
I Compare the fitted value f̂ (xi ) with the observed response yi .
I The difference f̂ (xi ) − yi is called the residual.
A plot of the x values vs the residuals should be
I centered around zero.
I constant variance. (no structure should be detected).
4 – Measuring the quality of fit 48/67
The residual plot

0.3
0.2
0.1
0.0 Residual plot
Residual

−0.1
−0.2
−0.3
−0.4

−1.0 −0.5 0.0 0.5 1.0

x
4 – Measuring the performance of the model 49/67
Training, validation and test observations

Statistical learning:
I Use data to learn about the function f̂ .
I Methodology: minimize the MSE of a given data set.
Training data set:
I The data used to learn about the function f̂ .
A small training MSE is not a real surprise4
I because we have designed our program such that the MSE is
minimized.

4A small training MSE does not mean the model is good, but a large training MSE
means the model is not good.
4 – Measuring the performance of the model 50/67
Training, validation and test observations

Example:
I Consider historical stock market data.
I Question: Predict if a stock goes up/down using past returns.
I Small Training RMSE = We can accurately predict what happened in
the past.
Does a small training MSE ensure that we can accurately predict if a
stock goes up/down in the future?
I Answer: no.
I A small training MSE implies that we are very good in predicting the
past.
4 – Measuring the performance of the model 51/67
Training, validation and test observations

Validation data set:

I An independent data set which was not used to estimate the model.
I If no validation data set is available: split the data set randomly in two
parts: training and validation data set.
Validation data:
 
xV V
j , yj , j = 1, 2, 3, . . . , m.

Validation RMSE:

1 m  V  2
Validation MSE = ∑
m j=1
yj − f̂ xV
j .
4 – Measuring the performance of the model 52/67
Training, validation and test observations

If the Validation MSE is ‘sufficiently’ small:

I The model is able to accurately predict the response of an independent
data set.
I The current model is likely to perform well in a real challenge.
If the Validation MSE is ‘too large’:
I The current version of the model is not good enough.
I we can go back to the training set and (try to) improve the model.
I For example: by determining any tuning parameters.
Repeat this procedure (training and validation) to find the best model.
4 – Measuring the quality of the fit 53/67
Training, validation and test observations

Test set:
I The validated model can be used on a test set.
I A low test MSE indicates the model is likely to be adequate.
Model selection:
I Assume we have a selection of different validated models.
I The best model is the one which has the lowest test MSE.
I For example: Kaggle competitions are determined by the performance
of the different models on a test set, which is posted at the end of the
competition.
Note that sometimes:
I Validation and test set coincide.
I there is only a training set available.
4 – Measuring the quality of the fit 54/67
Training and test, validating a model

Model

Training Fitted Model

𝑓𝑓 =? ? ? ? ? ?

We have a problem! Training set Approximation: 𝑓𝑓̂

Validation

Validated
Model
Test set

Validation set
4 – Measuring the performance of the model 55/67
Training, validation and test observations: Example

We randomly split our data set in two parts.

Training set with 800 observations:
Train=sample(1000,800)
Train_X=X[Train]
Train_Y=Y[Train]
The remaining observations form the validation/test set.
Test_X=X[-Train]
Test_Y=Y[-Train]
Results:
I Training MSE: 1.15%.
I Validation/test MSE: 1.20%.
4 – Measuring the performance of the model 56/67
Training, validation and test observations: Example

2.0
1.5
1.0 Predicted values on the Valiation
Test_Y

0.5
0.0

−1.0 −0.5 0.0 0.5 1.0

Test_X
4 – Measuring the performance of the model 57/67
Training, validation and test observations: Example

The parameter k is called a tuning parameter.

I So far: k was assumed to be fixed.
Using the validation MSE, we can try to improve the model,...
I by changing the tuning parameter k.
I Search for the k that minimizes the Validation MSE.
Note that the optimal model was developed using the training and
the validation MSE.
4 – Measuring the performance of the model 58/67
Training, validation and test observations: Example

1.6
1.5
1.4 The validation MSE
MSE

1.3
1.2
1.1

0 20 40 60 80

k
4 – Bias-Variance Tradeoff 59/67
Variance of the estimate

Consider a test observation (x, y)

I Aim: The prediction f̂ (x) should be close to the real response Y.
I The available training data set is a randomly generated subset of the
reality.
I Different training sets = different estimates f̂ .
I The prediction f̂ (x) is a random variable
Estimate f using two different training sets:
I the corresponding estimates are f̂1 and f̂2 ,
I then we hope that f̂1 (x) ≈ f̂2 (x) .

The Variance of f̂ (x) should be small.

I Large variance: small changes in the training set, may lead to very
different estimates f̂1 (x) and f̂2 (x) .
4 – Bias-Variance Tradeoff 60/67
Bias of the estimate

Bias of the prediction random variable:

h i
Bias = f (x) − E f̂ (x) .

I Note that f̂ (x) is a random variable.

The prediction random variable f̂ (x) should have:

I Low Bias:
F On average, the prediction is equal to the correct value.
I Low Variance:
F stability of the estimate.
4 – Bias-Variance Tradeoff 61/67
Variance and bias of the estimate: Example

Histogram with k=5 Histogram with k=200

20

60
50
15

40
Density

Density
10

30
20
5

10
0

0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.35 0.40 0.45 0.50 0.55 0.60 0.65

Predicted value f(0.5) Predicted value f(0.5)

4 – Bias-Variance Tradeoff 62/67
Variance and bias of the estimate: Example

> #Mean
> mean(y_hat1)
[1] 0.5016762
> mean(y_hat2)
[1] 0.5203711
> var(y_hat1)
[1] 0.002024215
> var(y_hat2)
[1] 4.971299e-05
> #Bias
> fx=2*x^2
> fx-mean(y_hat1)
[1] -0.001676238
> fx-mean(y_hat2)
[1] -0.02037111
4 – Bias-Variance Tradeoff 63/67
Theorem

Theorem
 2  h i  h i2
E Y − f̂ (x) = Var[e] + Var f̂ (x) + Bias f̂ (x) .

Flexible models: Large variance, but low bias.

Inflexible models: Low variance, but large bias.
4 – Bias-Variance Tradeoff 64/67
Example

Variance Bias
0.10

0.00
0.08

−0.05
0.06
Variance

Bias
0.04

−0.10
0.02

−0.15

0 100 200 300 400 500 0 100 200 300 400 500

k k
5 – Statistical learning, predictive analytics and data science
65/67
Statistical learning and data science

Statistical learning:
I A set of techniques which are employed to find structure and relations
in a data set.
I More technical: a set of techniques for estimating the regression
function f 5 .

5 see James et al. (2013)

5 – Statistical learning, predictive analytics and data science
66/67
Predictive analytics and data science

Predictive analytics / predictive modeling:

I Predictive analytics is the use of data, statistical algorithms and
machine learning techniques to identify the likelihood of future
outcomes based on historical data. The goal is to go beyond knowing
what has happened to providing a best assessment of what will happen
in the future. 6
A predictive model is employed to
I make a scientific guess about what will happen in the future.
I Compare with the unconditional guess.
Predictive analytics is a subset of data science.

6 from the SAS Institute: [Link].