STATISTICAL
LEARNING
ASSESSING MODEL ACCURACY
WHAT IS STATISTICAL LEARNING
Suppose we have quantitative response Y and p different predictors X1,X2,
….Xp for the input variable.
The relationship between Y and X is:
Y = f(X) + e
Where e is a random error term, independent of X and has zero mean.
WHAT IS STATISTICAL LEARNING
• f is two dimensional surface that must be estimated based on the observed
data
• f is the systematic information that X provides about Y
Set of approaches for estimating f is referred to as Statistical Learning.
WHY ESTIMATE F ?
The two main reasons why we want to estimate f are:
1. Prediction
In many situations X inputs are available but output Y cannot be easily
obtained.
Y can be predicted as:
YŶ= fcap (X)
Where fcap gives the estimate for f and YŶrepresents predictions for Y,
considering the error term averages to zero.
WHY ESTIMATE F ?
2. Inference
We need to know the exact form of f to answer the following questions:
• Which predictors are associated with the response
• What is the relationship between the response and each predictor
• Can the relation between Y and Xp be summarized using linear equation
or is it more complicated
EXAMPLES
Prediction : Direct marketing campaign
• Identify individuals who respond positively to a mailing, which depends
on demographic (DG) variables of each individual
• DG variables are predictors and response to the campaign (positive or
negative) is outcome
• How accurately can the company predict the responses using the
predictors !!!!!!!
EXAMPLES
Inference : Sales of a product along with the advertising budget across TV,
radio, Newspaper
Some interesting questions would be :
• Which media are associated with sales
• Which media generate biggest boost in sales
• How increase in TV advertising affecting the sales growth
EXAMPLES
Inference : Brand of a product purchased based on price, store location,
discount etc.
• Association of each variable and the probability of purchase
• How the product price is associated with sales
• Is there any connection of sales with the store location
EXAMPLES
Prediction and Inference : Real estate business
●
Inference
• How can we relate cost of home to inputs such as crime rate, distance from
river, schools nearby, air quality and so on
• So the cost of house will be higher for instance if it has a view of the river
Prediction
• Predicting the house value by its characteristics : under or over valued?
HOW TO ESTIMATE F
• Parametric methods
Fitting the model using least squares
• Non parametric methods
No explicit assumption to f is made
Thin plate spline fit to the model
ASSESSING MODEL ACCURACY
• There is no single method good for all data fitting
• One method might be the best for one data set while some other
method may work better for a different data set
• Deciding which model fits the data best is the most challenging part of
statistical learning
QUALITY OF A FIT
•
TRAINING DATA vs. TEST DATA
The MSE is computed using the training data used to fit the model
So the MSE is called as Training MSE
But, we are instead interested in accuracy of the predictions obtained when we apply
the method to unseen TEST data
TRAINING DATA vs. TEST DATA
Training data is the subset of original data that is used to train the model
Testing data is used to check the accuracy of the model
The training dataset is generally larger in size compared to the testing dataset.
TRAINING DATA vs. TEST DATA
Developing an algorithm to predict a stock’s price based on previous stock returns
The method can be trained using stock returns from the past 6 months
We don’t care how well our method predicts last week’s stock price
We care about how well it will predict tomorrow’s price or next month’s price.
TRAINING DATA vs. TEST DATA
Clinical measurements (weight, blood pressure, height, age, etc.) for a number of
patients, and the information about whether each patient has diabetes.
We can use them as training data for a statistical learning method to predict risk of
diabetes based on clinical measurements.
We want this method to accurately predict diabetes risk for future patients based on
their clinical measurements
We are not interested in whether or not the method accurately predicts diabetes risk
for patients used as training data
Mathematical form of Test MSE
Let (x1,y1) (x2,y2) (x3,y3).... are the training observations
We use a model to estimate fcap on this data
We can then compute fcap(x1), fcap(x2), fcap(x3) and so on
If these values are approximately equal to y1,y2,y3..., then the training MSE is small
Mathematical form of Test MSE
We are though not interested to know whether
fcap(xn)=yn or not
We want to know whether fcap(xo)≈yo or not
where (x 0 , y 0 ) is a previously unseen test observation not used to train the statistical
learning method
Mathematical form of Test MSE
For a large number of test observations,
Ave (yo-fcap(xo))2
is the average squared prediction error for these test observations (x 0 , y 0 ).
This quantity should be as small as possible
Training vs. Test MSE
●
Left: Data simulated from f (black).
●
Three estimates of f are shown:
●
the linear regression line (orange curve)
●
two smoothing spline fits (blue and
green curves)
Training vs. Test MSE
●
Right: Training MSE (grey curve), test
MSE (red curve)
●
minimum possible test MSE over all
methods (dashed line)
●
Squares represent the training and test
MSEs for the three fits shown in the
left-hand panel
Training vs. Test MSE
Low training MSE
and high test MSE
means overfitting
data
Training vs. Test MSE
A different true f that is much closer to
linear.
In this setting, linear regression
provides a very good fit to the data
Training vs. Test MSE
To the right, because the truth is close
to linear, the test MSE only decreases
slightly before increasing again
So the orange least squares fit is
substantially better than the highly
flexible green curve
Training vs. Test MSE
A different f that is far from linear
In this setting, linear regression
provides a very poor fit to the data
Training vs. Test MSE
To the right : There is a rapid decrease
in both curves before the test MSE
starts to increase slowly
The Bias-Variance Trade-Off
●
The U-shape observed in the test MSE curves is the result of two competing
properties of statistical learning methods
● The expected test MSE, for a given value x 0 , can be decomposed into the sum of
three fundamental quantities:
The variance of fcap(x 0 ), the squared bias of fcap(x 0 ) and the variance of the error
terms e
The Bias-Variance Trade-Off
Expected test MSE at xo
The overall expected test MSE can be computed by averaging E over all possible values of xo
in the test data set
The Bias-Variance Trade-Off
To minimize the expected test error, we need to select a statistical learning method
that simultaneously achieves low variance and low bias.
Both variance and squared bias are nonnegative quantities inherently
The Bias-Variance Trade-Off
Variance refers to the amount by which fcap would change if we
estimated it using a different training data set.
Since the training data are used to fit the statistical learning method,
different training data sets will result in a different fcap .
The Bias-Variance Trade-Off
Bias refers to the error that is introduced by approximating a real-life problem,
which may be extremely complicated, by a much simpler model