Introduction to Statistical Learning

Instructor: Nhat Ho

The University of Texas, Austin

Outline
• Definition of statistical learning
• Parametric versus nonparametric methods
• Assessing model accuracy
• Train/ test set
• Bias and variance trade-off
Definition of Statistical Learning
• Y: quantitative response
• Example: Income, house’s price, etc.
• X.1, …, X.p: p different predictors
• Example: Education, age, temperature, etc.
• Assume that Y and X = (X.1, …, X.p) are related through the following equation:
Y = f(X) + ε
• f is some fixed but unknown function
• ε is random error term and independent of X
Definition of Statistical Learning
• Y = f(X) + ε
• f represents the information that X provides about Y

• Definition: Statistical Learning refers to a set of techniques/ approaches to

estimate the function f

• Questions: Why should we estimate f and what are the techniques to

estimate f ?
Real Estate Example
• Open the data “Real_estate.csv” (Can be downloaded here: https://
[Link]/drive/u/0/folders/1JY1yE_9oujj9fgpTlquLIOlTTvrQQQ3z)

• Data source: Kaggle ([Link]

prediction)

• The data is about predicting real estate price (response variable) based on
some predictive variables (predictors), such as house age, distance to the
nearest station, number of convenience stores
Real Estate Example X.1 X.2 X.3

• Assume that Y = real estate price, X = (house age, distance to station, number
of stores)

• We would like to estimate f such that Y = f(X) + ε

Prediction versus Inference
• Assume that the estimation of f is f ̂

• Obtaining f ̂ is good for two purposes:

• Prediction: Given house age, distance to station, and number of
̂ ̂ ̂
stores, we predict the real estate price using Y = f(X) ( f can be
treated as a black-box)

• Inference: Understand how the real estate price changes as a function

of house age, distance to station, and number of stores (require explicit
form of f )̂

• Which predictor is most important?

• Which predictor generates the biggest price?
Estimate f : Parametric Methods
• We assume parametric form of f:
• Linear form: f(X) = aX + b

2
• Quadratic form: f(X) = aX + bX + c

m m−1
• Polynomial form: f(X) = a1X + a2X + . . . + am X + am+1
How to Estimate f ?
• The training data looks like this
(Y1, X1), . . . , (Yn, Xn)
• Step 1: We choose the form of f, e.g., f(X) = aX + b
• Step 2: We select a loss function that measures the difference between f(X)
and Y. An example is squared loss (a.k.a. least squares):
n
2
∑
L( f ) = ( f(Xi) − Yi)
i=1

• Step 3: We choose the parameters to minimize L( f )

Estimate f : Parametric Methods
Y = real estate price, X = house age

2
f(X) = aX + b f(X) = aX + bX + c
Question: Which form
will we choose? Why?
Estimate f : Parametric Methods
Y = real estate price, X = distance to stations

f(X) = aX + b 2
f(X) = aX + bX + c
Question: Which form
will we choose? Why?
Estimate f : Parametric Methods
Y = real estate price, X = number of stores

f(X) = aX + b Question: Which form

2
f(X) = aX + bX + c
will we choose? Why?
Estimate f : Nonparametric Methods
• We do not have any assumption on the form of f
• The model is much richer yet harder to obtain an estimate of f (There is no
parameter to estimate like the parametric cases)

• Popular nonparametric methods:

• K-nearest neighbors
• Kernel regression
Nonparametric Method: K-nearest Neighbors
• The training data: (Y1, X1), . . . , (Yn, Xn)
• Assume that we have new data point X* and we want to predict the value of
Y* at this new point
• K-nearest neighbors:
• Step 1: We choose K points in the set {X1, . . . , Xn} that are closest to
X*

• Step 2: We average the values of Yi at these points to obtain the Y*

Nonparametric Method: K-nearest Neighbors

K = 50 K = 10 K=2

Question: Which value of K

looks reasonable to us? Why?
 Nonparametric Method: K-nearest Neighbors

K = 50 K = 10 K=2

• Small K: complex model, very wiggly (small bias, high variance) Bias-variance
tradeoff
• Large K: smooth but simple model (large bias, small variance)
Nonparametric Methods: K-nearest Neighbors

K = 50 K = 10 K=2
Nonparametric Methods: K-nearest Neighbors

K = 50 K = 10 K=2
Nonparametric Method: Kernel Regression
• K-nearest neighbors: equal weights for all Yi (can be undesirable if some Yi
are more important than the others)

• Kernel regression: Assigning weights to Yi based on certain distributions,

such as Gaussian distribution
 Nonparametric Method: Kernel Regression

bandwidth = 2.5 bandwidth = 1 bandwidth = 0.1

• Small bandwidth leads to good bias but high variance Question: What is
“best” bandwidth?
• Large bandwidth leads to poor bias but low variance
Nonparametric Method: Kernel Regression

bandwidth = 400 bandwidth = 100 bandwidth = 20

Rule of Thumb: The bandwidth should be chosen based on the scale of data
Nonparametric Method: Kernel Regression

bandwidth = 1.5 bandwidth = 0.5

Assessing Model Accuracy
• In both kernel and K-nearest neighbor methods, how to choose good K or
bandwidth?

• We first need to define a way to assess model accuracy

• Mean square error (MSE):
n
1 ̂ 2
n∑
MSE = (Yi − f(Xi))
i=1

where ̂ ) is the prediction at data point X

f(Xi i

• MSE is important to understand the trade-off between bias and variance

• In practice, we can use RMSE = MSE to evaluate the model accuracy
Assessing Model Accuracy: Parametric Method
RMSE = 13.29 RMSE = 12.14

2
f(X) = aX + b f(X) = aX + bX + c

Question: Should we prefer quadratic

model to linear model?
 Assessing Model Accuracy: Nonparametric Method
• We use Y = real estate price, X = house age
• K-nearest neighbors method is used for estimating f

Should we choose
K = 12 as the best
value of K?

Smallest RMSE at
K = 12
Train/ Test Set
• All the previous RMSE are calculated via all the training data
• The method that obtains best accuracy in training data does not necessarily
imply good performance in the test/ new data

• The famous “over-fitting” phenomenon refers to when we have almost

perfect performance of the method in training data but have poor
performance on test/ new data

Can we K-nearest
guarantee neighbors
that this with K = 12
model still
performs well
for new/ test
data?
Train/ Test Set
• Training data: (Y1, X1), . . . , (Yn, Xn)
• Test data: (Y′1, X′1), . . . , (Y′m, X′m)
• MSE for test data:
m
1 ̂ 2
m∑
MSEout = (Y′i − f(X′i ))
i=1
̂
where f is the estimate we obtain from training data

• RMSEout = MSEout
Train/ Test Set
• In practice, test set is not available and we only have training set
• A popular approach to create a test set is to split the whole training set into
new training set and test set

• (Y1, X1), . . . , (Yn, Xn): whole training set

• (Y1, X1), . . . , (Ym, Xm): new training set (m < n)
• (Ym+1, Xm+1), . . . , (Yn, Xn): test set

• Rule of Thumb: We should not have overlapped data between training set and
test set
Train/ Test Set: Parametric Method
• We consider Y = real estate price, X = house age
• We split the whole training data into new training set and test set with the
ratio 3:1 (75% training and 25% testing data)
Train/ Test Set: Parametric Method
• First, we fit linear and quadratic models to training data

f(X) = aX + b 2
f(X) = aX + bX + c
Train/ Test Set: Parametric Method
• Now, we use these fits to the test data
RMSEout = 13.57 RMSEout = 11.77

f(X) = aX + b 2
f(X) = aX + bX + c

The quadratic model is preferred

Train/ Test Set: Nonparametric Method
• We consider Y = real estate price, X = house age
• K-nearest neighbors method is used for estimating f in the training data

K = 50 K = 10 K=3
Train/ Test Set: Nonparametric Method
• Now, we use these K-nearest neighbors fits to the test data
RMSEout = 12.13 RMSEout = 12.73 RMSEout = 14.76

K = 50 K = 10 K=3
Train/ Test Set: Nonparametric Method

Train data: best K = 12

Test data: best K = 39

K = 39 balances the simplicity and explanatory power

Train/ Test Set: Nonparametric Method
K-nearest neighbors at “best” K = 39

RMSEout = 11.88
RMSEout = 15.13

Training set Testing set

Train/ Test Set: Take-Home Messages
• Without test set, the RMSE from the training set is usually optimistic

• The RMSE from test set helps us from going too wrong

• The “best” model is the one that balances the simplicity and explanatory
power
Train/ Test Set: Practice Examples
• Thus far, we have assessed model accuracy with Y = real estate price and X =
house age

• Now, we would like to assess model accuracy when X = distance to station or

X = number of stores

• Open the “Intro_Stats_Learning_Practice_Dist.R” and

“Intro_Stats_Learning_Practice_Store.R” from the folder in this link https://
[Link]/drive/u/0/folders/1Rd5nf2486caBpr48VcVi2aaDX_XtVrKH
Train/ Test Set: Price versus Distance
• Assume that Y = real estate price and X = distance to station
• Questions:
• Between quadratic and linear models, which one we prefer? Should we
use higher order polynomial models?

• For K-nearest neighbors method, what is the best choice of K?

Train/ Test Set: Price versus Distance
• Y = real estate price, X = distance to station
Train/ Test Set: Price versus Distance
We first consider
parametric methods

RMSEout = 10.48 RMSEout = 9.87

f(X) = aX + b 2
f(X) = aX + bX + c
Train/ Test Set: Price versus Distance
We then consider K-nearest neighbors method

K = 50 K = 10 K=3

Training data
Train/ Test Set: Price versus Distance

K = 50 K = 10 K=3

Testing data
Train/ Test Set: Price versus Distance

Train data: best K = 22

Test data: best K = 7

Train/ Test Set: Price versus Distance
K-nearest neighbors at “best” K = 7

Training data Testing data

Train/ Test Set: Price versus Number of Stores
• Assume that Y = real estate price and X = number of stores
• Questions:
• Between quadratic and linear models, which one we prefer? Should we
use higher order polynomial models?

• For K-nearest neighbors method, what is the best choice of K?

Train/ Test Set: Price versus Number of Stores
• Now, we consider Y = real estate price, X = number of stores
Train/ Test Set: Price versus Number of Stores
Parametric methods

RMSEout = 10.98
RMSEout = 10.98

f(X) = aX + b 2
f(X) = aX + bX + c
Train/ Test Set: Price versus Number of Stores
K-nearest neighbors
method

Train data: best K = 28

Test data: best K = 13

Train/ Test Set: Price versus Number of Stores
K-nearest neighbors at “best” K = 13

Training data Testing data

Bias-Variance Trade-Off
• Recall what we have seen earlier with K-nearest neighbors method:
• Large K: high bias, small variance
• Small K: low bias, large variance

• What are the precise definitions of “bias” and “variance”?

Bias-Variance Trade-Off
• The training data: (Y1, X1), . . . , (Yn, Xn)
• Y = f(X) + ε
• We have a new data point X* and Y* is the corresponding value at this point
̂
• Assume that f is an estimate of f based on the training data
• Expected Mean Square Error at X* is given by:
EMSE = 𝔼[(Y* − f(X*)) ]
̂ 2

where the expectation is taken with respect to the training data

Bias-Variance Trade-Off
• We need a bit of calculations here
• EMSE = 𝔼[(Y* − f(X*)) ]
̂ 2

= 𝔼[(Y* − 𝔼[ f(X*)] + 𝔼[ f(X*)] − f(X*)) ]

2
̂ ̂ ̂

= 𝔼{(Y* − 𝔼[ f(X*)]) } + 𝔼{(𝔼[ f(X*)] − f(X*)) }

2
̂ 2 ̂ ̂

2 2
• Var(Y) = 𝔼(Y ) − 𝔼 (Y) for any random variable Y

( )
2
• 𝔼{ 𝔼[ ̂
f(X )] − ̂
f(X ) } = Var( ̂
f(X ))
* * *
Bias-Variance Trade-Off
• EMSE = 𝔼[(Y* − f(X*)) ]
̂ 2

̂ 2 ̂ ))
= 𝔼{(Y* − 𝔼[ f(X*)]) } + Var( f(X*

• Recall that, Y* = f(X*) + ε

̂ 2 ̂ 2
• 𝔼{(Y* − 𝔼[ f(X*)]) } = 𝔼{( f(X*) + ε − 𝔼[ f(X*)]) }
̂ 2 2
= 𝔼{( f(X*) − 𝔼[ f(X*)]) } + 𝔼(ε )
̂ 2 2
= ( f(X*) − 𝔼[ f(X*)]) + 𝔼(ε )
Bias-Variance Trade-Off
• EMSE = 𝔼[(Y* − f(X*)) ]
̂ 2

̂ 2 ̂ 2
= ( f(X*) − 𝔼[ f(X*)]) + Var( f(X*)) + 𝔼(ε )

̂
{Bias( f(X*))}2

The amount by which f ̂

would change if we estimate
Squared Bias Variance it using a different training
data set

The error that is introduced by

approximating a complex model by a
much simpler model
Bias-Variance Trade-Off
• EMSE = 𝔼[(Y* − f(X*)) ] = Squared Bias + Variance + 𝔼(ε )
̂ 2 2

• Ideally, we want to make Squared Bias, Variance as small as possible

• How to make these terms small?

Bias-Variance Trade-Off
• Here are rules of thumbs:
• Small variance: Less complex model
• Small bias: More complex model

• The good model is one that balances the bias and variance