STAT 4001

Data Mining and Statistical Learning

2 Linear Regression

Zhixiang Lin

Department of Statistics, CUHK

1 / 53
I Supervised learning: both Y and X
I Regression
I Linear regression
I Classification
I Unsupervised learning: just X

2 / 53
Linear regression

I Linear regression is a simple approach in supervised learning

I It assumes that the dependence of Y on X1 , X2 , ..., Xp is
linear.

3 / 53
Example: Advertising data

X1 : Advertisements on TV
Y : sales X2 : Advertisements on Radio
X3 : Advertisements on Newspaper

4 / 53
Let’s first consider simple linear regression using a single predictor
X.

Y : sales
X: advertisements on TV

5 / 53
Simple linear regression using a single predictor X.

Step 1: build the model

I We assume a model
Y = β0 + β1 X + 
β0 and β1 represent the intercept and slope, also known as
coefficients or parameters
 is the error term.
Step 2: estimate β0 and β1
I The training data {yi , xi }, i = 1, ..., n

6 / 53
Simple linear regression using a single predictor X.

Step 2: estimate β0 and β1

The training data {yi , xi }, i = 1, ..., n
Given some estimates β̂0 and β̂1 for the model coefficients, we
predict yi using

ŷi = β̂0 + β̂1 xi

Residual ei = yi − ŷi
Residual sum of squares (RSS) ni=1 e2i = ni=1 (yi − β̂0 − β̂1 xi )2
P P

The estimates of parameters that minimize RSS:

Pn
β̂1 = Pn(xi −x̄)(yi 2−ȳ) ,
i=1
β̂0 = ȳ − β̂1 x̄
i=1 (xi −x̄)

7 / 53
Why estimate the parametersPby minimizingP RSS?
Residual sum of squares (RSS) ni=1 ei 2 = ni=1 (yi − β̂0 − β̂1 xi )2
I The estimates β̂0 , β̂1 that minimize RSS are the same as the
Maximum Likelihood Estimate (MLE) assuming Gaussian
noise.
I We can construct confidence intervals
I We can perform hypothesis testing

8 / 53
Assessing the Accuracy of the Coefficient Estimates

The standard error of an estimator reflects how much it varies.

We have
Pn
β̂1 = Pn(xi −x̄)(yi 2−ȳ) ,
i=1
β̂0 = ȳ − β̂1 x̄
i=1 (xi −x̄)

And
2
SE(β̂1 )2 = V ar(β̂1 ) = Pn σ 2,
i=1 (xi −x̄)
2
SE(β̂0 )2 = V ar(β̂0 ) = σ 2 [ n1 + Pn x̄(xi −x̄)2 ],
i=1

where σ 2 = V ar()

Show the derivation

9 / 53
Assessing the Accuracy of the Coefficient Estimates

I Confidence interval. A 95% confidence interval is defined as a

range of values such that with 95% probability, the range will
contain the true unknown value of the parameter
[β̂0 − 2 · SE(β̂0 ), β̂0 + 2 · SE(β̂0 )],
[β̂1 − 2 · SE(β̂1 ), β̂1 + 2 · SE(β̂1 )]

10 / 53
Extension: how to estimate standard error when it is hard
to solve analytically?

We can do Bootstrapping (a resampling based method). We will

discuss it in more detail later.

11 / 53
Hypothesis testing

I Standard errors can also be used to perform hypothesis tests

on the coefficients.
I The most common hypothesis test involves testing the null
hypothesis of
I H0 : There is no relationship between X and Y versus the
alternative hypothesis
I HA : There is some relationship between X and Y .
I Mathematically, this corresponds to testing
H0 : β1 = 0 versus HA : β1 6= 0
Since β1 = 0, the model reduces to Y = β0 + 

12 / 53
Hypothesis testing

I To test the null hypothesis, we compute a t-statistic, given by

β̂1 −0
t= SE(β̂1 )
I Assuming β1 = 0, the t-statistic will have a t-distribution with
n − 2 degrees of freedom.
I Using statistical software, it is easy to compute the probability
of observing any value equal to |t| or larger. We call this
probability the p-value.

13 / 53
Assessing the Overall Accuracy of the Model

I The total sum of squares (TSS)

Pn
i=1 (yi − ȳ)2
I Residual sum of squares (RSS)
Pn
i=1 (yi − ŷi )2
I R-squared
T SS−RSS RSS
R2 = T SS
=1− T SS
I It can be shown that in this simple linear regression setting,
we have R2 = r2 , where r is the sample correlation between
X and Y

14 / 53
Demonstration using the advertising data

Implement linear regression through R and get the following

outputs:
I Estimated parameters
I Standard error
I Confidence interval
I Results of hypothesis testing
I R-square

15 / 53
Interpreting regression coefficients

I Claims of causality should be stated with cautious

I Correlation should not be confused with causality

Y : murder rate
X: IE market share
Y is correlated with X
over time

16 / 53
Example: Advertising data

X1 : Advertisements on TV
Y : sales X2 : Advertisements on Radio
X3 : Advertisements on Newspaper

17 / 53
Multiple Linear Regression

I Here our model is

Y = β0 + β1 X1 + β2 X2 + · · · + βp Xp + 
I We interpret βj as the average effect on Y of a one unit
increase in Xj , holding all other predictors fixed.
I In the advertising example, the model becomes
sales=β0 + β1 ×TV+β2 ×radio+β3 ×newspaper+

18 / 53
Estimation and Prediction for Multiple Regression

I Given estimates β̂0 , β̂1 , · · ·, β̂p we can make predictions using

the formula
ŷ = β̂0 + β̂1 x1 + β̂2 x2 + · · · + β̂p xp
I We estimate β̂0 , β̂1 , · · ·, β̂p as the values that minimize the
sum of squared residuals
n
X
RSS = (yi − ŷi )2
i=1
Xn
= (yi − β̂0 − β̂1 xi1 − β̂2 xi2 − · · · − β̂p xip )2
i=1

The values β̂0 , β̂1 , β̂2 , · · ·, β̂p that minimize RSS can be obtained
through standard statistical software

19 / 53
Demonstration using the advertising data

Implement linear regression through R and get the following

outputs:
I Estimated parameters
I Standard error
I Confidence interval
I Results of hypothesis testing
I R-square

20 / 53
Results for advertising data - issue of correlation in X
I Linear regression with all predictors

I Linear regression with newspaper only

21 / 53
Results for advertising data - issue of correlation in X
Correlations:
TV radio newspaper sales
TV 1.0000 0.0548 0.0567 0.7822
radio 1.0000 0.3541 0.5762
newspaper 1.0000 0.2283
sales 1.0000
The effect of newspaper ads on Sales is masked by its
correlation with radio.

22 / 53
Interpreting regression coefficients

I Correlations among the predictors X can cause problem

I Be cautious about the interpretations of the regression
coefficients
I Even the sign can be misleading!
I My suggestion: check the correlation between Y and each
individual predictor

23 / 53
Categorical variables in the Regression Model

I Some variables are not continuous, taking a discrete set of

values.
I These are also called categorical variables or factor variables.
I Examples: gender (male/female), ethnicity (Caucasian,
African American (AA) or Asian)

24 / 53
Credit card balance example

We want to study how different variables affect balance

25 / 53
Categorical variables - two levels

Investigate differences in credit card balance between males and

females, ignoring the other variables.
We create a new variable
(
1 if ith person is female
xi =
0 if ith person is male
Resulting model:
(
β0 + β1 + i if ith person is female
yi = β0 + β1 xi + i =
β0 + i if ith person is male

x is called dummy variable

Male is the baseline

26 / 53
Categorical variables - two levels

Results for gender model:

Implementation in R

27 / 53
Categorical variables - more than two levels

I Ethnicity: Caucasian(white), African

American(black), Asian
I Let’s use African American as baseline
I We need more dummy variables. The first could be
(
1 if ith person is Asian
xi1 =
0 if ith person is not Asian
and the second could be
(
1 if ith person is Caucasian
xi2 =
0 if ith person is not Caucasian

28 / 53
Results for ethnicity

Interpretation?

29 / 53
Extensions of the Linear Model

I Interactions
I Non-linear effects

30 / 53
Interactions in the Linear Model

I For example, the linear model:

sales=β0 + β1 ×TV+β2 ×radio+β3 ×newspaper
states that the average effect on sales of a one-unit increase
in TV is always β1 , regardless of the amount spent on radio.
I In the Advertising data, we assumed that the effect on sales
of increasing one advertising medium is independent of the
amount spent on the other media.

31 / 53
Interactions in the Linear Model

I But suppose that spending money on radio advertising

actually increases the effectiveness of TV advertising, so that
the slope term for TV should increase as radio increases.
I In this situation, given a fixed budget of $100, 000, spending
half on radio and half on TV may increase saless more than
allocating the entire amount to either TV or to radio.
I In marketing, this is known as a synergy effect, and in
statistics it is referred to as an interaction effect.

32 / 53
Modelling interactions - Advertising data

Model takes the form

sales = β0 + β1 × TV + β2 × radio + β3 × (TV × radio) + 

= β0 + (β1 + β3 × radio) × TV + β2 × radio + 

Result:

Implement it in R

33 / 53
Interpretation

I The results in this table suggests that interactions are

important.
I The p-value for the interaction term TV×radio is extremely
low, indicating that there is strong evidence for the
interaction.
I The R2 for the interaction model is 96.8%, compared to only
89.7% for the model that predicts sales using TV and radio
without an interaction term.

34 / 53
Interpretation - continued

I This means that (96.8 − 89.7)/(100 − 89.7) = 69% of the

variability in sales that remains after fitting the additive
model has been explained by the interaction term.
I An increase in TV advertising of $1, 000 is associated with
increased sales of
(β̂1 + β̂3 ×radio)×1000 = 19 + 1.1×radio units.
I An increase in radio advertising of $1, 000 will be associated
with an increase in sales of
(β̂2 + β̂3 ×TV)×1000 = 29 + 1.1×TV units.

35 / 53
Hierarchy

I Sometimes it is the case that an interaction term has a very

small p-value, but the associated main effects (in this case, TV
and radio) do not.
I The hierarchy principle:
If we include an interaction in a model, we should also include
the main effects, even if the p-values associated with their
coefficients are not significant.

36 / 53
We have talked about the interaction between two continuous
variables (Ads in TV and ratio)

What about the interaction between a continuous and a

categorical variable?

37 / 53
Credit card balance example

38 / 53
Credit card balance example

If a person have $0 income, who would spend more?

Student or non-student?
Student may spend $600, non-student may spend $400

What about that person has $1000 income?

Student may spend $600 + $200, non-student may spend $400 +
$300

Student v.s. non-student status can affect the slope ($1000 more
income, how much increase in spending)

39 / 53
Interaction between a continuous and a categorical variable

40 / 53
Interaction between a continuous and a categorical variable

Consider the Credit data set, and suppose that we wish to predict
balance using income (continuous) and student (categorical).

Without an interaction term, the model takes the form

(
β2 if ith person is a student
balancei ≈ β0 + β1 × incomei +
0 if ith person is not a student
(
β0 + β2 if ith person is a student
= β1 × incomei +
β0 if ith person is not a student

41 / 53
With interactions, it takes the form

(
β2 + β3 × incomei if student
balancei ≈ β0 + β1 × incomei +
0 if not student
(
(β0 + β2 ) + (β1 + β3 ) × incomei if student
=
β0 + β1 × incomei if not student

42 / 53
Implement it

43 / 53
Extensions of the Linear Model

I Interactions
I Non-linear effects

44 / 53
Non-linear effects of the variables
polynomial regression on Auto data

45 / 53
The figure suggests that
mpg=β0 + β1 × horsepower + β2 × horsepower2 + 

may provide a better fit.

Implement it

46 / 53
Some issues with linear regression
Outliers: unusual yi given xi

Remove it

47 / 53
Some issues with linear regression
High leverage points: unusual xi

Remove it

48 / 53
Some issues with linear regression
Non-constant variance of error terms

log(Y ) can help

49 / 53
Generalizations of the Linear Model

In much of the rest of this course, we discuss methods that expand

the scope of linear models and how they are fit:
I Classification problems: logistic regression, support vector
machines
I Regularized fitting: ridge regression and lasso
I Non-linearity: kernel smoothing, splines and generalized
additive models; nearest neighbor methods.
I Interactions: Tree-based methods, bagging, random forests
and boosting (these also capture non-linearities)

50 / 53
Key points

I Simple linear regression

I Estimate the parameters by minimizing RSS
I Standard error of the estimated parameters
I Confidence interval and hypothesis testing
I Multiple linear regression
I Interpretation of the regression coefficients
I Cautious about claiming causality
I Incorporate categorical variables with dummy variables
I Incorporate Interactions

51 / 53
Two quotes by famous Statisticians

”Essentially, all models are wrong, but some are useful”

George Box

About causality
”The only way to find out what will happen when a complex
system is disturbed is to disturb the system, not merely to observe
it passively”

Fred Mosteller and John Tukey, paraphrasing George Box

52 / 53
Credits

Parts of this material were adapted from slides by Trevor Hastie,

Robert Tibshirani, Lester Mackey, Abbass Sharif, Ying Ying Wei
and WWW

53 / 53