Regression

LESSON 1: Regression Basics

© Faculty of Management
Regression

Which variable can explain life expectancy?

# of doctors
# of TVs

2
Covariance and
Correlation

© Faculty of Management
Variance of multiple dependent r.v.’s

Recall variance of independent random variables:

Var[a1 X 1 + a2 X 2 ] = a12Var[ X 1 ] + a22Var[ X 2 ]

What about random variables that are not independent?

E.g.: Stock prices – if economy is good, all stocks tend to move up.

House prices and square footage – if square footage is higher (bigger

house), price is likely to be higher too.

How do we account for this “togetherness”?

4
Covariance and correlation
Variance must account for “complementarity” or “together-ness” of X1 and X2
 If X1 and X2 are complements, both move in the same direction
 If X1 and X2 are substitutes, they move in opposite directions
Need a measure of “together-ness” of 2 random variables.

Covariance:
Cov(X,Y) = E[(X-μX)(Y-μY)]= E[XY] – E[X]E[Y]= μXY - μXμY

Correlation:
covariance divided by standard deviations of the variables

Cov( X , Y )
 ( X ,Y ) =
 X Y

5
Covariance

Cov(X,Y) = E[(X- X)(Y- Y)] = E[XY]-E[X]E[Y]

Properties
1) Cov > 0: Positively correlated; move together
2) Cov < 0: Negatively correlated; move in opposite directions
3) Symmetric: Cov(X, Y) = Cov(Y, X) Cov(X,X) = Var(X)
Q: If X & Y independent → Cov(X, Y) is ( zero or not zero)

Sample covariance:

( xi − X )( yi − Y )
n
Cov( X , Y ) = 
i =1 n −1

6
Covariance

Example: Price P of an economy seat from (one-way) Montreal to

Paris flight averaged E[P] = $345. Demand D averaged E[D] =
120 passengers per flight. Price and demand are negatively
correlated. What is average revenue per flight?

Need to find E[PD].

We need to know the covariance. Suppose that Cov(P,D) = -11200
Use covariance formula:
Cov(P,D) = E[PD] – E[P] * E[D]
-11200 = E[PD] – 345 * 120
E[PD]=30200

7
Correlation

Covariance: Difficult to tell the significance of relationship.

e.g. Cov(X,Y)=100K ($2) when Var(X)=2 Mil or 20K ($2)
Correlation coefficient, denoted  (rho):
Covariance scaled by product of standard deviations
Shows the significance of the linear relationship.
Cov( X , Y )
 X ,Y =  Cov( X , Y ) =  X  Y  X ,Y
 XY
For sample data, denoted 𝑟𝑋,𝑌 and computed by using sample covar/stdevs.

𝑥𝑖 − 𝑋ሜ 𝑦𝑖 − 𝑌ሜ
σ𝑛𝑖=1 ∗
𝑠𝑋 𝑠𝑌
𝑟𝑋,𝑌 =
𝑛−1

Complicated? Excel does the job for you!

8
Example

© Faculty of Management
Correlation

Tools / Data Analysis / Correlation

Example: Open Salary Dataset and calculate correlation between

Salary, Years of Experience and GPA

Excel Correlation matrix

10
Correlation (  or r)
Properties
i. -1    1, unitless, scale-free.
ii.  = 0: No linear relationship. Same as Cov = 0
iii. As  gets closer to 1 or (-1), the linear relationship gets stronger
•  = +1: Perfectly linear positive relationship.
•  = -1: Perfectly linear negative relationship.
iv. Slope of the line is not related to correlation coefficient.

 (or r) = 0  (or r) = 1
120 800
700
100
600
80
500
60 400
300
40
200
20 100

0 0
0 20 40 60 80 100 120 0 20 40 60 80 100 120

11
 Correlation (  or r)
Make a guess for the following cases
(i)  = -1.0, (ii)  = -0.7 (iii)  = 0,
(iv)  = +0.7, (v)  = 1.0
250 A 4.5 B 800 C
4 700
200 3.5
600
3
150 500
2.5
400
2
100 300
1.5
200
50 1
0.5 100

0 0 0
0 20 40 60 80 100 120 0 20 40 60 80 100 120 0 20 40 60 80 100 120

0
0
D 20 40 60 80 100 120
120
E 60 F
-50 100 40

-100 80 20

-150 60 0
0 20 40 60 80 100 120
-200 40 -20

-250 -40
20

-300 -60
0
0 20 40 60 80 100 120 -80
-350

12
 Correlation (  or r)
Make a guess for the following cases
(i)  = -1.0, D (ii)  = -0.7 F (iii)  = 0, E
(iv)  = +0.7, A (v)  = 1.0 B C
250 A 4.5 B 800 C
4 700
200 3.5
600
3
150 500
2.5
400
2
100 300
1.5
200
50 1
0.5 100

0 0 0
0 20 40 60 80 100 120 0 20 40 60 80 100 120 0 20 40 60 80 100 120

0
0
D 20 40 60 80 100 120
120
E 60 F
-50 100 40

-100 80 20

-150 60 0
0 20 40 60 80 100 120
-200 40 -20

-250 -40
20

-300 -60
0
0 20 40 60 80 100 120 -80
-350

13
Linear Regression

© Faculty of Management
Linear regression

 Basic model
 Estimating coefficients
 Using Excel
 Interpreting results
 Coefficient estimates: Confidence intervals and hypothesis
tests
 Evaluating the overall model: ANOVA and R2
 Standard error
 Prediction and confidence Intervals
 Validating the model

15
Linear regression

 Regression Analysis: used to estimate the value of one variable

on the basis of other variables.
 Identifies a relationship, not causality
 allows us to estimate (or predict) the dependent variable
 allows us to identify variables with significant explanatory power
 allows us to estimate parameters of interest (e.g.  of a stock,
impact of sq. ft. on house price)
 Dependent variable (Y): the variable we want to explain
(estimate)
 Independent variables (X1, X2,…Xp): variables used to explain Y
 (a.k.a. explanatory variables, regressors)
 Example: How does Starbucks choose a profitable spot?

16
Regression: Location analysis for Starbuck’s

 Should Starbucks have another store in MTL?

 If so, where?
 Dependent variable Y: Store profitability
 Independent variables Xi’s Map of Montreal

Rent, population, income levels, tax rates,

nearby businesses, education levels,
nearby residences, street traffic,
competition, …

 Why are we interested in relationship among variables?

17
Using regression analysis
Hypothesized
Business Reality relationships
Need for location analysis Profit = 0 + 1XPop + 2XCollege…

Interpretation Estimation and

Analysis
Add or drop vars.
Y= b0+b1Xpop+b2XCollege …..
Change the model

Estimation and analysis:

1. Does the model explain the relationship effectively?
2. Is population (Xpop) important for estimating profit?
3. Estimating the average annual profit for a potential location
4. Impact of a neighborhood college to annual profit

18
Simple linear regression

 Basic model
 Estimating coefficients
 Using Excel
 Interpreting results
 Coefficient estimates: Confidence intervals and hypothesis
tests
 Evaluating the overall model: ANOVA and R2
 Standard error
 Prediction and confidence Intervals
 Validating the model

19
Example: Montreal house price and square footage

 How important is sq footage in explaining price?

 What are other factors impacting the value of a house?
 If I invest in expanding my house by 200 sq ft by adding
another room, how much will it increase the house's value
on average?
 If my house is 1500 sq ft, how much can I expect to sell it
for?

 We take a sample of 18 houses in Montreal area.

20
Example: Montreal house price and square footage
 Chart ➔ Scatter plot
 Add a trend line (sample regression line)
 Right click data points ➔ trend line / options / show equation
Datapoint Sq. Ft Price (in $100's)
Montreal House Price by Sq. Ft.
1 1185 2400

2 1308 2399 9000

3 1688 2599 8000 y = 1.5554x + 325.4

4 1504 2599
7000
5 2050 2890
Selling price ($000's)
6000
6 1757 3195

7 1898 3299 5000

8 1960 3499 4000

9 2400 3590
3000
10 2544 3900
2000
11 2201 3949
1000
12 2437 4290

13 2877 4499 0
0 1000 2000 3000 4000 5000 6000
14 2944 5190
Square footage
15 3721 6140

16 3190 6200

17 3992 7350

18 5015 7350

21
This is great! Why learn regression analysis?
Price and Sq. Ft

Sample 1: y = 1.5554x + 325.4 Sample 2: y = 1.4792x + 370.27

10000

9000
Sample 1
8000

7000 Y= 325.4 + 1.5554X

Sample 2
6000
Price

5000

4000

3000
Y= 370.27+ 1.4792X
2000

1000

0
0 1000 2000 3000 4000 5000 6000

Sq. Ft

 Trendlines (regression lines) were built from samples: sampling error

 For different samples, we get different slopes and intercepts.
 Drawing a line is not the only task. Need to know:
 the magnitude of sampling error
 whether the model explains relationships
 reasonable values for slope ( 1) or intercept ( 0)

22
Simple Linear
Regression

© Faculty of Management
 Simple regression model: Population model yi = 0 + 1xi + i

 One dependent variable, y and one independent variable, x

 Relationship between x and y is a linear function
 A change in y value is explained by a change in x value

Population regression model: yi = β0+ β1xi +εi

 β0 = Intercept → the value of y when x = 0 (when x = 0 makes
sense). Otherwise, the average value of y not explained by x.
 β1 = Slope → the average increase in y for a unit increase in x
 ε = Random error (residual)

 Regression Assumptions:
1.  is independent of x and follows N(0, ) for all x.
2. i do not exhibit autocorrelation (are independent of each other):
Cov(i, j)=0

24
Simple regression model

 A regression line estimates a population reg. line.

 Use sample data to determine b0 and b1. (statistics)
 Are b0 and b1 random variables? (Yes or No)
b0 estimates β0 , std. error: sb
yi = 0 + 1xi +  i b1 estimates β1 , std. error: sb
0

yˆ i estimates  Y|x =  0 + 1 xi
ˆi = b0 + b1 xi
y
the average value of y at x i
How do we find b0 and b1?
ei: residual (difference between an actual y and a predicted value ŷ)

residual : e i = y i - ŷ i = yi - (b 0 + b1x i )

Find a straight line which minimize the sum of ei or ei2

25
Simple regression: Least square criterion

 Why minimize the sum of squared residuals?

 Least square estimators: unbiased and consistent estimator for i
 How do we find b0 and b1?
min e i
2
= (y i −ŷi )2 = (y i − (b0 + b1x i )) 2
y Residual for this x ŷ = b 0 + b1x
value
Observed value of
y for xi
ei Slope = b1
Estimated value of
y for xi

Intercept = b0

xi x
26
Estimating Regression
Coefficients

© Faculty of Management
Simple linear regression: Estimating coefficients

min  ei = 2
 i i =
(y −ŷ ) 2
 i 0 1 i
(y − (b + b x )) 2

σ(𝑥 − 𝑥)(𝑦
lj − 𝑦)
lj 𝐶𝑜𝑣(𝑥, 𝑦) 𝑟𝑋,𝑌 𝑠𝑋 𝑠𝑌 𝑠𝑌
𝑏1 = = = = 𝑟𝑋,𝑌
σ(𝑥 − 𝑥)lj 2 𝑠𝑥2 𝑠𝑥2 𝑠𝑋

A sample regression line always passes through ( x , y ) : b0 = y − b1 x

y ŷ = b 0 + b1x

y
ei Slope = b1

Intercept =
b0
xi x x
28
Using Excel: Montreal house price example
Want to explain the house price (Y) with the sq. ft. (X)
Pop. model: yi=β0+β1xi+εi Estimated model: yˆ i = b0 + b1 xi
Tools / Data Analysis / Regression
Range of Y
(include label)

Range of X
(include label)

Check if you have labels

output
range

output options

29
Using Excel: Montreal house example Excel output
SUMMARY OUTPUT

Regression Statistics
Multiple R 0.9623 3. Regression summary statistics
R Square 0.9261
Adjusted R Square 0.9214
Standard Error 455.2725
Observations 18 2. Overall significance of a model
ANOVA
df SS MS F Significance F
Regression 1 41530449.03 41530449 200.3659 1.81687E-10
Residual 16 3316369.417 207273.1
Total 17 44846818.44

Coefficients Standard Error t Stat P-value Lower 95% Upper 95%

Intercept 325.403 293.0507 1.1104 0.2832 -295.8366 946.6430
Sq. Ft 1.555 0.1099 14.1551 0.0000 1.3224 1.7883

1. Information on b0 and b1.

Estimated regression line: 𝑦𝑥 = 325.403 + 1.555 x

How do we interpret the coefficients and the model?

30
Montreal house example: Test for a slope parameter
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 325.403 293.0507 1.1104 0.2832 -295.8366 946.6430
Sq. Ft 1.555 0.1099 14.1551 0.0000 1.3224 1.7883

Excel default test: H0: i = 0 HA: i  0

Implication: H0: 1 = 0 (no linear relationship)
HA: 1  0 (linear relationship does exist)
Which one is useful?
Sample regression lines if
Sample regression lines if H0 is not true
H0 is true
y y y

x x x
The expression “the coefficient of ___ is significant” refers to this default
hypothesis test: You can conclude the coefficient is different from zero (at
the specified α)
31
Evaluating Regression
Model

© Faculty of Management
Evaluating the overall regression model
Pop. regression model yi = 0 +1xi +I
Our fit line: ŷi = b0 +b1xi

How well does our model explain the variation in y (from the mean y )?

Total Variation in y (SST) = explained variation (SSR) + unexplained variation (SSE)

y
yi
SSE = ∑(yi - ŷi )2 ŷ

SST = ∑(yi - y̅)2

ŷ
SSR = ∑(ŷi - y̅)2
y̅ y̅

xi x
33
Evaluating the overall regression model
n

 i
SST (Total Sum of Squares)
variation of the y values around their mean
( y − y ) 2

i =1

=
SSR (Regression Sum of Squares) n
portion of the total sum of squares explained
by the estimated model
 i
( ˆ
y
i =1
− y ) 2

+
SSE (Sum of Squared Errors) n
the sum of squares of the errors (between the
model prediction and the actual observed
 i i
( ˆ
y
i =1
− y ) 2

data)

Where do I see these in Excel Output?

Look at ANOVA [i.e., Analysis of Variance] table.
 Evaluating overall model: ANOVA information

 ANOVA table gives us information for computing

 ANOVA = Analysis of Variance
 R2
 Proportion of total variation in data explained by regression model
 se
 Standard error of the estimate: estimated standard deviation of error term
 Recall, we assume that εi ~ N(0, σ). This is the estimate of σ.

35
Evaluating the overall model: ANOVA Table
Regression Statistics
SSR =  ( yˆ i − y ) 2

SSE =  ( yˆ i − yi ) 2
Multiple R 0.9623
R Square 0.9261
Adjusted R Square 0.9214
Standard Error 455.2725 SST =  ( yi − y ) 2
Observations 18

ANOVA
df SS MS F Significance F
Regression 1 41530449.03 41530449.03 200.366 1.81687E-10
Residual 16 3316369.417 207273.0886
Total 17 44846818.44

36
Evaluating the overall model: Coefficient of determination, R2
Regression Statistics 𝑆𝑆𝑅 𝑆𝑆𝑇 − 𝑆𝑆𝐸 original − unexplained
Multiple R 0.9623 𝑅2 = = =
R Square 0.9261
𝑆𝑆𝑇 𝑆𝑆𝑇 original
Adjusted R Square 0.9214
Standard Error 455.2725 The portion of the total variation explained
Observations 18
by the regression model.
ANOVA
df SS MS F Significance F
Regression 1 41530449.03 41530449.03 200.366 1.81687E-10
Residual 16 3316369.417 207273.0886
Total 17 44846818.44

In our case, R2 = SSR/SST = 41,530,449.03 / 44,846,818.44

( 92.61%) of the variation in the sample data explained by the model.
 Always between 0 and 1.
 In a simple regression model (one independent variable), R2= r2
 The closer R2 is to 1, the better the fit.

37
 Evaluating the overall model: How do I determine a bad model?
Regression Statistics
Multiple R 0.9623 1. If you cannot reject H0 that all independent
R Square 0.9261 variable parameters i are zero, or
Adjusted R Square 0.9214 2. R2 is too small, or
Standard Error 455.2725
Observations 18
3. if sb is so large that CI for i includes zero

ANOVA
df SS MS F Significance F
Regression 1 41530449.03 41530449.03 200.366 1.81687E-10
Residual 16 3316369.417 207273.0886
Total 17 44846818.44

Then, rethink your model before proceeding with further analysis.

38
Evaluating the overall model: Estimating σ
Regression Statistics Recall regression model yi= 0 +1xi +i
Multiple R 0.9623
R Square 0.9261 where i~N(0, ).
Adjusted R Square 0.9214
Standard Error 455.2725 This is the estimate of σ: the standard deviation of
Observations 18 the errors around the regression line. It is called
the “standard error of the estimate”
ANOVA
df SS MS F Significance F
Regression 1 41530449.03 41530449.03 200.366 1.81687E-10
Residual 16 3316369.417 207273.0886
Total 17 44846818.44

𝑆𝑆𝐸
𝑠𝜀 = 𝑀𝑆𝐸 =
𝑛−𝑝−1
is an unbiased estimator for 

𝑆𝑆𝐸 3,316,369.417
𝑠𝜀 = = = 455.2725
𝑛−𝑝−1 16

39
 Confidence and
Prediction Intervals

© Faculty of Management
 Estimation for mean value (y|x) & prediction for an individual value (yi)

Estimated regression line: yˆ = b0 + b1 x

For given x, how likely is that ŷ exactly equals y|x=0+1x ?
( Vey unlikely ) why? Because of sampling error
Useful to have a range (interval) rather than just a point estimate for ŷ
Two cases:
1) Confidence Interval for the average y value for given x
(estimating y|x=0+1x using ŷ=b0 + b1x )
2) Prediction Interval for a particular y value for given x
(estimating yi = 0+1x + i using ŷ=b0 + b1x )
Example:
a) Want to determine the avg. price of 2600 sq. ft. houses: (CI or PI)
b) Want to predict the price of a 2600 sq. ft house (CI or PI)

41
100(1-)% CI for the avg. y value at x (y|x)
Confidence Interval for the average y value for given x
(estimating y|xp=0+1x using ŷ=b0 + b1x )

100(1-)% CI: Point Estimate  (Crit. value)(Std. Err)

Point estimate: 𝑦ො𝒙 = 𝑏0 + 𝑏1 𝒙
Critical value: t/2 with df = n-p-1
Standard error of the avg. y value at x

Exact MGCR-614 Class

1 lj 2
(𝒙 − 𝑥) lj 2
1 (𝒙 − 𝑥) 1
𝑠𝜀 + = 𝑠𝜀 + s
𝑛 σ(𝑥𝒊 − 𝑥) lj 2 𝑛 (𝑛 − 1)𝑠𝑥2 n

Exact 100(1-)% CI MGCR-614 Class CI

lj 2
1 (𝒙 − 𝑥) 1
𝑦𝑥𝑝 ± 𝑡𝛼/2 𝑠𝜀 + y x p  t / 2 s
𝑛 (𝑛 − 1)𝑠𝑥2 n

42
MTL house example: CI for the avg. y value at x (y|x)
Construct a 95% CI for the avg. price of the 2600 sq. ft. houses.
Coefficients Standard Error t Stat P-value Regression Statistics
Intercept 325.403 293.0507 1.1104 0.2832 Multiple R 0.9623
R Square 0.9261
Sq. Ft 1.555 0.1099 14.1551 0.0000 Adjusted R Square 0.9214
Standard Error 455.2725
Observations 18

Point estimate: 𝑦ො𝑥 = 𝑏0 + 𝑏1 𝑥 325 + 1.555*2600 = 4368 ($ ’00)

Critical value: t/2 with df = 18-1-1=16 2.12

1
Std. err. of the avg. y s 455.27 * √(1/18) = 107.31
n
Approximated 95% CI
1 4368 ± 2.12 * 107.31
y x p  t / 2 s = [ 4141, 4595] ($’00)
n
43
Prediction interval for an individual y value
Start with the sample point estimate and critical values
𝑦ො𝒙 = 𝑏0 + 𝑏1 𝒙 crit-value: t/2 with df = n-p-1
Which one is bigger?
(var. of the sample mean or var. of an individual value)
need to account for a bigger source of variability.

Exact 100(1-)% Pred. Interval MGCR-614 Class PI

lj 2
1 (𝑥 − 𝑥) 1
𝑦𝑥 ± 𝑡𝛼/2 𝑠𝜀 1+ + 𝑦𝑥 ± 𝑡𝛼/2 𝑠𝜀 1 +
𝑛 (𝑛 − 1)𝑠𝑥2 𝑛

For a given x, the prediction interval is (wider or narrower) than the CI.

44
MTL house example: Prediction interval for a specific y value
Construct a 95% Prediction Interval for the price of a 2600 sq. ft. house.
Coefficients Standard Error t Stat P-value Regression Statistics
Multiple R 0.9623
Intercept 325.403 293.0507 1.1104 0.2832
R Square 0.9261
Sq. Ft 1.555 0.1099 14.1551 0.0000 Adjusted R Square 0.9214
Standard Error 455.2725
Observations 18

Point estimate: 𝑦ො𝑥 = 𝑏0 + 𝑏1 𝑥 4,368 ($’00)

Critical value: t/2 with df = 16 2.12

Std. err. of the avg. y s 1+ 1 455.27 1 +

1
= 467.75
n 18

Approximated 95% prediction interval

1 4,368 ± 2.12 * 467.75

𝑦𝑥 ± 𝑡𝛼/2 𝑠𝜀 1+ [ 3376, 5360] ($’00)
𝑛

45
 Confidence and prediction interval: Exact vs. approximated

Exact PI for an individual y, given xp

y appr. PI

4368.4 appr. CI

Exact CI for the mean of y,

given x
x
x
x x=2600
The closer x is to its mean, the more precise estimation (prediction) becomes.
Caution: Regression should not be trusted when extrapolating beyond the range
of data (x) collected.
All conclusions are valid only over the range of data in the sample.
Q: Can I trust a prediction on the price of a 200 sq. ft house?
46
Key learning points: Simple linear regression
 Population model: Independent variables explain dependent
variable
 Examples
 Highest college degree ➔ salary
 Square-ft of a house ➔ price
 Salesperson hours of “customer engagement time” ➔ $ sales for period
 Estimating a regression model
 Use sample data to estimate parameters for population model
 Testing significance of:
Individual coefficients (default test) [Coefficients section of Excel output]

 Overall model [ANOVA table of Excel output]
 Overall fit of model to data: R2 [Summary section of Excel output]
 Estimating
 Confidence Interval: Mean value of y for given X
 Prediction Interval: Value of individual y for given X

47