Chapter 15
Multiple Regression
The Multiple Regression
Model
Examine the linear relationship between
1 dependent (y) & 2 or more independent variables (xi)
Population model:
Y-intercept Population slopes Random Error
y = β0 + β1x1 + β 2 x 2 + + βk x k + ε
Estimated multiple regression model:
Estimated Estimated
(or predicted) intercept Estimated slope coefficients
value of y
ŷ = b0 + b1x1 + b 2 x 2 + + bk x k
Multiple Regression Model
Two variable model
y
ŷ = b0 + b1x1 + b 2 x 2
x1
e
abl
i
var
r
fo
ope x2
Sl
varia ble x 2
e fo r
S lo p
x1
Multiple Regression Model
Two variable model
y Sample
<yi
observation ŷ = b0 + b1x1 + b 2 x 2
yi
<
e = (y – y)
x2i
x2
<
x1i The best fit equation, y ,
is found by minimizing the
x1 sum of squared errors, Σ e2
Multiple Regression
Assumptions
Errors (residuals) from the regression model:
<
e = (y – y)
The errors are normally distributed
The mean of the errors is zero
Errors have a constant variance
The model errors are independent
Model Specification
Decide what you want to do and select the
dependent variable
Determine the potential independent variables for
your model
Gather sample data (observations) for all variables
Example
A distributor of frozen desert pies wants to
evaluate factors thought to influence demand
Dependent variable: Pie sales (units per week)
Independent variables: Price (in $)
Advertising ($100’s)
Data are collected for 15 weeks
Pie Sales Model
Week Pie Sales Price Advertising
($) ($100s)
1 350 5.50 3.3
Multiple regression model:
2 460 7.50 3.3
3 350 8.00 3.0 Sales = b0 + b1 (Price)
4 430 8.00 4.5
5 350 6.80 3.0 + b2 (Advertising)
6 380 7.50 4.0
7 430 4.50 3.0
8 470 6.40 3.7
Correlation matrix:
9 450 7.00 3.5
Pie Sales Price Advertising
10 490 5.00 4.0
Pie Sales 1
11 340 7.20 3.5
Price -0.44327 1
12 300 7.90 3.2
Advertising 0.55632 0.03044 1
13 440 5.90 4.0
14 450 5.00 3.5
15 300 7.00 2.7
Interpretation of Estimated
Coefficients
Slope (bi)
Estimates that the average value of y changes by bi units
for each 1 unit increase in Xi holding all other variables
constant
Example: if b1 = -20, then sales (y) is expected to decrease
by an estimated 20 pies per week for each $1 increase in
selling price (x1), net of the effects of changes due to
advertising (x2)
y-intercept (b0)
The estimated average value of y when all xi = 0 (assuming
all xi = 0 is within the range of observed values)
Pie Sales Correlation Matrix
Pie Sales Price Advertising
Pie Sales 1
Price -0.44327 1
Advertising 0.55632 0.03044 1
Price vs. Sales : r = -0.44327
There is a negative association between
price and sales
Advertising vs. Sales : r = 0.55632
There is a positive association between
advertising and sales
Scatter Diagrams
Sales Sales vs. Price
600
500
400
300
Sales vs. Advertising
200 Sales
100 600
0 500
Price
0 2 4 6 8 10
400
300
200
100
0
Advertising
0 1 2 3 4
Multiple Regression Output
Regression Statistics
Multiple R 0.72213
R Square 0.52148
Adjusted R Square 0.44172
Standard Error 47.46341
Sales = 306.526 - 24.975(Pri ce) + 74.131(Adv ertising)
Observations 15
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
The Multiple Regression
Equation
Sales = 306.526 - 24.975(Price) + 74.131(Adv ertising)
where
Sales is in number of pies per week
Price is in $
Advertising is in $100’s.
b1 = -24.975: sales b2 = 74.131: sales will
will decrease, on increase, on average,
average, by 24.975 by 74.131 pies per
pies per week for week for each $100
each $1 increase in increase in
selling price, net of advertising, net of the
the effects of effects of changes
changes due to due to price
advertising
Using The Model to Make
Predictions
Predict sales for a week in which the selling
price is $5.50 and advertising is $350:
Sales = 306.526 - 24.975(Price) + 74.131(Advertising)
= 306.526 - 24.975 (5.50) + 74.131(3.5)
= 428.62
Note that Advertising is
Predicted sales in $100’s, so $350
means that x2 = 3.5
is 428.62 pies
Multiple Coefficient of
Determination
Reports the proportion of total variation in y
explained by all x variables taken together
SSR Sum of squares regression
R = 2
=
SST Total sum of squares
Multiple Coefficient of
Determination
(continued)
Regression Statistics
Multiple R 0.72213 SSR 29460.0
R =2
= = .52148
R Square 0.52148
SST 56493.3
Adjusted R Square 0.44172
Standard Error 47.46341 52.1% of the variation in pie sales
Observations 15 is explained by the variation in
price and advertising
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
Adjusted R2
R2 never decreases when a new x variable is
added to the model
This can be a disadvantage when comparing
models
What is the net effect of adding a new variable?
We lose a degree of freedom when a new x
variable is added
Did the new x variable add enough
explanatory power to offset the loss of one
degree of freedom?
Adjusted R2
(continued)
Shows the proportion of variation in y explained by all x
variables adjusted for the number of x variables used
n −1
R = 1 − (1 − R )
2 2
A size, k = number of independent variables)
n − k − 1
(where n = sample
Penalize excessive use of unimportant independent variables
Smaller than R2
Useful in comparing among models
Multiple Coefficient of
Determination
(continued)
Regression Statistics
Multiple R 0.72213 R 2A = .44172
R Square 0.52148
Adjusted R Square 0.44172 44.2% of the variation in pie sales is
Standard Error 47.46341 explained by the variation in price and
Observations 15 advertising, taking into account the sample
size and number of independent variables
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
Is the Model Significant?
F-Test for Overall Significance of the Model
Shows if there is a linear relationship between all
of the x variables considered together and y
Use F test statistic
Hypotheses:
H0: β1 = β2 = … = βk = 0 (no linear relationship)
HA: at least one βi ≠ 0 (at least one independent
variable affects y)
F-Test for Overall
Significance
(continued)
Test statistic:
SSR
k MSR
F= =
SSE MSE
n − k −1
where F has (numerator) D1 = k and
(denominator) D2 = (n – k - 1)
degrees of freedom
F-Test for Overall
Significance
(continued)
Regression Statistics
Multiple R 0.72213
R Square 0.52148 MSR 14730.0
Adjusted R Square 0.44172
F= = = 6.5386
Standard Error 47.46341
MSE 2252.8
Observations 15 With 2 and 12 degrees P-value for
of freedom the F-Test
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
F-Test for Overall
Significance
(continued)
H0: β1 = β2 = 0 Test Statistic:
HA: β1 and β2 not both zero MSR
F= = 6.5386
α = .05 MSE
df1= 2 df2 = 12
Decision:
Critical Reject H0 at α =
Value:
0.05
Conclusion:
Fα =
3.885 The regression model does explain
α =. a significant portion of the
05 variation in pie sales
0 Do not Reject H0
F (There is evidence that at least one
reject H0
F.05 = 3.885 independent variable affects y)
Are Individual Variables
Significant?
Use t-tests of individual variable slopes
Shows if there is a linear relationship between the
variable xi and y
Hypotheses:
H0: βi = 0 (no linear relationship)
HA: βi ≠ 0 (linear relationship does exist
between xi and y)
Are Individual Variables
Significant?
(continued)
H0: βi = 0 (no linear relationship)
HA: βi ≠ 0 (linear relationship does exist
between xi and y)
Test Statistic:
bi − 0
t= (df = n – k – 1)
sbi
Are Individual Variables
Significant?
(continued)
Regression Statistics
Multiple R 0.72213 t-value for Price is t = -2.306, with
R Square 0.52148 p-value .0398
Adjusted R Square 0.44172
Standard Error 47.46341 t-value for Advertising is t = 2.855,
Observations 15 with p-value .0145
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
Inferences about the Slope:
t Test Example
From Excel output:
H0: βi = 0 Coefficients Standard Error t Stat P-value
HA: βi ≠ 0 Price -24.97509 10.83213 -2.30565 0.03979
Advertising 74.13096 25.96732 2.85478 0.01449
d.f. = 15-2-1 = 12
α = .05 The test statistic for each variable falls
tα /2 = 2.1788 in the rejection region (p-values < .05)
Decision:
α /2=.025 α /2=.025
Reject H0 for each variable
Conclusion:
Reject H0 Do not reject H0 Reject H0
There is evidence that both
-tα/2 tα/2 Price and Advertising affect
0
-2.1788 2.1788 pie sales at α = .05
Confidence Interval Estimate
for the Slope
Confidence interval for the population slope β1
(the effect of changes in price on pie sales):
b i ± t α / 2 sbi where t has
(n – k – 1) d.f.
Coefficients Standard Error … Lower 95% Upper 95%
Intercept 306.52619 114.25389 … 57.58835 555.46404
Price -24.97509 10.83213 … -48.57626 -1.37392
Advertising 74.13096 25.96732 … 17.55303 130.70888
Example: Weekly sales are estimated to be reduced
by between 1.37 to 48.58 pies for each increase of $1
in the selling price
Standard Deviation of the
Regression Model
The estimate of the standard deviation of the
regression model is:
SSE
sε = = MSE
n − k −1
Is this value large or small? Must compare to the
mean size of y for comparison
Standard Deviation of the
Regression Model
(continued)
Regression Statistics
Multiple R 0.72213
R Square 0.52148 The standard deviation of the
Adjusted R Square 0.44172 regression model is 47.46
Standard Error 47.46341
Observations 15
df SS MS F Significance F
ANOVA
Regression 2 29460.027 14730.013 6.53861 0.01201
Residual 12 27033.306 2252.776
Total 14 56493.333
Coefficients Standard Error t Stat P-value Lower 95% Upper 95%
Intercept 306.52619 114.25389 2.68285 0.01993 57.58835 555.46404
Price -24.97509 10.83213 -2.30565 0.03979 -48.57626 -1.37392
Advertising 74.13096 25.96732 2.85478 0.01449 17.55303 130.70888
Standard Deviation of the
Regression Model
(continued)
The standard deviation of the regression model is
47.46
A rough prediction range for pie sales in a given
week is ± 2(47.46) = 94.2
Pie sales in the sample were in the 300 to 500
per week range, so this range is probably too
large to be acceptable. The analyst may want to
look for additional variables that can explain more
of the variation in weekly sales
