Module 10: Inference in

Simple Linear Regression

ISOM2500 BUSINESS STATISTICS
(L1, Sprin 2026)
Jason HO
Contents
• Inference in Regression
• Confidence Interval for the Mean Response
• Prediction Interval for a New Observation
• Regression Diagnostics [Out of scope of exam]

2
Procedure in applying SLR (Recall from Module 9)

1. Produce a scatterplot of y on x
• Check for linearity
2. Find the LS equation, and obtain residuals
3. Perform regression diagnostics (Module 10)
4. Test the slope parameter (Module 10)
5. Interpret the LS equation
• intercept and slope estimates, R2
6. Make prediction of new observations (Module 10)

3
Inference in regression

4
Population VS sample (Recall from Module 9, Slide 34)
unobserved observed

Theory: SLR model Mean of Y for X = x Reality

Y
Y
Population Sample

The purple line

& the red line
are
NOT the same
Normal with
equal variance σ2

X X
5
Inference in regression

• Recall: Three parameters, and , identify/characterize the

population described by SLR

• The LS method provides their estimates:

• Often, we have from data, but it does NOT imply

• A fitted model/equation provides the foundation for statistical
inference once it remains valid after regression diagnostics [Step 3
of fitting SLR, to be discussed at the end of this Module]
6
 Regression model equation

Validity of SLR

After getting the LS line, the validity test, a 2-sided test of H0:
is the first and foremost test to carry out
No The SLR reduces to

• Estimate of y is (i.e., X is of no use)

Reject
H0
• Call X a significant/useful predictor of Y
Yes • Use the LS line (based on X) to (1) estimate mean
of Y, and (2) predict any new observation Y
7
Parameters and inference

In Modules 6-8, in estimation of a normal population mean :

1. is estimated by the sample mean
2. Inference of is based on sampling distribution of
• With , the standardized sample mean
when is
or unknown

Parameters in Statistics
In SLR: population from data
estimate, or
, point estimates of , LS estimates

• Inference: based on sampling distributions of and 8

Sampling distribution of Module 9, Slide 47

• The LS estimate is unbiased for (i.e., )

• The estimated standard errors are given by

in terms of (1) SE of the regression , (2) sample SD of the x

values , (3) sample size n
• The sampling distribution of the standardized is given by

9
Confidence interval for the slope parameter

A 95% confidence interval for the parameter in SLR is the

interval

where the t quantile

(obtainable in MS Excel by
[Link]( )) is defined Area
from a Student’s t-distribution = 0.975
with n-2 degrees of freedom,
as in Module 7, Slide 17
10
Hypothesis test for the slope parameter

• 2-sided test: , for any hypothesized value

• Observed test statistic
(also called t-statistic) :

• Reject H0 at level α if , or
p-value =
• 1-sided test:
• Same observed test statistic as above
• Reject H0 if or for
or for
11
Equivalent inferences in SLR

We reject the claim (i.e., H0 below) that the parameter in

SLR equals any value with 5% chance of committing Type I
error (or with 95% confidence)

If any of the following is correct:

1. The absolute value of the t-statistic (in Slide 11) is larger
than ;
2. The p-value computed from the t-statistic (in Slide 11) is
less than 0.05; or
3. lies outside the 95% CI for (in Slide 10)
12
Hypothesis test for the validity of SLR

• The validity of SLR can be done by testing

• Observed test statistic:

, or
• Reject H0 at level α if
p-value =
• Alternatively, it is equivalent to test if the true correlation of X and Y
(introduced in Module 9, Slide 13) is different from zero
• Observed test statistic:
(resulting in the same t value)
• Same rejection rule as above
13
Example 1: Diamond Cost (1/6)

Consider the data (as A scatterplot of 93 emerald-cut diamonds

shown below) from Module 9.
An apparent upward trend
Let’s perform a regression
analysis to predict a diamond
cost (Y) based on its weight (X)
with the aid of MS Excel, rather
than hand calculation
The dataset
Weight Cost
0.31 890
0.36 899
93 rows 0.45 1280
…
…

14
Example 1 (2/6) – regression outputs from MS Excel
Coefficient of determination

Estimate of SD of error
[out of scope]

95% CI for
t-statistic and p-value for
the 2-tailed test of H0: 15
Example 1 (3/6) – using regression outputs from MS
Excel
0.509
144.866

• The coefficient of determination R2 = 0.509 . That is, 51% of variation in

diamond cost can be explained by weight
• The error SD, , is estimated by SE of the regression
• The LS estimates are with respective
estimated SEs

16
Example 1 (4/6) – validity test

• 3rd & 4th columns give the t-statistic and p-value of the 2-sided test to
check if the slope (or intercept) is different from zero, e.g., from row 2:
1. The hypotheses:
2. The t-statistic:

3. The p-value:
4. Reject H0 at any reasonable significance level α
5. Conclude that is different from zero, i.e., weight is a significant
predictor of cost of a diamond 17
Example 1 (5/6) – t-test for

• To test if the mean change in diamond cost per carat increase in weight
is different from $2,500:
1. The hypotheses:
2. The t-statistic:

3. The p-value:
4. Do not reject H0 at 5% level
5. Conclude that 2500 is a plausible value for
18
Example 1 (6/6) – constructing CI

• A 95% CI for is

We are 95% confident that the true is between 2,152.5 and 3,242.0
• It is possible to use this CI to repeat the last 2-sided test (in Slide 17)
of whether the true is different from 2500:
• As 2500 lies inside the above CI for , we do not reject H0: = 2500
at 5% level, and we conclude that mean change in diamond cost
per carat increase in weight is no different from $2,500 19
Example 2: locating a franchise outlet (1/6)
Consider where to locate a new gas station
Q1: On a highway or another location with lower traffic?

Does traffic volume affect sales of gas?

Q2: We want to choose between two sites via predicting their sales
volumes
• One site is on a highway that averages 40,000 “drives-by” a day
• Another averages 32,000 “drives-by” a day

How much more gas can we expect to sell at the busier location?
20
Example 2 (2/6)
– a regression analysis approach
• We won’t find many stations that have 32,000 and 40,000 drives-by a
day, but possibly be able to find sales at stations with various levels
of traffic volume for fitting a regression model of Y on X
• Y: (average) sales of gas per day (in 1,000 gallons)
• X: (average) daily traffic volume (in 1,000 cars)
• Data: Collect sales data and traffic volume during a recent month at
n = 80 franchise outlets
• The intercept sets a baseline of gas sales that occur regardless of
traffic intensity (perhaps due to loyal customers)
• The slope measures the mean sales per passing car
21
Example 2 (3/6)
– Step 1: linearity in the scatterplot
• The scatterplot below suggests that the association is linear
• No obvious lurking variable – These 80 stations are in similar
areas with comparable prices

22
Example 2 (4/6) – Step 3: residual diagnostics
[to be discussed after Slide 50]
• Residual plot and histogram of residuals reveal:
• Residuals are evidently independent (D), have similar variances
(B), and are nearly normal (C) with zero mean (A)

23
Example 2 (5/6) – Steps 2,4,5

• The fitted equation is

• The 2-sided t-test for H0: is highly significant (p-value < 0.0001):
• Answering Q1: Traffic volume is a significant predictor of sales
• Coefficient of determination R2 = 54.86% :
• The LS equation explains ~55% of total variation in daily sales by
daily traffic volume
24
Example 2 (6/6)
– A CI for 8,000 x
• The parameter, 8,000 x , gives the difference in expected daily
gasoline sales (in gallons) between a station on a highway location
with 40,000 drives-by and another with 32,000 drives-by
• A 99% CI for is given by

• A 99% CI for 8,000 is given by

• We are 99% confident that a station located at a site with 40,000

drives-by per day sell on average from 1,392 to 2,392 more gallons
of gasoline daily than a location with 32,000 drives-by per day
25
26
Confidence interval for the
mean and prediction interval

27
Population & its Population of all costs
distribution (Recall from Module 9, Slide 23)
Sub-population Cost of all
An illustration using of diamonds
the Example: diamonds with
with weight weight X=0.4
• Sub-populations X=0.4
formed by different
(infinitely many) values
of weight (X) Independent

What is a CI for Cost

the mean in a
Sub-population
certain sub- of diamonds
with weight X=0.5
population? 28
 Pop. line:
A CI for the mean response LS line:

• The SLR states that the mean response for any given value x0 of
X as
• To estimate it, instead of using the point estimate given by the
fitted value

a 95% CI for the mean response at a given value x0 of X is given

by

29
Example 3: CI for the mean response CI for the mean, Slide 29

• Consider the “Locating a Franchise Outlet” Example (in Slide 20)

What is an interval estimate of the mean daily gasoline
sales for outlets with 40,000 drives-by per day?
• The LS equation is given by
• A 95% CI for the mean daily sales for outlets with traffic = 40 is sample mean &
sample variance
of traffic volume
of the 80 stations

• We are 95% confident that the mean daily gasoline sales for outlets
with 40,000 drives-by per day is between 7,791 and 8,471 gallons 30
Population & its Population of all costs
distribution (Recall from Module 9, Slide 23)
Sub-population
An illustration using of diamonds
Cost of all
the Example: diamonds with
with weight weight X=0.4
• Sub-populations formed X=0.4
by different (infinitely
many) values of weight (X)
Independent
What is an interval ynew
estimate for a random Cost
draw ynew from a certain Sub-population
of diamonds
sub-population? with weight X=0.5
31
The prediction interval (PI) CI for the mean, Slide 29

– an interval estimate for a new observation

• One powerful use of SLR is to predict the value ynew of a new
observation knowing its X-value given as xnew
• The SLR states:
Estimated by
(identical to in Slide 29 with )
• A 95% prediction interval (PI) for a new observation at a given
value xnew of X is given by

32
Example 4: managing natural resources (1/7)

• Overfishing endangered many fish species (e.g., bluefin tuna,

cod)
• Consider how number of days that fishing boats trapping
Dungeness crabs can operate in water (limited by the Canadian
government) and the size of catch are related based on n = 28
yearly data (1980-2007) near Vancouver Island
• Response Y: Dungeness weight (in 1,000 lbs)
• Predictor X: days of effort or number of limited boat-days
• The slope of a regression equation of Y on X indicates the
average catch per additional day a boat operates
33
Example 4 (2/7)
– Step 1: linear relationship
• Positive linear association between harvest weight and effort

Ignore the
PINK region at
the moment

34
Example 4 (3/7)
– Step 2: obtain the LS line

• A time plot of the residuals shows no clear pattern, suggesting

independence of residuals

35
Example 4 (4/7)
– Step 3: residual diagnostics
• The residual plot confirms independence and similar variances of
residuals
• Histogram of residuals is perhaps NOT bell-shaped, but with such a small
sample these data could be normally distributed

36
Example 4 (5/7)
– Step 4: days of effort as a significant predictor

• The fitted line/equation

explains 84% of variation in catch (Y) with days of effort (X)

• Reject H0: at any significant level as p-value < 0.0001 , and conclude
that days of effort (X) is a significant predictor of Dungeness weight
• The regression model of weight on days of effort provides a better
prediction of the catch weight than we can generate without knowing
days of effort
37
Example 4 (6/7)
– Step 6: a prediction interval for the catch
• Suppose that a 7,500 days of effort will be imposed in the coming year
• We can predict the catch (with xnew = 7500) by

• A 95% prediction interval (PI) of the catch (with xnew = 7500) is given by
PI for a new response, Slide 32

with , , and
• We are 95% confident that next year’s catch will be from 908.42 to
1,438.08 thousand pounds when 7,500 days of effort are to be imposed
38
Example 4 (7/7) – 95% prediction bands
• The 2 lines enclosing the region in PINK below are called 95% prediction
bands, constructed by joining endpoints of PI’s at every value of X

1,438.08
*
*
908.42 If SLR holds true,
* 95% of the data
* * should be within
A 95% pi of the catch (with xnew = 7500)
7,500
marked as * the 95% prediction
Obtain and mark PI’s as
* bands
* for all values of X
39
Procedure in applying SLR (Recall from Module 9)

1. Produce a scatterplot of y on x
• Check for linearity
2. Find the LS equation, and obtain residuals
3. Extremely
Perform regression diagnostics (Module 10) important!
4. Test the slope parameter (Module 10)
5. Interpret the LS equation
• intercept and slope estimates, R2
6. Make prediction of new observations (Module 10)

40
Regression Diagnostics
[Out of scope of exam]

41
All models are wrong:
Imperative to verify assumptions!
SLR: fit a regression model to some observed data
The data were generated by such a data generating
process (described in Module 9, Slide 21)
i.e., based on the model equation, all values of Y:

• Impossible to validate the above assumption – no real-world

data are based on any mathematical equation/model
• All models are wrong, but some are useful (Box, 1976)
• In case our fitted model fails to satisfy the imposed
assumptions, its credibility/validity would be even more
doubtful 42
Model assumptions/properties Module 9, Slides 21
on the random errors
Theory of SLR (in Module 9, Slide 21) states/assumes the
following for the (unobserved) random errors :
A. Zero Mean
• At any value of X, the errors have mean zero
B. Constant Variance (homoskedasticity)
• At any value of X, the errors have the same variance
C. Normality
• At any value of X, the errors are normally distributed
D. Independence
• Any error is independent of one another 43
Regression diagnostics
or residual analysis
• In practice, model assumptions are checked/verified through
the random errors in the observed data: for ,

• Such a task is referred to as regression diagnostics (or, as for

other models, residual analysis/diagnostics)
• concerns the (unobserved) error terms
• is performed with the observed/computed residuals in place
of the errors

44
Regression diagnostics via visual means

Use visual means with the (computed) residuals

from the LS/fitted equation:
1. Construct a residual plot (i.e., e VS x) and refer to next 2
slides for more discussion – check for (A) zero mean, (B)
constant variance, (C) outlier, (D) independence
2. [Optional] If the data are time series, construct a time plot of
the residuals (i.e., e VS time) – check for (A) zero mean, (B)
constant variance, (D) independence
3. Inspect the histogram of the residuals – check for (A) zero
mean, (C) normality
45
Regression diagnostics
– independence assumption
Independence e
e
x
x
Random pattern
Not Independence
e e
x x

46
Regression diagnostics
– constant variance assumption

e e

x x

Constant variance Non-constant variance

Vertical spread or scatter of Vertical spread or scatter of

residuals remains the same residuals increases with x
regardless of predictor value x

47
Example 5: plotting the residuals
Consider Example 1, in the residual plot (e VS x): Notice
1. Consistent vertical scatter throughout B. Constant Variance
2. Neither bends or outliers; no pattern D. Independence
A Residual Plot
It is indeed rather
subjective to say
whether there is
any pattern in a
residual plot.
A more objective
approach is to
compare it with
some scrambled
residual plots

48
Example 6: comparing with
scrambled residual plots
1. Shuffle e’s and randomly pair 2. Compare with the residual plot
with Weight data (x’s) to produce
(a few) scrambled residual
plots (which serve as
benchmarks of “no pattern”)

Difference?
NO pattern in the residual plot
No pattern
D. Independence
49
Example 7: checking
zero mean and normality
• Histogram of all residuals shows roughly a zero mean A. zero mean
• Histogram of all residuals is roughly bell-shaped C. normality
A Residual Plot

50
Takeaway
• Sampling distribution of the slope estimate
• Inference in regression
• Validity of the SLR model
• Confidence interval & hypothesis test of slope
• Confidence interval for the mean response
• Prediction interval for a new response
• Regression diagnostics [Out of scope of exam]

51