Regression Analysis Techniques Help Uncover

Relationships Between Variables

Regression analysis is used to:
◦ Predict the value of a dependent variable based on the value of
at least one independent variable.
◦ Explain the impact of changes in an independent variable on
the dependent variable.
Dependent variable: the variable we wish to
predict or explain.
Independent variable: the variable used to predict
or explain the dependent
variable.

ABHISHEK VASHISHTH | IIM SHILLONG

Preliminary Analysis – Scatter Plots

A scatter plot can be used to:

◦Visualize the relationship between X and Y
variables.
◦Help suggest a starting point for regression
analysis.

ABHISHEK VASHISHTH | IIM SHILLONG

 Types of Relationships
Y Y
Linear
Relationships

X X

Y Y
Y
Curvilinear
Relationships

X X X

Y Y
No
Relationship

X X
ABHISHEK VASHISHTH | IIM SHILLONG
Simple Linear Regression Model

Only one independent variable, X.

Relationship between X and Y is
described by a linear function.
Changes in Y are assumed to be related
to changes in X.

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression Model

Population Random
Population Independent Error
Slope
Y intercept Variable term
Coefficient
Dependent
Variable

Yi  β0  β1Xi  ε i
Linear component Random Error
component

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression Model

Y Yi  β0  β1Xi  ε i
Observed Value
of Y for Xi

εi Slope = β1
Predicted Value Random Error
of Y for Xi
for this Xi value

Intercept = β0

Xi X
ABHISHEK VASHISHTH | IIM SHILLONG
Simple Linear Regression
Equation (Prediction Line)
The simple linear regression equation provides an
estimate of the population regression line.

Estimated
(or predicted) Estimate of Estimate of the
Y value for the regression regression slope
observation i intercept

Value of X for

Ŷi  b0  b1Xi
observation i

ABHISHEK VASHISHTH | IIM SHILLONG

 The Least Squares Method
b0 and b1 are obtained by finding the values that
minimize the sum of the squared differences
between Y and Ŷ :
min  (Yi Ŷi )2  min  (Yi  (b0  b1Xi ))2

ABHISHEK VASHISHTH | IIM SHILLONG

Interpretation of the
Slope and the Intercept

b0 is the estimated mean value of Y when the

value of X is zero.

b1 is the estimated change in the mean value

of Y as a result of a one-unit increase in X.

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression
Example
A real estate agent wishes to examine the relationship
between the selling price of a home and its size
(measured in square feet).

A random sample of 10 houses is selected.

◦ Dependent variable (Y) = house price in $1,000s.
◦ Independent variable (X) = square feet.

ABHISHEK VASHISHTH | IIM SHILLONG

Simple Linear Regression
Example: Data
House Price in $1000s Square Feet
(Y) (X)
245 1,400
312 1,600
279 1,700
308 1,875
199 1,100
219 1,550
405 2,350
324 2,450
319 1,425
255 1,700

ABHISHEK VASHISHTH | IIM SHILLONG

Simple Linear Regression Example: Scatter Plot
House price model: Scatter Plot

450
400
House Price ($1000s)

350
300
250
200
150
100
50
0
0 500 1000 1500 2000 2500 3000
Square Feet

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression Example: Excel
Output
Regression Statistics
Multiple R 0.76211 The regression equation is:
R Square 0.58082
Adjusted R Square 0.52842 house price  98.24833  0.10977 (square feet)
Standard Error 41.33032
Observations 10

ANOVA
df SS MS F Significance F
Regression 1 18934.9348 18934.9348 11.0848 0.01039
Residual 8 13665.5652 1708.1957
Total 9 32600.5000

Coefficients Standard Error t Stat P-value Lower 95% Upper 95%

Intercept 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386
Square Feet 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580

ABHISHEK VASHISHTH | IIM SHILLONG

Simple Linear Regression Example: JMP &
Minitab Output

The regression equation is

Price = 98.2 + 0.110 Square Feet

Predictor Coef SE Coef T P

Constant 98.25 58.03 1.69 0.129
Square Feet 0.10977 0.03297 3.33 0.010

S = 41.3303 R-Sq = 58.1% R-Sq(adj) = 52.8%

Analysis of Variance

Source DF SS MS F P
Regression 1 18935 18935 11.08 0.010
Residual Error 8 13666 1708
Total 9 32600

ABHISHEK VASHISHTH | IIM SHILLONG

Simple Linear Regression Example: Graphical
Representation

House price model: Scatter Plot and Prediction Line

450
400
House Price ($1000s)

350
Slope
300
250
= 0.10977
200
150
100
50
Intercept 0
= 98.248 0 500 1000 1500 2000 2500 3000
Square Feet

house price  98.24833  0.10977 (square feet)

ABHISHEK VASHISHTH | IIM SHILLONG
 Simple Linear Regression Example:
Interpretation of bo

house price  98.24833  0.10977 (square feet)

b0 is the estimated mean value of Y when the value of X is zero (if X = 0 is in
the range of observed X values)
Because a house cannot have a square footage of 0, b0 has no practical
application

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression Example:
Interpreting b1
house price  98.24833  0.10977 (square feet)

b1 estimates the change in the mean value

of Y as a result of a one-unit increase in X.
Here, b1 = 0.10977 tells us that the mean value of a
house increases by 0.10977($1,000) = $109.77, on
average, for each additional one square foot of size.

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression
Example: Making Predictions
Predict the price for a house
with 2,000 square feet:

house price  98.25  0.1098 ([Link].)

 98.25  0.1098(2,000)
 317.85

The predicted price for a house with 2,000

square feet is 317.85($1,000s) = $317,850
ABHISHEK VASHISHTH | IIM SHILLONG
 Simple Linear Regression
Example: Making Predictions
When using a regression model for prediction, only predict
within the relevant range of data
Relevant range for
interpolation
450
400
House Price ($1000s)

350
300
250
200
150
100
50 Do not try to
0 extrapolate
0 500 1000 1500 2000 2500 3000
beyond the range
Square Feet
of observed X’s
ABHISHEK VASHISHTH | IIM SHILLONG
 Computing b0 & b1

For small data sets, b0 & b1 can be calculated using a hand calculator
SSXY
b1  b0  Y  b1 X
SSX n n

n n
(  X i )(  Yi )
Where: SSXY   ( X i  X )(Yi  Y )   X iYi  i 1 i 1

i 1 i 1 n
n

n n
(  X i )2
SSX   ( X i  X )   X i2  2 i 1

i 1 i 1 n
n n

Y i X i
Y i 1
and X  i 1
n n

ABHISHEK VASHISHTH | IIM SHILLONG

 Computing b0 & b1 -- Home Prices
House House Price in $1000s (Y) Square Feet (X) X2 Y2 XY

1 245 1,400 1,960,000 60,025 343,000

2 312 1,600 2,560,000 97,344 499,200
3 279 1,700 2,890,000 77,841 474,300
4 308 1,875 3,515,625 94,864 577,500
5 199 1,100 1,210,000 39,601 218,900
6 219 1,550 2,402,500 47,961 339,450
7 405 2,350 5,522,500 164,025 951,750
8 324 2,450 6,002,500 104,976 793,800
9 319 1,425 2,030,625 101,761 454,575
10 255 1,700 2,890,000 65,025 433,500
Total 2865 17,150 30,983,750 853,423 5,085,975
n n

n
(  X i )(  Yi )
( 17 ,150 )( 2,865 )
SSXY   X iYi  i 1 i 1
 5 ,085 ,975   172,500
i 1 n 10
n
(  X i )2
n
17 ,150 2
SSX   X i2  i 1
 30,983,750   1,571,500
i 1 n 10
n n

Y i
2,865 X i
17 ,150
Y i 1
= =286.5 and X  i 1
  1,715
n 10 n 10
ABHISHEK VASHISHTH | IIM SHILLONG
 Computing b0 & b1 -- Home Prices (Con’t)

n n

n
(  X i )(  Yi )
( 17 ,150 )( 2 ,865 )
SSXY   X iYi  i 1 i 1
 5 ,085 ,975   172 ,500
i 1 n 10
n
(  X i )2
n
17 ,150 2
SSX   X  i
2 i 1
 30 ,983,750   1,571,500
i 1 n 10
n n

Y i
2 ,865 X i
17 ,150
Y  i1
= =286.5 and X  i1
  1,715
n 10 n 10

SSXY 172 ,500

b1    0.1098
SSX 1,571,500

b0  Y  b1 X  286.5  0.1098 * 1,715  98.193

ABHISHEK VASHISHTH | IIM SHILLONG
 Measures of Variation
DCOVA
Total variation is made up of two parts:

SST  SSR  SSE

Total Sum of Regression Sum Error Sum of
Squares of Squares Squares

SST   ( Yi  Y )2 SSR   ( Ŷi  Y )2 SSE   ( Yi  Ŷi )2

where:
Y = Mean value of the dependent variable
Yi = Observed value of the dependent variable
Yˆi = Predicted value of Y for the given Xi value
ABHISHEK VASHISHTH | IIM SHILLONG
 Measures of Variation

SST = total sum of squares (Total Variation.)

◦ Measures the variation of the Yi values around their mean Y.

SSR = regression sum of squares (Explained Variation.)

◦ Variation attributable to the relationship between X and Y.

SSE = error sum of squares (Unexplained Variation.)

◦ Variation in Y attributable to factors other than X.

ABHISHEK VASHISHTH | IIM SHILLONG

 Measures of Variation
Y
Yi  
SSE = (Yi - Yi )2 Y
_
SST = (Yi - Y)2

Y  _
_ SSR = (Yi - Y)2 _
Y Y

Xi X
ABHISHEK VASHISHTH | IIM SHILLONG
Excel Output Of The Measures Of
Variation

SST = SSR + SSE

32,600.5000 = 18,934.9348 + 13,665.5652

ABHISHEK VASHISHTH | IIM SHILLONG

 Coefficient of Determination, r2
The coefficient of determination is the portion of the
total variation in the dependent variable that is
explained by variation in the independent variable.
The coefficient of determination is also called r-
square and is denoted as r2.

SSR regression sum of squares

r 
2

SST total sum of squares

note: 0  r2  1
ABHISHEK VASHISHTH | IIM SHILLONG
 Examples of Approximate
r2 Values
Y

Perfect linear relationship

between X and Y.
X
r2 =1
Y 100% of the variation in Y is
explained by variation in X.

X
r2 =1
ABHISHEK VASHISHTH | IIM SHILLONG
 Examples of Approximate
r2 Values
Y
0 < r2 < 1

Weaker linear relationships

between X and Y.
X
Some but not all of the
Y
variation in Y is explained
by variation in X.

X
ABHISHEK VASHISHTH | IIM SHILLONG
 Examples of Approximate
r2 Values

r2 = 0
Y
No linear relationship
between X and Y.

The value of Y does not

X depend on X. (None of the
r2 = 0
variation in Y is explained
by variation in X.)

ABHISHEK VASHISHTH | IIM SHILLONG

Simple Linear Regression Example: Coefficient
of Determination, r2 in Excel
SSR 18,934.9348
Regression Statistics
r2    0.58082
Multiple R 0.76211 SST 32,600.5000
R Square 0.58082
Adjusted R Square 0.52842 58.08% of the variation in
Standard Error 41.33032 house prices is explained by
Observations 10
variation in square feet.
ANOVA
df SS MS F Significance F
Regression 1 18934.9348 18934.9348 11.0848 0.01039
Residual 8 13665.5652 1708.1957
Total 9 32600.5000

Coefficients Standard Error t Stat P-value Lower 95% Upper 95%

Intercept 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386
Square Feet 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580

ABHISHEK VASHISHTH | IIM SHILLONG

Coefficient of Determination, r2 in JMP &
Minitab
SSR 18,934.9348
r2    0.58082
SST 32,600.5000

The regression equation is

Price = 98.2 + 0.110 Square Feet

Predictor Coef SE Coef T P

Constant 98.25 58.03 1.69 0.129
Square Feet 0.10977 0.03297 3.33 0.010

S = 41.3303 R-Sq = 58.1% R-Sq(adj) = 52.8%

Analysis of Variance

Source DF SS MS F P
Regression 1 18935 18935 11.08 0.010
Residual Error 8 13666 1708
Total 9 32600

ABHISHEK VASHISHTH | IIM SHILLONG

Sum Of Squares Computational Formulae

ABHISHEK VASHISHTH | IIM SHILLONG

Standard Error of Estimate
The standard deviation of the variation of observations around the
regression line is estimated by:

SSE
 (Yi  Yˆi ) 2
i 1
S YX  
n2 n2
Where
SSE = error sum of squares
n = sample size

ABHISHEK VASHISHTH | IIM SHILLONG

 Simple Linear Regression Example:
Standard Error of Estimate in Excel
Regression Statistics
Multiple R 0.76211
R Square 0.58082
S YX  41.33032
Adjusted R Square 0.52842
Standard Error 41.33032
Observations 10

ANOVA
df SS MS F Significance F
Regression 1 18934.9348 18934.9348 11.0848 0.01039

Residual 8 13665.5652 1708.1957

Total 9 32600.5000

Coefficients Standard Error t Stat P-value Lower 95% Upper 95%

Intercept 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386
Square Feet 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580

ABHISHEK VASHISHTH | IIM SHILLONG

Standard Error of Estimate in JMP & Minitab
S YX  41.33032

The regression equation is

Price = 98.2 + 0.110 Square Feet

Predictor Coef SE Coef T P

Constant 98.25 58.03 1.69 0.129
Square Feet 0.10977 0.03297 3.33 0.010

S = 41.3303 R-Sq = 58.1% R-Sq(adj) = 52.8%

Analysis of Variance

Source DF SS MS F P
Regression 1 18935 18935 11.08 0.010
Residual Error 8 13666 1708
Total 9 32600

ABHISHEK VASHISHTH | IIM SHILLONG

Comparing Standard Errors
SYX is a measure of the variation of observed
Y values from the regression line.

Y Y

small SYX X large SYX X

The magnitude of SYX should always be judged relative to the

size of the Y values in the sample data.
i.e., SYX = $41.33K is moderately small relative to house prices in
the $200K - $400K range
ABHISHEK VASHISHTH | IIM SHILLONG
Assumptions of Regression L.I.N.E
Linearity:
◦ The relationship between X and Y is linear.
Independence of Errors:
◦ Error values are statistically independent.
◦ Particularly important when data are collected over a period
of time.
Normality of Error:
◦ Error values are normally distributed for any given value of X.
Equal Variance (also called homoscedasticity):
◦ The probability distribution of the errors has constant
variance.

ABHISHEK VASHISHTH | IIM SHILLONG

 Residual Analysis
ei  Yi  Ŷi
The residual for observation i, ei, is the difference between its
observed and predicted value.
Check the assumptions of regression by examining the
residuals:
◦ Examine for linearity assumption.
◦ Evaluate independence assumption.
◦ Evaluate normal distribution assumption.
◦ Examine for constant variance (homoscedasticity) for all levels of X.

Graphical Analysis of Residuals

◦ Can plot residuals vs. X.

ABHISHEK VASHISHTH | IIM SHILLONG

 Residual Analysis for Linearity
Y Y

x x
residuals

residuals
x x

Not Linear

ABHISHEK VASHISHTH | IIM SHILLONG
Linear
 Residual Analysis for
Independence
Cyclical Pattern:
Not Independent
 No Cyclical Pattern
Independent
residuals

residuals
X
residuals

ABHISHEK VASHISHTH | IIM SHILLONG

Checking for Normality
Examine the Stem-and-Leaf Display of the Residuals.
Examine the Boxplot of the Residuals.
Examine the Histogram of the Residuals.
Construct a Normal Probability Plot of the Residuals.

ABHISHEK VASHISHTH | IIM SHILLONG

 Residual Analysis for Normality
When using a normal probability plot, normal
errors will approximately display in a straight line.

Percent
100

0
-3 -2 -1 0 1 2 3
Residual
ABHISHEK VASHISHTH | IIM SHILLONG
 Residual Analysis for Equal Variance

Y Y

x x
residuals

residuals
x x

Non-constant variance  Constant variance

ABHISHEK VASHISHTH | IIM SHILLONG
 Residual Analysis: Checking For Linearity
RESIDUAL OUTPUT
Predicted
House Price Model Residual Plot
House Price Residuals
80
1 251.9232 -6.9232
2 273.8767 38.1233 60

3 284.8535 -5.8535 40

Residuals
4 304.0628 3.9371 20
5 218.9928 -19.9928 0
6 268.3883 -49.3883 0 1000 2000 3000
-20
7 356.2025 48.7975
-40
8 367.1793 -43.1793
-60
9 254.6674 64.3326
Square Feet
10 284.8535 -29.8535

Linear Model Assumption Is Appropriate

ABHISHEK VASHISHTH | IIM SHILLONG
 Residual Analysis: Checking For Independence
(con’t.)

RESIDUAL OUTPUT Residuals vs Observation Number

Predicted 80

House Price Residuals

60
1 251.9232 -6.9232
2 273.8767 38.1233 40

3 284.8535 -5.8535
4 304.0628 3.9371 20

5 218.9928 -19.9928
0
0 2 4 6 8 10 12
6 268.3883 -49.3883
7 356.2025 48.7975 -20

8 367.1793 -43.1793
-40
9 254.6674 64.3326
10 284.8535 -29.8535 -60

Independence Assumption Is Appropriate

ABHISHEK VASHISHTH | IIM SHILLONG
 Residual Analysis: Checking For Normality (con’t.)

RESIDUAL OUTPUT Normal Probability Plot For

Predicted House Prices Residuals
House Price Residuals 80
1 251.9232 -6.9232 60
2 273.8767 38.1233
40
3 284.8535 -5.8535

Residuals
20
4 304.0628 3.9371
0
5 218.9928 -19.9928
6 268.3883 -49.3883 -20

7 356.2025 48.7975 -40

8 367.1793 -43.1793 -60
9 254.6674 64.3326 -2 -1 0 1 2 3
Z Value
10 284.8535 -29.8535

Normality Assumption Is Appropriate

ABHISHEK VASHISHTH | IIM SHILLONG
 Residual Analysis: Checking For Constant Variance
(con’t.)

RESIDUAL OUTPUT
Residuals vs Predicted Value
Predicted
80
House Price Residuals
1 251.9232 -6.9232 60

2 273.8767 38.1233 40
3 284.8535 -5.8535
20
4 304.0628 3.9371
5 218.9928 -19.9928 0

6 268.3883 -49.3883 -20

7 356.2025 48.7975
-40
8 367.1793 -43.1793
9 254.6674 64.3326 -60
200 220 240 260 280 300 320 340 360 380

10 284.8535 -29.8535

Constant Variance Assumption Is Appropriate

ABHISHEK VASHISHTH | IIM SHILLONG
 Autocorrelation:

Used when data are collected over time to

detect if autocorrelation is present.
Autocorrelation exists if residuals in one
time period are related to residuals in
another period.

ABHISHEK VASHISHTH | IIM SHILLONG

Autocorrelation
Autocorrelation is correlation of the errors (residuals) over time.

Time (t) Residual Plot

15
 Here, residuals show a 10
5
cyclical pattern, not
random. Cyclical Residuals 0
-5 0 2 4 6 8
patterns are a sign of -10
positive autocorrelation. -15
Time (t)

 Violates the regression assumption that

residuals are random and independent.
ABHISHEK VASHISHTH | IIM SHILLONG
 Inferences About the Slope
The standard error of the regression slope coefficient (b1) is estimated by:

S YX S YX
Sb1  
SSX  i
(X  X ) 2

where:
Sb1 = Estimate of the standard error of the slope.

SSE = Standard error of the estimate.

S YX 
n2
ABHISHEK VASHISHTH | IIM SHILLONG
Inferences About the Slope:
t Test t test for a population slope:
◦ Is there a linear relationship between X and Y?

Null and alternative hypotheses:

◦ H0: β1 = 0 (no linear relationship)
◦ H1: β1 ≠ 0 (linear relationship does exist)

Test statistic :
where:
b1  β 1
t STAT  b1 = regression slope
coefficient
Sb
1 β1 = hypothesized slope

d.f.  n  2 Sb1 = standard

error of the slope
ABHISHEK VASHISHTH | IIM SHILLONG
Inferences About the Slope:
t Test Example
House Price Estimated Regression Equation:
Square Feet
in $1000s
(x)
(y)
house price  98.25  0.1098 ([Link].)
245 1400
312 1600
279 1700
308 1875 The slope of this model is 0.1098
199 1100
219 1550
Is there a relationship between the
405 2350 square footage of the house and its
324 2450 sales price?
319 1425
255 1700

ABHISHEK VASHISHTH | IIM SHILLONG

Inferences About the Slope:
t Test Example H :β =0 0 1

From Excel output: H1 : β 1 ≠ 0

Coefficients Standard Error t Stat P-value
Intercept 98.24833 58.03348 1.69296 0.12892
Square Feet 0.10977 0.03297 3.32938 0.01039

b1 Sb1

b1  β 1 0.10977  0
t STAT    3.32938
Sb 0.03297
1

ABHISHEK VASHISHTH | IIM SHILLONG

Inferences About the Slope:
t Test Example
H0 : β 1 = 0
Test Statistic: tSTAT = 3.329
H1 : β 1 ≠ 0

d.f. = 10- 2 = 8

/2=.025 /2=.025
Decision: Reject H0.

There is sufficient evidence

Reject H0
-tα/2
Do not reject H0
tα/2
Reject H0 that square footage affects
0
-2.3060 2.3060 3.329 house price.

ABHISHEK VASHISHTH | IIM SHILLONG

 Inferences About the Slope:
t Test Example
H0 : β 1 = 0
From Excel output: H1 : β 1 ≠ 0
Coefficients Standard Error t Stat P-value
Intercept 98.24833 58.03348 1.69296 0.12892
Square Feet 0.10977 0.03297 3.32938 0.01039

p-value
Decision: Reject H0, since p-value < α.
There is sufficient evidence that
square footage affects house price.

ABHISHEK VASHISHTH | IIM SHILLONG

 F Test for The Slope

F Test statistic: MSR

FSTAT 
MSE
where
SSR
MSR   SSR
1
SSE
MSE 
n 2

where FSTAT follows an F distribution with 1 numerator and (n – 2)

denominator degrees of freedom.

ABHISHEK VASHISHTH | IIM SHILLONG

 F-Test for The Slope
Excel Output

Regression Statistics
Multiple R 0.76211
MSR 18,934.9348
R Square 0.58082 FSTAT    11.0848
Adjusted R Square 0.52842 MSE 1,708.1957
Standard Error 41.33032
Observations 10 With 1 and 8 degrees p-value for
of freedom the F-Test
ANOVA
df SS MS F Significance F
Regression 1 18934.9348 18934.9348 11.0848 0.01039
Residual 8 13665.5652 1708.1957
Total 9 32600.5000

ABHISHEK VASHISHTH | IIM SHILLONG

F-Test for The Slope
JMP & Minitab Output
FSTAT The regression equation is

Price = 98.2 + 0.110 Square Feet

Predictor Coef SE Coef T P

Constant 98.25 58.03 1.69 0.129
Square Feet 0.10977 0.03297 3.33 0.010

S = 41.3303 R-Sq = 58.1% R-Sq(adj) = 52.8%

Analysis of Variance

Source DF SS MS F P
Regression 1 18935 18935 11.08 0.010
Residual Error 8 13666 1708
Total 9 32600
ABHISHEK VASHISHTH | IIM SHILLONG
 F Test for The Slope

H0 : β 1 = 0 Test Statistic:
H1 : β 1 ≠ 0 MSR
FSTAT   11.08
 = .05 MSE
df1= 1 df2 = 8
Decision:
Critical Reject H0 at  = 0.05.
Value:
F = 5.32
 = .05 Conclusion:
There is sufficient evidence that
0 F house size affects selling price.
Do not Reject H0
reject H0
F.05 = 5.32
ABHISHEK VASHISHTH | IIM SHILLONG