CTHAPTER TWO

SIMPLE LINEAR REGRESSION

10/12/2025
Monday, October 13, 2025
Regression Analysis by Nurye S. 1
 2.1 Introduction
 We consider the modeling between the dependent and one independent variable.

 When there are only one independent variables in the model, then the linear model is
termed as the multiple linear regression model.

 When there are more than one independent variables in the model, then the linear model
is termed as the multiple linear regression model.

 A regression equation which tries to relate one quantitative dependent variable to only
one independent variable is called a simple regression equation.

 In simple linear regression model settings we have a response variable Y (also known as
the dependent variable) and a regressor variable X (also known as the independent or
predictor variable).
10/12/2025 Regression Analysis by Nurye S. 2
 Cont.
 The simple linear regression model , that is, a model with a single regressor x that has a
relationship with a response y that is a straight line.

 In simple linear regression, we assume that there is perfect linear relationship between the
average value of the response variable at particular value of the predictor variable and the
predictor variable itself. This simple linear regression model is

where y is termed as the dependent or study variable and X is termed as independent or

explanatory variable.
 The terms 𝛽0 and 𝛽1 are unknown parameters of the model. The parameter 𝛽0 is termed as intercept
term and the parameter 𝛽1 is termed as slope parameter. These parameters are usually called as regression
coefficients.
10/12/2025 Regression Analysis by Nurye S. 3
 Cont.
 The parameter β0​ (the intercept) in a linear regression model represents the expected or
predicted value of the dependent variable (Y) when the independent variable (X) is equal to
zero.
 If the range of 𝑋 doesn’t include zero, then β0 has no practical interpretation.
 β1 (slope) is the rate of change in E(Y |X = x) for a unit change in X.
• The sign of β1 (slope) will determine the type of relationship between the response variable
and the independent variable.
• if β1 (slope) is positive then there is positive relationship between the two variables( as X
increases, Y also increases or vice versa)
• if β1(slope) is negative then there is negative relationship between the two variables( as X
increases, Y decreases or vice versa)
• if β1 (slope) is zero then there will not be relationship between the two variables

10/12/2025 Regression Analysis by Nurye S. 4

 Cont.
 Where 𝜀 is assumed to be a random error terms representing the discrepancy in the
approximation. It accounts for the failure of the model to fit the data exactly.

 The unobservable error component (𝜺) accounts for the failure of data to lie on the
straight line and represents the difference between the true and observed realization of y.
The errors are assumed to have mean zero and unknown variance σ 2 .

 Additionally, we usually assume that the errors are uncorrelated. This means that the
value of one error does not depend on the value of any other error.

 It is convenient to view the regressor x as controlled by the data analyst and measured
with negligible error, while the response y is a random variable. That is, there is a
probability distribution for y at each possible value for x. The mean of this distribution is
10/12/2025 Regression Analysis by Nurye S. 5
 Cont.

If

10/12/2025 Regression Analysis by Nurye S. 6

 Cont.
• Furthermore each observation of the response variable can be written mathematically as

Yi= 𝛽𝑜 + 𝛽1𝑋𝑖 + ℰ𝑖
 The subscript i indicates the particular observational unit, i = 1, 2. . . n, yi represents the ith
value of the response variable Y , Xi represents the ith value of the predictor variable X and
ℰ𝑖are called statistical (error terms) and are the differences between particular value of
the response variable and the average value of the response variable a given fixed value of
the predictor.

10/12/2025 Regression Analysis by Nurye S. 7

 Cont.
Assumptions of the simple linear regression
1. Assumption about the form of the model
• Linearity: The model should specify a linear relationship between dependent variable and
the predictor variable. In the regression context, linearity refers to the manner in which
the parameters and the error term enter the equation, not necessarily to the relationship
among the variables.
2. Assumptions about the error term
• Normality: The error term is a random variable and should have a normal distribution
with mean equal to zero and variance ( 𝛿 2 ) . Symbolically, this is shown
as ℰ − 𝑁(0, 𝛿 2 ).
• Homoscedasticity: The error variance (𝛿 2 ) is assumed to be constant. The variance
should not increase as as the value of the independent variable increases.
• Non autocorrelation: The error term should also have no correlation to each other. The
errors associated to different observations should be independent to each other. The
deviation of 𝑦𝑖 from the line should not affect the deviation of 𝑦𝑗 from the line. That is
cor (ℰ𝑖, ℰ𝑗) = 0
10/12/2025 Regression Analysis by Nurye S. 8
 Cont.
The consequence of the assumptions of the error term is
• For fixed value of 𝑋(𝑋 = 𝑥), 𝑌 is random variable and should have a normal
distribution with mean “𝛽𝑜 + 𝛽1𝑥 “and variance "𝛿 2 " and the variance of 𝑌 is the
same for all possible values of 𝑋. 𝑇ℎ𝑎𝑡 𝑖𝑠 𝑌 − 𝑁(𝛽𝑜 + 𝛽1𝑋, 𝛿 2 ).
• The 𝑌 values have to be statistically independent of one another. Another way of saying
this is that the observations are independent. Independency can be only be achieved by
taking random sample.
Why we make those assumptions? They make it possible for us to
 Develop measures of reliability for regression coefficients
 Test hypothesis about the association between Y and X: draw inferences.
 Small or minor violations of the underlying assumptions do not invalidate the inferences
or conclusions drawn from the analysis in a major way. Gross violations of the model
assumptions can, however, seriously distort conclusions. Consequently, it is important to
investigate the structure of the residuals and the data pattern through graphs.
10/12/2025 Regression Analysis by Nurye S. 9
 2.2 Estimation of the regression parameters
 Least square method
 Based on the available data, we wish to estimate the parameters 𝛽𝑜 and𝛽1.
 This is equivalent to finding the straight line that gives the best fit (representation) of the points in
the scatter plot of the response versus the predictor variable.
 We can estimate the parameters in the simple regression model using the popular least squares
method, which gives the line that minimizes the sum of squares of the vertical distances from each
point to the line. The vertical distances represent the errors in the response variable.
 This method is also known as the ordinary least squares estimation.
 The least squares principle for the simple linear regression model is to find the
estimates 𝑏0 𝑎𝑛𝑑 𝑏0 for 𝛽𝑜 and 𝛽1 respectively such that the sum of the squared distance
from actual value (yi) and the fitted value (𝑦𝑖 = 𝑏0 + 𝑏1 𝑥𝑖 ) reaches the minimum among all
possible
10/12/2025choices of regression coefficients 𝑏0Analysis
Regression 𝑎𝑛𝑑 by𝑏0Nurye
. S. 10
 Cont.
 That is to find the estimates of 𝑏0 𝑎𝑛𝑑 𝑏0 so that the square distance between the
actual value (yi) and the fitted value (yi) will be minimum.

 Mathematically, the least squares estimates of the simple linear regression model
( 𝑏0 𝑎𝑛𝑑𝑏0 ) are given by :

𝑦𝑖𝑥𝑖−𝑛𝑥𝑦 𝑆𝑥𝑦
= = 𝑆𝑥𝑥
𝑥𝑖 2 −𝑛𝑥 2

10/12/2025 Regression Analysis by Nurye S. 11

 Cont.
 Note that 𝑏𝑜 is a least square estimator of 𝛽𝑜 and have the same interpretation to it,
and 𝑏1 is also least square estimator of 𝛽1 and have the same interpretation to it.

 The fitted value of the simple linear regression is denoted as(𝑦𝑖 ) = 𝑏𝑜 + 𝑏1𝑥𝑖.

 The line of the fitted equation is called the sample line of best fit. Among the different
possible line that passes through points on scatter plot, the line of best fit is the one that
reduce sum of the squares of the residual.

 The difference between actual value (yi) and the fitted value (𝑦𝑖 ) is called residual
(𝑒𝑖 = 𝑦𝑖 − 𝑦𝑖).

10/12/2025 Regression Analysis by Nurye S. 12

 Cont.
Example: The following data shows the score of 12 students for Accounting and Statistics
examinations.
i. Fit a regression line of Statistics on Accounting using least square
estimates and interpret the coefficients
ii. Predict the score of Statistics if the score of accounting is 85.
Student 1 2 3 4 5 6 7 8 9 10 11 12
Acct(X) 74 93 55 41 23 92 64 40 71 33 30 71
Stat(Y) 81 86 67 35 30 100 55 52 76 24 48 87
Solution: First draw scatter plot for the data to see a kind of
relationship that exist between the two variables (score on accounting
and score on statistics).

10/12/2025 Regression Analysis by Nurye S. 13

 Cont.

• As you see from the scatter plot, it seems there is some linear
relationship between the variables. Therefore we can now perform
regression analysis for further investigation on the relationship
between the two variables.
10/12/2025 Regression Analysis by Nurye S. 14
 Cont.
i. Thus the fitted regression equation is given by
𝑛 = 12, 𝑋 = 57.25, 𝑌 = 61.75 , 𝑋 2 = 3277.5625
𝑋𝑖𝑌𝑖 = 48407, 𝑋𝑖 2 = 45,591, 𝑌𝑖 2 = 52,525

𝑦𝑖𝑥𝑖−𝑛𝑥 𝑦 48407−12 57.25 (61.75)

• 𝑏1 = = = 0.956
𝑥𝑖 2 −𝑛𝑥 2 45591−12(3277.5625)
• 𝑏𝑜 = 𝑌 − 𝑏1𝑋 = 61.75 − 0.956 57.25 = 7.02
Therefore the fitted regression equation is given by stat = 7.02 + 0.956 acc
Interpretation
• The constant coefficient is 7.02 and it is the average score of the students on statistics course when a student
score zero on accounting course. That is a student who gets zero on accounting course will score 7.2 on
average in statistics course.
• The coefficient associated to the independent variable (score on accounting) is 0.956 that is when the
students mark on accounting course increase by 1 mark, the average score of the students on statistics
course increased by 0.956 marks.
ii. Exercise
10/12/2025 Regression Analysis by Nurye S. 15
 Cont.
Exercise: The iodine value (x) is the amount of iodine necessary to
saturate a sample of 100 g of oil. The article’s authors fit the simple
linear regression model to this data, so let’s do the same.

Fit a regression line

10/12/2025 Regression Analysis by Nurye S. 16

 2.3. Desirable properties of the least-square estimator
 The least - square estimators 𝑏0 and 𝑏1 have several important properties.
[Link] estimators 𝑏𝑜 𝑎𝑛𝑑 𝑏1 are linear combinations of observations(𝑦1, 𝑦2, … 𝑦𝑛)
𝑦𝑖𝑥𝑖−𝑛𝑥𝑦 (𝑥𝑖−𝑥)𝑦𝑖
• For instance 𝑏1 = =
𝑥𝑖 2 −𝑛𝑥 2 (𝑥𝑖−𝑥 )2
𝑥𝑖−𝑥
• If we let 𝑐𝑖 = , 𝑏1 = 𝑐𝑖𝑦𝑖 which indicates that b1 is the linear
(𝑥𝑖−𝑥 )2
combination of the observations 𝑦𝑖.

10/12/2025 Regression Analysis by Nurye S. 17

 Cont.
2. Least square estimators are unbiased that is the least squares estimator 𝑏𝑜 is an unbiased
estimator of βo and the least squares estimator of 𝑏1 is also unbiased estimator of β1
Thus 𝐸(𝑏𝑜) = 𝛽𝑜 and 𝐸(𝑏1) = 𝛽1 (proof !!!)

10/12/2025 Regression Analysis by Nurye S. 18

 Cont.
3. The sum of the residuals in any regression model that contains an intercept 𝛽0 is always
zero, that is,

4. The sum of the observed values 𝑦𝑖 equals the sum of the fitted values 𝑦𝑖 or

5. The least-squares regression line always passes through the centroid [the point (y, x)] of the
data.

6. The sum of the residuals weighted by the corresponding value of the regressor variable
always equals zero, that is

[Link] sum of the residuals weighted by the corresponding fitted value always equals zero, that
is,
10/12/2025 Regression Analysis by Nurye S. 19
 Cont.
8. Minimum variance፡ The least square estimators have minimum variance and the variance
𝛿2 1 𝑋2
of 𝑏1 is given by and the variance of 𝑏𝑜 is given by ( + )𝛿 2
𝑆𝑥𝑥 𝑛 𝑆𝑥𝑥
𝛿2
𝑖. 𝑒. 𝑉𝑎𝑟(𝑏1) = (𝑝𝑟𝑜𝑜𝑓)
𝑋𝑖 2 −𝑛𝑥 2
1 𝑋2
𝑉𝑎𝑟(𝑏𝑜) = ( + )𝛿 2 (𝑝𝑟𝑜𝑜𝑓)
𝑛 𝑆𝑥𝑥

10/12/2025 Regression Analysis by Nurye S. 20

 Cont.

10/12/2025 Regression Analysis by Nurye S. 21

 Estimation of variance(𝝈𝟐 )
• In addition to estimating β0 and β1, an estimate of 𝛿 2 is required to test hypotheses and construct interval
estimates pertinent to the regression model.
• Since, the variance that used in the computation of the variance of least square estimators is unknown in most cases, it
needs to be estimated by using the sample data. Then, unbiased estimator of 𝛿 2 is denoted by 𝛿 2 and is given by
1 𝑛 𝑛
𝛿 2 = 𝑛−2 𝑖=1(𝑦𝑖 − 𝑦)2 where the quantity 𝑖=1(𝑦𝑖 − 𝑦)2 is called sum of squares of residual (SSres)

𝑆𝑆𝑟𝑒𝑠 = 𝑒𝑖 2 = (𝑦𝑖 − 𝑦)2

𝑆𝑆𝑟𝑒𝑠 = (𝑦𝑖 − 𝑏𝑜 + 𝑏1𝑥𝑖 )2

𝑆𝑆𝑟𝑒𝑠 = (𝑦𝑖 − 𝑏𝑜 − 𝑏1𝑥𝑖)2

𝑆𝑆𝑟𝑒𝑠 = (𝑦𝑖 − 𝑏𝑜 − 𝑏1𝑥𝑖)2 𝑏𝑢𝑡 𝑏𝑜 = 𝑌 − 𝑏1𝑋

𝑆𝑆𝑟𝑒𝑠 = (𝑦𝑖 − (𝑌 − 𝑏1𝑋) − 𝑏1𝑥𝑖)2

2
𝑆𝑆𝑟𝑒𝑠 = 𝑦𝑖 − 𝑌 + 𝑏1𝑋 − 𝑏1𝑥𝑖

𝑆𝑆𝑟𝑒𝑠 = ( 𝑦𝑖 − 𝑌) − 𝑏1(𝑥𝑖 − 𝑋 )2
10/12/2025 Regression Analysis by Nurye S. 22
 C ont.

10/12/2025 Regression Analysis by Nurye S. 23

 Cont.
Where is just the corrected sum of squares of the response observations, so

• Thus 𝛿 2 which is the ratio of sum of squares of the residual (SSRes) and the degree of freedom
(𝑛 − 2) is given by
2 𝑦𝑖 2 −𝑛𝑦 2 −𝑏12 ( 𝑋𝑖 2 −𝑛𝑥 2 𝑦𝑖 2 −𝑛𝑦 2 −𝑏1( 𝑥𝑖𝑦𝑖−𝑛𝑥𝑦))
𝛿 = or 𝛿 2
= and is sometimes called
𝑛−2 𝑛−2
mean squared residual (MSRes).
• The expected value of SSRes is E(SSRes)=(n − 2) 𝛿 2 , so an unbiased estimator of 𝛿 2 is

 The square root of 𝛿 2 is sometimes called the standard error of regression, and it has the same
units as the response variable y.
 The residual sum of squares has n − 2 degrees of freedom, because two degrees of freedom are
associated with the estimates 𝑏1 𝑎𝑛𝑑 𝑏𝑜 involved in obtaining 𝑦.
 Another view of choosing 𝑛 − 2 is that in the simple linear regression model there are n
observations and we lose two degrees of freedom while we estimate the two parameters in the
model.
10/12/2025 Regression Analysis by Nurye S. 24
 Cont.
Example: Estimate the SST ,SSRes , error variance ( 𝛿 2 ) or MSRes for the linear regression model in previous
students score data
Solution
SST = =52525-12*3813.0625=6768.25
• o or
=52525-12*3813.0625-0.956(48407-(12*57.25*61.75))=1047.3
The error variance is estimated by 𝛿 2 which is given by
1
𝛿2 = ( 𝑦 2 − 𝑛𝑦 2 − 𝑏1( 𝑥𝑦 − 𝑛𝑥 𝑦))
𝑛−2
Compute the following quantities from the data
𝑛 = 12, 𝑋 = 57.25, 𝑌 = 61.75 , 𝑏1 = 0.956, 𝑦 2 = 3813.0625
𝑋𝑖𝑌𝑖 = 48407, 𝑌𝑖 2 = 52,525

Substitute those values in the above formula

1
𝛿 2= (52,525 − 12(3813.0625) − 0.956(48407 − 12(57.25)(61.75)=104.73
12−2
Therefore the error variance is estimated to be 104.73
• 𝑉𝑎𝑟 𝑏1 =? 𝑎𝑛𝑑 𝑉𝑎𝑟 𝑏𝑜 =? (Exercise)
10/12/2025 Regression Analysis by Nurye S. 25