Simple Linear Regression Model with Distribution of

Error Terms Unspecified

Mean and Variance of Yi

Mean of Yi
 First-order regression model
o Yi = value of the response variable
o β0, β1 = regression coefficients  regression function
 β0: y-intercept of regression line o linear
 β1: slope of the regression line o regression coefficients:
 β0 + β1Xi: regression function  : average value of Y if X=0
o Xi = value of the predictor variable  : more important parameter
= known, fixed : increase/decrease in the
o εi = ith random error term with: mean of Y per unit increase/decrease in
X.

 Variance of Yi

= difference between the response Alternative Version of Simple Regression Line

value Yi and the regression line
Using the deviation of the predictor variable from its
β0+β1Xi.
mean, i.e. (Xi−X̅ ) as the predictor to be included in the
 Simple, linear in the parameter and in the IV regression model.

Then, an alternative version of the simple linear
regression model is given by

Yi = β0* + β1(Xi−X̅ ) + εi

 Where β0* = β0 + β1X̅

 This will provide interpretations on how Y

behave with respect to the deviation from the
average value of X.
 This will provide meaningful interpretations
for the intercept if the scope of the model does
not include X = 0

Justification of the Existence of the Error Terms 𝛆: Estimation of Regression Function

1. the effect of many factors (apart from the X’s) Least Squares Estimators of β0 and β1
omitted from the model which do affect the
response variable to some extent, and which
vary at random without reference to the
independent variables.
2. Over and above the total effect of all relevant
Properties of Least Squares Estimation
factors, there is a basic and unpredictable
element of randomness in responses which can 1. .
be characterized by the inclusion of a random 2. .
error term. 3.
3. ε accounts for errors in observations or 4. estimators b0 & b1: linear functions of the Yi’s
measurement in recording Y. 5. Gauss-Markov Theorem:
 The least squares estimators b0 and b1
are unbiased and have minimum variance
among all unbiased linear estimators.

Estimated Regression Function

Remark:
Given sample estimators b0 and b1 of the parameters in
the regression function E(Y)=β0+β1X, 1. Residuals can be computed from the observed
responses and fitted values;
Estimate the regression function as
 therefore, residuals are observable.
 error term, ε, in the regression model
is unobservable.
 : value of the estimated regression function at 2. Residuals play an important role in the
level K of X regression diagnosis, i.e. whether the
: point estimator of the mean response when regression model is appropriate for the data at
the level of K=X is X. hand.
: unbiased estimator of E(Y) and has the Estimation of Error Terms Variance 𝛔𝟐
minimum variance in the class of unbiased
linear estimators. Simple Linear Regression model: the variances of least
 E(Y): mean response squares estimators b0, b1 involve the variance σ2 of the
: mean of the probability distribution of Y error terms εi: unobservable, ⁖ there is a need to
corresponding to the level K of X estimate σ2 so that statistical inferences can be
established later.
 : fitted value for the ith case and
is different from Yi (observed value of Y) Estimating the variance of the error terms will provide
information on the variability of the probability
Residuals

𝑒𝑖: ith residual ei=Yi−Ŷi.

distribution of the response variable Y.

: difference between the observed value 𝑌𝑖 and the

At this point, no distributional assumptions were made
for these error terms except for:
corresponding fitted value i.  E(εi) = 0
Properties of the Residuals and the Fitted Regression  Var(εi) = σ2
Line  Cov(εi,εj) = 0 for i ≠ j

1. Sum of the residuals is zero. Error Mean Square

2. Sum of the squared residuals is a minimum

 requirement to be satisfied in deriving

the least squares estimators of the  Unbiased estimator of σ2
regression parameters. o where: SSE =
3. Sum of the observed values Yi equals the sum  variance of the residuals ei but unlike the usual
of the fitted values formula for variance with denominator of n-1,
4. Sum of the weighted residuals is zero when the above has n-2.
ith residual is weighted by the level of the ith o because for simple linear regression
independent variable. model, two parameters are being
5. sum of the weighted residual is zero when the estimated β0 and β1 which results to
ith residual is weighted by the fitted value of the loss of 2 degrees of freedom.
the ith response variable Alternative Computational Formulas for SSE
Normal Error Regression Model

to aid us in forming inferences such as confidence

intervals and statistical tests, we need to make more
distributional assumptions about εi.

Suppose that the error terms ε i are independent and

normally distributed with mean 0 and variance σ2.
Then, we define the normal error regression model as
follows: Yi=β0+β1Xi+εi where:

 εi = independent N(0,σ2) or εi~iid N(0,σ2) Estimation of Parameters by Method of Maximum

Likelihood
Remarks:
The maximum likelihood estimates of the simple linear
1. The uncorrelatedness of error terms in the
regression can be developed if we assume that the
previous regression function translates to the
error terms are normally distributed which translates to
independence of these error terms if we
Y having a normal distribution also. Given the sample
assume that they are normally distributed.
observations Y1,Y2,…, Yn, the
2. The same with our original model, Yi’s are
independent with Likelihood Function is given by:
 E(Yi) = β0 + β1Xi and Var(Yi) = σ2
3. The normality assumption for the error terms
is justifiable in many situations because the
error terms frequently represent the effect of
other factors omitted from the model that
affect Y and that vary at random without
reference to X. These error terms represent:
a. measurement errors in recording Y or In finding estimators of β0 and β1, we need to find its
b. other factors that may affect Y but are values that maximize the likelihood function L.
not explicitly stated in the model A simple linear regression model Y i = β0 + β1Xi + εi
because their effect on Y is negligible. with εi~N(0,σ2), the
4. If the error terms are normally distributed,
then it can be shown that Yi’s are also Maximum Likelihood Estimators of β0, β1, and σ2 :
normally distributed, i.e. Yi~iid N(E(Yi) =
β0+β1X, σ2).

Note:

1. The maximum likelihood estimators of β 0 and

β1 are the same estimators as those provided
 by the method of least squares. Thus,
maximum likelihood estimators have the same
properties as the least squares estimators.
2. Least squares estimators are good estimators
and reasonable even if the error terms are not
normally distributed.
3. The MLE of σ2 is a biased estimator of σ2.
The MSE is an unbiased estimator of σ2, then

is an unbiased estimator of σ2.

is an asymptotically unbiased estimator of

2
σ , which coincides with the classical theory of
MLE.