Mekdela Amba University

College of natural and computational science

Department of Statistics
Lecture Notes On
Regression analysis(Stat-M2041)
By: Birtukan F. ([Link]. in Statistics (Biostatistics))

Tulu Awulia, Ethiopia

November, 2019

1
 OUT LINES
Chapter -2: Simple Linear Regression
2.1. Introduction
2.2. Estimation of the regression parameters
2.3. Desirable properties of the least-square estimator
2.4. Inference about the intercept and the slope parameters
 Test of the significance of regression parameters

 Interval estimation of the regression parameters

2.5. Prediction of new observations
 Interval estimation of a predicted new observation

 Hypothesis testing about a predicted new observation

2.6. Covariance and correlation coefficient

 Interval estimation of a population correlation coefficient

 Hypothesis testing about a population correlation coefficient

[Link] Rank correlation coefficient

2
2. Simple Linear Regression
Analysis
2.1

introduction
When there is only one independent variable in the linear
regression model, the model is generally termed as simple linear
regression model.
 When there are more than one independent variables in the

model, then the linear model is termed as the multiple linear

regression model.
 Consider a simple linear regression model

y= + +
where y is termed as the dependent or study variable and X is
termed as independent or explanatory variable.
֍ The terms and are the parameters of the model.

֍ The parameter is termed as intercept term and the parameter is

termed as slope parameter.

3
 These parameters are usually called as regression
coefficients.
 The unobservable error component accounts for the
failure of data to lie on the straight line and
represents the difference between the true and
observed realization of y.
 We assume that is observed as independent and identically
distributed with mean zero and constant variance .
 y is viewed as a random variable with
E(y) = + and Var(y) =.
 Sometimes X can also be a random variable. In such a
case, we consider the conditional mean of y given X = x
as
and the

4
 When the values of , and are known, the model is
completely described.
 The parameters , and are generally unknown in
practice and ε is unobserved.
 The determination of the statistical model
y= + +
depends on the estimation of , and .
 In order to know the values of these parameters, n
pairs of observations (), i=1,...,n on (x , y) are
observed/collected and are used to determine these
unknown parameters.

5
2.2 Estimation of the regression
parameters:
 Among various methods of parameter estimation, the methods of

least squares and maximum likelihood are the popular methods

of estimation.
Least squares estimation
 Suppose a sample of n sets of paired observations (), i=1,...,n are

available. These observations are assumed to satisfy the simple

linear regression model and so we can write
(i=1,2,….,n)
 The principle of least squares estimates the parameters and is
minimizing the sum of squares of difference between the
observations and the line in the scatter diagram.
 When the vertical difference between the observations and the line

in the scatter diagram is considered and its sum of squares is

minimized to obtain the estimates of and , the method is known as
direct regression.

6
 No assumption is required about the form of probability
distribution of in deriving the least squares estimates.
 For the purpose of deriving the statistical inferences only, we
assume that ’s are random variable with
E()= 0, Var) = and Cov(, )= 0 for all ( i, j = 1, 2,..., ).
 This assumption is needed to find the mean, variance and other
properties of the least square estimates.
 The assumption that 's are normally distributed is utilized while
constructing the tests of hypotheses and confidence intervals of
the parameters.
 Based on these approaches, different estimates of and are
obtained which have different statistical properties.
 Among them the direct regression approach is more popular.
 Generally, the direct regression estimates are referred as the
least squares estimates or ordinary least squares estimates.

7
Direct regression method
 This method is also known as the ordinary least squares
estimation.
 Assuming that a set of n paired observations on (,), i=1, 2,...,n are
available which satisfy the linear regression model
y =+.
 So we can write the model for each observation as
+, (i =1,2,...,n) .
 The direct regression approach minimizes the sum of squares

with respect to and .

 The partial derivatives of with respect to is

8
and the partial derivatives of with respect to is

 The solutions of and are obtained by setting

and .
 The solutions of these two equations are called the direct
regression estimators, or usually called as the ordinary least
squares (OLS) estimators of and .
 This gives the ordinary least squares estimates of and of as

where .

9
 The fitted line or the fitted linear regression model is

 The predicted values are

(i=1,2,….,n).
 The difference between the observed value and the
fitted (or predicted) value is called as a residual.
 The residual is defined as

10
2.3 Desirable properties of the least-
square estimator
Unbiased
 property:
Note that and are the linear combinations of .
 Therefore .
 Note that and , So

 Thus is an unbiased estimator of .

 Next

 Thus is an unbiased estimator of .

11
Variances:
 Using the assumption that 's are independently distributed, the
variance of is

 The variance of is

 The , So

12
Covariance:
 The covariance between and is

 It can further be shown that the ordinary least squares

estimators and possess the minimum variance in the
class of linear and unbiased estimators.
 So they are termed as the Best Linear Unbiased
Estimators (BLUE).
 Such a property is known as the Gauss-Markov
theorem which is discussed later in multiple linear
regression model.

13
Residual sum of squares:
 The residual sum of squares is given as

14
Estimation of
• The estimator of is obtained from the residual sum of squares.
• Assuming that is normally distributed, it follows that has a
distribution with (n-2) degrees of freedom, so

• Thus using the result about the expectation of a chi-square random

variable, we have

• Thus an unbiased estimator of is

• Note that has only (n-2) degrees of freedom. The two degrees of
freedom are lost due to estimation of and .
• Since depends on the estimates and , so it is a model dependent
estimate of .

15
Estimate of variances of and
 The estimators of variances of and are obtained by replacing by its
estimate follows:

 It is observed that since so

 In the light of this property, can be regarded as an estimate of
unknown (i= 1,...,n).
 This helps in verifying the different model assumptions on the basis
of the given sample ( ),i= 1,2,...,n.

i. The fitted line always passes through (, ).

16
2.4 Inference about the intercept and the
slope parameters
Testing of hypotheses and confidence interval estimation
for slope parameter:
 Now we consider the tests of hypothesis and confidence interval
estimation for the slope parameter of the model under two cases, i.e.,
when is known and when is unknown.
Case 1: When is known:
 Consider the simple linear regression model

=+ (i=1, 2,...,n).
It is assumed that 's are independent and identically distributed and follow
N(0,).
 First we develop a test for the null hypothesis related to the slope

parameter
H0:, where is some given constant.

17
 Assuming to be known, we know that = and
is a linear combination of normally distributed s. So

so the statistic can be constructed

which is distributed as N(0,1) when is true.

 A decision rule to test can be framed as follows:

 Reject if ||> where is the percent points on normal distribution.

 Similarly, the decision rule for one sided alternative hypothesis can

also be framed.

18
 The 100 (1-)% confidence interval for can be obtained using the
statistic as follows:

 So the 100 (1-)% confidence interval for is

where is the percent points on normal distribution.

19
Case 2: When is unknown:
 When is unknown then we proceed as follows. We know that

and

 Further, and are independently distributed.

 This result also follows from the result that under normal distribution,
the maximum likelihood estimates, i.e., the sample mean (estimator
of population mean) and the sample variance (estimator of
population variance) are independently distributed, so and are also
independently distributed.

20
 Thus the following statistic can be constructed:

which follows a t -distribution with (n - 2) degrees of freedom, denoted

as , when is true.
 A decision rule to test can be framed as follows:

 Reject if ||> where is the percent points of the t -distribution with (n-

2) degrees of freedom.
 Similarly, the decision rule for one sided alternative hypothesis can

also be framed.
 The 100 (1-)% confidence interval for can be obtained using the

statistic as follows:

21
 Consider

 So the 100 (1-)% confidence interval for is

where is the percent points of the t -distribution with (n-2) degrees of

freedom.

22
Testing of hypotheses and confidence interval
estimation for intercept term:
 Now, we consider the tests of hypothesis and confidence interval
estimation for intercept term under two cases, when is known and
when is unknown.
Case 1: When is known:
 , where is some given constant.
 where to be known, then the result that E()= , and is a linear

combination of normally distributed random variables, the following

statistic

which is distributed as N(0,1) when is true.

23
 A decision rule to test can be framed as follows:
 Reject if ||> where is the percent points on normal distribution.
 Similarly, the decision rule for one sided alternative hypothesis can
also be framed.
 The 100 (1-)% confidence interval for can be obtained using the
statistic as follows:

 So the 100 (1-)% confidence interval for is

24
Case 2: When is unknown:
 When is unknown, then the following statistic is constructed

which follows a t -distribution with (n - 2) degrees of freedom, denoted

as , when is true.
 A decision rule to test is as follows:

 Reject if ||> where is the percent points of the t -distribution with (n-

2) degrees of freedom.
 Similarly, the decision rule for one sided alternative hypothesis can

also be framed.
 The 100 (1-)% confidence interval for can be obtained using the

statistic as follows:

25
 Consider

 So the 100 (1-)% confidence interval for is

26
Test of hypothesis for
 We have considered two types of test statistics for testing the
hypothesis about the intercept term and slope parameter- when is
known and when is unknown.
 While dealing with the case of known , the value of is known from

some external sources like past experience, long association of the

experimenter with the experiment, past studies etc.
 In such situations, the experimenter would like to test the hypothesis

like
against where is specified.
 The test statistic is based on the result .

 So the test statistic is

under
The decision rule is to reject if or

27
Confidence interval for
 A confidence interval for can also be derived as follows.
 Since , thus consider

 The corresponding 100(1-α)% confidence interval for is

28
2.5 Prediction of new
observations
An important use of linear regression model is to predict the
average and actual values of study variable.
 The term prediction of value of study variable corresponds to
knowing the value of E(y) (in case of average value) and value of y
(in case of actual value) for a given value of explanatory variable.
 We consider both the cases.
Case 1: Prediction of average value
 Under the linear regression model , the fitted model is where and
are the OLS estimators of and respectively.
 Suppose we want to predict the value of E(y) for a given value of .
Then the predictor is given by

29
Predictive bias
 Then the prediction error is given as

 Then

 Thus the predictor is an unbiased predictor of E(y).

30
Predictive variance:
 The predictive variance of is

Estimate of predictive variance

 The predictive variance can be estimated by substituting by as

31
Prediction interval estimation:
 The predictor is a linear combination of normally distributed random
variables, so it is also normally distributed as

 So if is known, then the distribution of

is N(0,1). So the 100(1-α)% prediction interval is obtained as

which gives the prediction interval for as

32
 When is unknown, it is replaced by and in this case the sampling
distribution of

is t -distribution with (n-2) degrees of freedom, i.e., .

 Note that the width of prediction interval is a function of .

 The interval width is minimum for and widens as || increases.

 This is expected also as the best estimates of y to be made at x -

values lie near the center of the data and the precision of estimation
to deteriorate as we move to the boundary of the x -space.

33
Case 2: Prediction of actual
value
 If is the value of the explanatory variable, then the
actual value predictor for y is

 The true value of y in the prediction period is given by

where indicates the value that would be drawn from the

distribution of random error in the prediction period.
 Note that the form of predictor is the same as of average

value predictor but its predictive error and other

properties are different. This is the dual nature of
predictor.

34
Predictive bias:
 The predictive error of is given by

=(
 Thus we found that

 which implies that is an unbiased predictor of .

35
Predictive variance
 Because the future observation is independent of , the predictive
variance of is
P

36
Estimate of predictive variance
 The estimate of predictive variance can be obtained by replacing by
its estimate as
P

Prediction interval:
 If is known, then the distribution of

is N(0,1). So the 100(1-α)% prediction interval is obtained as

37
 which gives the prediction interval for as

When is unknown, then

follows a t -distribution with (n-2) degrees of freedom, i.e., .

 The 100(1-α)% prediction interval for is obtained as follows:

38
which gives the prediction interval for as

 The prediction interval for is wider than the prediction interval for
because the prediction interval for depends on both the error from
the fitted model as well as the error associated with the future
observations.

39
2.6 Covariance and correlation
coefficient
Suppose we have observations on n subjects consisting of a
dependent or response variable Y and an explanatory variable X.
 We wish to measure both the direction and the strength of the

relationship between Y and X.

 Two related measures, known as the covariance and the correlation

coefficient.
Covariance:
 Covariance is a measure of how two variables are related.

 Variance tells us how a single variable varies but covariance tells

us how two variables vary together.

 The covariance between Y and X, is given as

indicates the direction of the linear relationship between Y and X.

40
 If then there is a positive relationship between Y and X, but
 if then the relationship is negative.
 Unfortunately, does not tell us much about the strength of such a
relationship because it is affected by changes in the units of
measurement.
 For example, we would get two different values for the if we report
Y and/or X in terms of thousands of dollars instead of dollars.
 To avoid this disadvantage of the covariance, we standardize the
data before computing the covariance.
 To standardize the Y data, we first subtract the mean from each
observation then divide by the standard deviation.

41
Correlation coefficient:
 The covariance between the standardized X and Y data is known as
the correlation coefficient between Y and X and is given by

 Equivalent formulas for the correlation coefficient are

 Thus, can be interpreted either as the covariance between the

standardized variables or the ratio of the covariance to the standard
deviations of the two Variables.

42
 The correlation coefficient is symmetric, that is,

 Unlike , is scale invariant, that is, it does not change if we change

the units of measurements.
 Furthermore, satisfies

 These properties make the a useful quantity for measuring both the
direction and the strength of the relationship between Y and X.
 The magnitude of measures the strength of the linear relationship
between Y and X.
 The closer is to 1 or -1, the stronger is the relationship between Y
and X.
 The sign of indicates the direction of the relationship between Y and
X.

43
 That is, implies that Y and X are positively related. Conversely,
 , implies that Y and X are negatively related.
 Note, however, that does not necessarily mean that Y and X are not
related.
 It only implies that they are not linearly related because the
correlation coefficient measures only linear relationships.
 In other words, the can still be zero when Y and X are nonlinearly
related.
 Although is a useful quantity for measuring the direction and the
strength of linear relationships, it cannot be used for prediction
purposes, that is, we cannot use to predict the value of one variable
given the value of the other.
 Furthermore, measures only pairwise relationships.
 Regression analysis, however, can be used to relate one or more
response variable to one or more predictor variables. It can also be
used in prediction.

44
Coefficient of determination
 Goodness of fit of regression model
 It can be noted that a fitted model can be said to be good when
residuals are small.
 Since is based on residuals, so a measure of quality of fitted model
can be based on .
 When intercept term is present in the model, a measure of
goodness of fit of the model is given by
 This is known as the coefficient of determination.
 This measure is based on the concept that how much variation in
y’s stated by is explainable by and how much unexplainable part is
contained in .
 The ratio describes the proportion of variability that is explained by
regression in relation to the total variability of y.

45
 The ratio describes the proportion of variability that is
not covered by the regression.
 It can be seen that

where is the simple correlation coefficient between x and

y.
 Clearly 0 ≤ ≤1, so a value of closer to one indicates the

better fit and value of closer to zero indicates the poor

fit.

46
2.7 Rank correlation
coefficient
The Karl Pearson’s coefficient of correlation cannot be used in
cases where the direct quantitative measurement of the
phenomenon under study is not possible.
 In such situations we can use the Spearman’s rank correlation
coefficient.
 The spearman’s rank correlation coefficient, denoted by , measures
the correlation between two paired samples of ranked data.
 This correlation coefficient is applied to the ranks in two paired
samples (not to the original scores).
 The formula for computing rank correlation by this method is:

47