Chapter Two; Regression Analysis

 The term regression was introduced by Francis Galton.

 Regression is probably the single most important tool at the

econometrician’s disposal.
 It is concerned with the study of the dependence of one variable, the

dependent variable, on one or more other variables, the explanatory

variable/s with the view to estimating/ and or predicting the
(population mean or average value).
 Cont.…
Regression analysis

Simple linear regression Multiple linear regression

 Terminology and notation
y x

Dependent variable Independent variable

Explained variable Explanatory variable

Response variable Control variable

Predicted variable Predicator variable

Regressand variable Regressor variable

 Cont.…

Single-equation Regression Models

 In single-equation regression models, one variable, called the

dependent variable, is expressed as a linear function of one or more

other variables, called the explanatory variables.
 In such models it is assumed implicitly that causal relationships, if

any, between the dependent and explanatory variables flow in one

direction only, namely, from the explanatory variables to the
dependent variable.
 Cont.…

 A model in which the dependent variable is expressed as a linear

function of only a single explanatory variable.

 E.g.,

Multiple Regression Model

 A model in which there are more than one explanatory variables.

 The dependent variable is explained by more than one explanatory

(independent) variables.
E.g.,
 Cont..

 The above equations are regression equation of Y on X.

 Since U is a random variable, Y is also a random variable.

 We can disaggregate the two components as follows
Y = 0 + 1X + U That is,
[variation in Y] = [systematic variation] + [random variation]

 In this regard it is essential to know what the term linear really

means, for it can be interpreted in two different ways. These

are, Linearity in the variables and Linearity in parameters
Reasons for the existence of error term in the model

 Omission of other variables

 Measurement error

 Randomness in human behavior

 Imperfect specification of the model

 Poor proxy variables

 Population and sample regression functions

 Imagine a hypothetical society with a total population of 20

families.
 Suppose divide these 20 families into 5 groups of approximately the

same income and examine the consumption expenditure of families

in each of these income groups.
 If we are interested in studying the relationship between
consumption expenditure, and family income.
 Cont.…

X 100 140 180 220 260

Y

65 75 95 105 115
70 80 100 110 120
75 85 105 115 125
80 90 110 120 130

 From the table we can construct conditional mean and

unconditional mean.
 Cont..

 Unconditional mean = E(Y) = 98.5

 Conditional mean when income level 100 = E(Y/X=100) = 72.5) .

Exercise: compute the conditional mean when the income is 140,

180, 220 and 260.
 What is the expected value of consumption expenditure of a

family?
 What is the expected value of consumption expenditure of a family

whose monthly income is, 220?

 Cont..

The conditional mean increases as X increases

 The above figure shows Conditional distribution of expenditure for

various levels of income
 The line in the figure is known as the population regression line or,
more generally, the population regression curve.
 Cont.…

 Geometrically, a population regression curve is simply the locus of

the conditional means or expectations of the dependent variable for

the fixed value of the explanatory variables.

 Consumption expenditure does not necessarily increase as income

level increases we incorporate the error term. That is,

 If we take the expected value in both sides

 Cont.…

 = since the mean of ui is zero

 In most practical situations, however, what we have is a sample of

Y values corresponding to some fixed X's.

 Therefore, our main task must be to estimate the population

regression function on the basis of the sample information.

 The regression function based on a sample collected from the

population is called sample regression function (SRF).

 Cont.…
Sample regression function
 Cont.…

 The sample regression function (which is a counterpart of the PRF

stated earlier) may be written as:

+, where and estimator of , = estimator of ,= estimator of .
 Population regression function:
 On the basis of population function the sample counterpart is
=+ since +
= ++
 The Method of Ordinary Least Squares
 The critical question now
1. Is SRF is an approximation of the PRF?
2. Can we devise a rule or a method that will make this
approximation as “close” as possible?
 In other words, how should the SRF be constructed so that is as

“close” as possible to the true 1 and is as “close” as possible to the

true 2 even though we never know the true 1 and 2.

 The method of ordinary least squares has some very attractive

statistical properties that have made it one of the most powerful and
popular method of regression analysis.
 Cont..
 Recall the two-variable (Y and X) PRF

 However, the PRF is not directly observable. Hence, we estimate it

from the SRF. That is,
= ++
=+
Where is the estimated (conditional mean) value of Yi

=
 This shows that , (the residuals) are simply the difference between

the actual and the estimated Y values.

 Cont.….

 Choose the SRF in such a way that the sum of the residuals  =

() is as small as possible.

 And to minimize taking both side squaring
 Cont.…

 Using the concept of partial derivatives one can minimize and set it

equal to zero and solving for &.

,
⟹n+…………eq(1)
⟹=+ …….eq(2)
 Cont.…

 Substituting the first eq. in place of of the second eq. we get ,

=
 Example

1. Based on the following consumption expenditure and income

data of the families
a. Compute the slope and intercept estimator (estimate the
consumption model).
b. How would you interpret the slope parameter?

Obs. 1 2 3 4 5
Yi 65 80 100 115 130
Xi 100 140 180 220 260
 Solution

Observation
1 65 100 4225 10000 6500
2 80 140 6400 19600 11200
3 100 180 10000 32400 18000
4 115 220 13225 48400 25300
5 130 260 16900 67600 33800
Sum 490 900 50750 178000 94800
, =180, =98, =, , Then = 0.4125 and = 23.75
b. When income increase by one unit (dollar or birr) then,
consumption expenditure increase by 0.4125 unit (dollar or birr).
 The classical linear regression model (CLRM) Assumptions

 Linear regression model is based on certain assumptions; some of

the assumptions are related to Ui and the relationship between Ui

and the explanatory variables, other assumptions are related to the
relationship between the explanatory variables themselves.
 There are 10 assumption in CLRM

Assumption 1: Linear regression model: - the regression model is

linear in the parameters.
Assumption 2: X (explanatory) values are fixed in repeated
sampling. Values taken by the regressor X are considered fixed in
repeated samples
 Cont.…

Assumption 3: Ui is a random real variable. Each value has a certain

probability of being assumed by U in any particular instance.
Assumption 4: Zero mean of the disturbance term. The mean or
expected value of the disturbance term Ui is zero.
Symbolically, we have: = 0.
Assumption 5: Homoscedasticity assumption which implies that the
variance of Ui is constant.
Symbolically, Var(=
 If the variance of the error term is not constant, heteroscedasticity

problem will occur in our regression analysis.

 Cont.….

Assumption 6: No autocorrelation between the disturbances. Given

any two U values, Xi and Xj (i  j), the correlation between any two

Ui and Uj (i  j) is zero.
 This implies that the error term committed for the i th observation is

independent of the error term committed for the jth observation.

 What if i=j

Assumption 7: Zero covariance between Ui and Xi or E(UiXi) = 0.

That is, the error term is independent of the explanatory variable(s).
 Cont.…

Assumption 8: the regression model is correctly specified. This

means that we have included all the important regressions explicitly in
the model and that its mathematical form is correct.
Assumption 9: The number of observations, n must be greater than
the number of parameters to be estimated.
 Properties of Least Squares Estimators: The Gauss-Markov
Theorem

 To understand this theorem, we need to consider the best linear

unbiasedness property of an estimator.

 The OLS estimator is said to be best linear unbiased estimator
(BLUE) of i if the following hold:
1. Linear Estimator. It is linear, that is, a linear function of a random
variable, such as the dependent variable Y in the regression model.
2. Unbiased estimator: an estimator is said to be unbiased if its
average or expected value, E(), is equal to the true value, .
3. Minimum Variance estimator (or best estimator) An estimator is
best when it has the smallest variance as compared with any other
estimate obtained from econometric methods.
 Precision or Standard Errors of Least Squares Estimates

 Since the data are likely to change from sample to sample, the
estimates will change. Therefore, what is needed is some measure
of “reliability” or precision of the estimators and .
Var(= =
Var(=
Se(
Se()= =
Cov( ) = -
 Example
1. Compute the variance and standard error
of the estimators for the previous example.
Solution
Var(0.0000625
Var( 2.225
Se() = 0.0079
Se() = 1.49
= = since they are unbiased estimator
 Correlation coefficient and coefficient of determination (r 2)

 Coefficient of determination (r2) is a measure of “goodness of fit”.

 The coefficient of determination (r2) is a summary measure that
tells how well the sample regression line fits the data.
To compute r2
= + or in deviation form = + squaring both sides and summing
over the sample

 In other words total sum of square (TSS) is equal to explained

sum of square (ESS) plus residuals sum of squares (RSS).
 Symbolically, TSS = ESS + RSS
Total variation = explained variation + unexplained variation
 Cont..
 Note that because OLS estimators minimizes the sum of
squared residuals (i.e., the unexplained variation) it
automatically maximizes r2.

or 1 =
 r2 measures the proportion or percentage of the total variation

= or ⟹
in Y explained by the regression model.
 Cont.…

Since =
=
Two properties of r2 may be noted
1. It is a non-negative quantity. This is because we
are dealing with sum of squares
2. Its limit are 0  r2  1.
If r2 = 1 means a perfect fit, that is = Yi for each i (or
alternatively Ui2 = 0)
If r2 = 0 means there is no relationship between the
regression and the regressor whatsoever.
 Coefficient of correlation

 A quantity closely related to but conceptually very much

different from .
 It measure the degree of association between two variables.
 It is a measure of linear association or linear dependence only.
 It can be computed either from:

or from its definition

 Cont..

Some of the properties of Coefficient of correlation are as

follows
1. It can be positive or negative
2. It lies between the limits of −1 and +1; that is, −1 ≤ r ≤ 1.
3. It is symmetrical in nature
Exercise
4. Compute TSS, ESS, RSS, coefficient of determination
and coefficient of correlation from the previous example.
5. Compute the variance of the error term
6. Obtain the variance and standard error of the estimators
 Solution

1. TSS = 2, 730, ESS = 2,722.5, RSS = 7.5,

= 0.9973 and r = 0.9986.
2. = 2.5
3. Var(0.00015625
Var( 5.5624987
Se() = 0.0125
Se() = 2.358
 Hypotheses testing

 Is a given observation or finding compatible with some stated

hypothesis or not?
 The word “compatible,” as used here, means “sufficiently” close to

the hypothesized value so that we do not reject the stated hypothesis.

 In the language of statistics, the stated hypothesis is known as the

null hypothesis and is denoted by the symbol H0.

 The null hypothesis is usually tested against an alternative

hypothesis denoted by H1.

 Hypothesis testing is concerned with developing rules or procedures

for deciding whether to reject or not reject the null hypothesis.

 Cont.…

There are two approaches for devising such rules, namely, test of

significance and confidence interval .

 Hypothesis Testing: The Test-of-significance Approach

 A test of significance is a procedure by which sample results are

used to verify the truth or falsity of a null hypothesis.

 The decision to accept or reject H0 is made on the basis of the value

of the test statistic obtained from the data at hand.

 Cont..
Steps in the Test-of-significance Approach
[Link] the t-statistic value

2. Obtain the t-critical value from the table; Critical t values) obtained
from the t table for (α/2) level of significance and n − 2 df,
3. Decision
(If t-statistic > t-critical reject Ho and accept H1 and the reverse is
true).
 A statistic is said to be statistically significant if the value of the test
statistic greater than the critical value.
 A test is said to be statistically insignificant if the value of the test
statistic less than the critical value.
 Exercise

Given from the table (tα/2, n-2) df is 3.182 and test the significance

level for the estimators for the pervious example using significance
approach.
Solution
For t-statistic value is 33, the value greater than t-critical value and

implies that accept H1 & reject Ho ( is statistically significant or

income has significant effect on consumption expenditure).
For t-statistic value is 10.07 and which is greater than t-critical

value and implies that accept H1 & reject Ho ( is statistically

significant).
Hypothesis Testing: The Confidence-interval Approach

Steps
 Construct 100(1 − α)% confidence interval for

)
 Example in the consumption–income postulate that

H0: = 0.3
H1:
 Construct a 100(1 − α)% confidence interval for .

If the under H0 falls within this confidence interval, do not reject

H0, but if it falls outside this interval, reject H0.

 Exercise

[Link] the pervious example construct 95% confidence interval for the

estimators and test the significance level.

Solution
95% CI for is from 0.3727 - 0.4522; under H0 is not falls within

the 95 confidence interval. Therefore, reject H0 & accept H1.

95% CI for is from 16.24 - 31.25; under H0 is not falls within
the 95 confidence interval. Therefore, reject H0 & accept H1.