IST172 STATISTICS II LECTURE NOTES 8

The Classical Model

From earlier discussions we know that formally to determine the specific line that minimizes the
sum of squared residuals (or that satisfies the least squares criterion), we start by writing the
problem as

∑( i ( 0 )
n n n 2
y − β ̂ + β1̂ xi)
∑( i
y − yî ) =
2
e2
∑ i
Minimize =
i=1 i=1 i=1

We solve this minimization problem by taking the derivatives of formula with respect to β0̂ and β1̂ ,
setting them equal to 0, and solving for the values of the parameters that satisfy the optimality
condition (in this case, β0̂ and β1̂ ). Doing so yields the following formulas:

∑i=1 (xi − x̄) (yi − ỹ)

n
· y)
Cor(x,
β1̂ = =
∑i=1 (xi − x̄) Var x
n 2

and

β0̂ = ȳ − β1̂ x̄

ŷ = β0̂ + β1̂ x

Below is an example of a calculation of this sort:

Based on these entries, we see that our estimated slope coefficient is

 ∑ (yi − ȳ) (xi − x̄)
n
· 1,100.98
β1 = i=1 n = = 7,931.71
∑i=1 (x1 − x̄) 0,138694
β0̂ = ȳ − β1̂ x̄ = 4,550.24 − (7,931.71)(0,086) = 3,870.55

ŷ = β0̂ + β1̂ x = 3,870.55 + 7,931.71x

Superimposing this estimated sample regression function over the scatter diagram for 150-
observation city property crime and unemployment sample, we have the figure below .

Interpreting The Estimated Sample Regression Function

The estimated sample regression function allows us to answer our two important questions of
interest:

1. What is the marginal effect that a one-unit change in the independent variable is expected to
have on the value of the dependent variable?

2. What is the predicted value of the dependent variable that is likely to be associated with a given
value of the independent variable?

To answer these questions, we must be able to correctly interpret our estimated sample regression
function.

The estimated intercept β0̂ provides an estimate (based on the observed sample data) of the average
value that the dependent variable is expected to take for an observation with an observed value of 0
for the independent variable. For our 150-observation city property crime sample, we would expect
a city with an unemployment rate of 0 to observe 3,401.39 property crimes per 100,000 residents.
We offer one word of caution here when it comes to interpreting the estimated intercept. In practice,
because we do not often observe values of continuous independent variables that are close to 0 (the
lowest property crime rate in our 150-observation sample is only 1,926.53), the intercept often fails
to provide a reliable estimate of the value that y would take on if x were indeed equal to 0.
Moreover, for reasons that we discuss later, the estimated value often turns out to be nonsensical.
Fortunately, the estimated intercept rarely plays a central role in the types of economic analyses that
we wish to perform.

The estimated slope coefficient (β1̂ ) provides an estimate (based on the observed sample data) of the
average effect that each one-unit change in the independent variable is likely to have on the
dependent variable. For our 150-observation city property crime sample, we would expect each
one-point increase in the unemployment rate to increase the number of property crimes per 100,000
residents by an average of 15,460.59.

Note that we highlight the word average in the preceding explanation for an important reason. The
marginal effects that we estimate through linear regression analysis are averaged over all
individuals in the sample. Accordingly, it would be incorrect to interpret an estimated coefficient as
indicating that every observation in the sample observing a one-unit increase in the independent
variable will observe exactly the same β1̂ increase in the dependent variable. Instead, all
observations observing a one-unit increase in the independent variable will observe different
changes in the dependent variable, with the average of those changes being β1̂ .

Predicting Outcomes Based On Our Estimated Sample Regression Function

We can estimate the value of the dependent variable that is most likely to be associated with a given
value of the independent variable by inserting a specific value for the independent variable into the
estimated sample regression function and calculating a prediction for the likely value of the
dependent variable. As examples, for cities with unemployment rates of 9 percent (x = 0.09) in our
150-observation city property crime sample, the predicted number of property crimes per 100,000
residents would be

ŷ = 3,870.55 + 7,931 ⋅ 71 ⋅ 0,09 = 4,584.39

while for a cities with unemployment rates of 13 percent (x = 0.13), the predicted number of
property crimes per 100,000 residents would be ŷ = 3,870.55 + 7,931.71.0,13 = 4,901.67
Assess The Goodness-Of-Fit Of The Estimated Sample Regression Function

Given the process by which we calculate it, we commonly refer to calculating an estimated sample
regression function as determining the “best-fit” line through the observed sample data. This
terminology raises an important question: Just how well does the best-fit line actually fit the
observed data? Using the term best fit in relation to the process of determining the estimated sample
regression function can be somewhat misleading for the following reason.

The OLS estimated sample regression function is only the “best fit” in the sense that it has a smaller
sum of squared residuals than all other potential linear estimators, a fact that does not in any way
guarantee that this “best-fit” line actually fits the sample data well. As an example, consider the
scenarios in figure above in which the same estimated sample regression function describes two
different hypothetical samples. While the estimated sample regression function is the best-fit line in
both cases, it clearly does not describe the underlying relationship between y and x equally well.
Once we have calculated the estimated sample regression function for our observed sample data,
how do we know whether our estimated sample regression function is a “better fit” or a “worse fit”
to the observed sample data? We refer to this as a question of the goodness-of-fit of our estimated
sample regression function.

Measuring the Explained and Unexplained Variation in y

To start our discussion, consider that if no statistical relationship exists between y and x, then the
estimated slope coefficient is 0 (i.e., β1̂ = 0 ). In such a case, because y and x are statistically
unrelated, the value of x has no influence on the predicted value of y, and our best guess for each
value of y is the average value of y, or ȳ (i.e., if β1̂ = 0 ), then the predicted value of y is
yî = β0̂ + β1̂ xi = β0̂ + 0xi = β0̂ . In other words, if β1̂ = 0, we conclude that yî = β0̂ = ȳ . This fact
relates to the concept of explained and unexplained variation in the following way. Because there is
no joint variation between y and x, the total observed variation in y would equal the unexplained
variation in y, a situation that would be represented as shown in the figure below.

Suppose that a statistical relationship does exist between y and x and that, as a starting point, the
estimated sample regression function perfectly explains the total observed variation in y. In such a
case, each individual data point would fall exactly on the estimated sample regression function and
the total observed variation in y would be perfectly explained by the joint variation between y and x,
a situation that would be represented as shown in figure below left.

In more general terms, the difference between the estimated sample regression function and the no-
relationship predicted value of y (i.e., the difference between the blue and black solid line in figure
above right) represents the percentage of the total variation in y that is explained by the observed
joint variation between y and x. In other words, the explained variation in y is the difference
between the predicted value of y and the mean value of y (or yî − ȳ).

With this in mind, we turn to the most realistic scenario that we are likely to encounter—the case in
which the observed sample data points are distributed around the estimated sample regression
function rather than on the estimated sample regression function (Figure above right). This scenario
compares to the earlier examples in the following ways. First, because the observed joint variation
between y and x will not explain all of the variation in y, we observe both explained and
unexplained variation in y. To calculate these different types of variation, note that the observed
total variation in (yi − ȳ) is divided into two parts: the explained variation in y (yî − ȳ) and the
̂ We can also represent this fact visually as shown in figure below.
unexplained variation in (yi − y).

While the preceding depictions include only one data point, the samples that we are likely to
encounter in the real world will include multiple data points. As an example, as we will discuss in
detail later, the estimated sample regression function for our 150-observation city property crime
sample is depicted as shown in figure above.

The scenario in this figure differs from that in previous figures in the following way. Rather than
having calculated total, explained, and unexplained variations for only one observation, we have
them for all 150 observations. This makes it more difficult to interpret our calculated measures of
variation. Fortunately, we can convert the 150 individual values into one sample value by
aggregating from the individual level to the sample level. While a first impulse for this aggregation
might be to sum the calculated values across all of the individual observations, such an approach
will not work because the positive and negative deviations cancel so that the aggregated values
always sum to 0. Again, we solve this problem by squaring the value for each individual
observation before summing them up. In other words, we calculate our aggregate measures of the
total, explained, and unexplained variation present in our sample by (1) calculating the variation for
each individual observation in the sample, (2) squaring that calculated value for each individual
observation, and (3) summing the squared values across all n observations in the sample. Doing so
provides the following important values.

Our aggregate measure of the total variation present in the sample data is referred to as the total sum
of squares (TSS) and is calculated as
 n

∑( i
y − ȳ)
2
TSS =
i=1

Our aggregate measure of the explained variation present in the sample data is referred to as the
explained sum of squares (ESS) and is calculated as

∑( i
ŷ − ȳ)
2
ESS =
i=1

Our aggregate measure of unexplained variation present in the sample data is referred to as the
unexplained sum of squares (USS) and is calculated as

∑( i
y − ȳ)
2
USS =
i=1

From the preceding discussion, it should be clear that within the observed sample

TSS = ESS + USS

To calculate these values, we need to calculate the predicted value of y for each observation and
sample mean value of y before using those values to calculate the TSS, ESS, and USS. For our 150-
observation city property crime sample the calculations look like the table above. In words, based
on the preceding calculations, we conclude that for our sample of U.S. cities, the estimated TSS,
ESS, and USS are;

TSS = 335,294,649.93
ESS = 8,725,496.72
USS = 326,569,153.21
Unfortunately, because these measures depend on the chosen units of measurement, we are unable
to judge the goodness-of-fit of our estimated sample regression function by looking at these values
alone (i.e., it is impossible to tell whether 335,294,649.93 is a relatively large or relatively small
TSS). Instead, we must convert these numbers into values that are more meaningful.

Two Potential Measures of the Relative Goodness-of-Fit of Our Estimated Sample Regression
Function
Generally speaking, the estimated sample regression function is a better fit for the observed sample
data if (1) the aggregate explained variation in y is a larger percentage of the aggregate total
variation in y (i.e., the ESS is a larger percentage of the TSS ) or if (2) the aggregate unexplained
variation in y is relatively smaller (i.e., the USS is smaller). These facts form the basis for our two
preferred measures of the relative goodness-of-fit of our estimated sample regression function.

The Coefficient of Determination (R2)

Our first measure of the relative goodness-of-fit of an estimated sample regression function is the
percentage of the total observed variation in y (the TSS) that is explained by the observed joint
variation between y and x (the ESS). We refer to this value as the coefficient of determination, or
the R -squared (R2) of the estimated sample regression function. We calculate this value as

∑ (yî − ȳi)
2
ESS
R2 = =
TSS ∑ (yi − ȳi)
2

To make sense of this value, we need to consider the range of potential outcomes that can be
observed for this calculation. As long as the regression equation has an intercept, the calculated
values must fall between 0 and 1 because (1) calculated variations can never be negative, meaning
that the quotient of the calculated ESS and TSS values can never be less than 0, and (2) the
observed joint variation between y and x can never explain more than the total observed variation in
y, meaning that the ESS can never exceed the TSS and therefore the quotient of their values can
never exceed 1.

As is often the case, the lower and upper bounds of the range of possible outcomes have specific
interpretations. To see this, start by considering that the calculated R2 can only equal 0 if the
calculated ESS equals 0, an event that can only occur if there is no statistical relationship between y
and x (in which case all observed data points lie on the horizontal line y). Conversely, the calculated
R2 can only equal 1 if the calculated ESS equals the calculated TSS, an event that can only occur if
the observed joint variation between y and x perfectly explains the total variation in y (in which
case all observed data points lie on the line yî = β0̂ + β1̂ xi).

Because we are unlikely to ever observe a sample for which there is either no relationship or a
perfectly defined relationship between y and x, we need to focus on the interpretation of calculated
R2 values that fall between 0 and 1. As a general rule then, we interpret calculated R2 values closer
to 1 as indicating that the estimated sample regression function does a better job explaining the
underlying relationship between y and x, while we interpret calculated R2 values closer to 0 as
indicating that the estimated sample regression function does a poorer job explaining the underlying
relationship between y and x. For our 150-observation city property crime sample, the coefficient of
determination is calculated as

ESS 8,725,496.72
R2 = = = 0,026
TSS 335,294,849.93

We interpret this value as saying that within our sample of 150 cities, 2.60 percent of the total
observed variation in property crimes rates is explained by the observed joint variation between
property crime rates and unemployment rates. Is this value relatively large or relatively small?
Unfortunately, as we discuss next, there is no set decision rule for answering this question.

The Standard Error of the Estimated Sample Regression Function (sy∣x)

Our second measure of the relative goodness-of-fit of our estimated sample regression function is a
function of the aggregate distance that individual observations tend to fall from the estimated
sample regression function. We refer to this measure as the standard error of the sample regression

function, (sy∣x), and we calculate its value as

∑ (yi − yî )
2
USS
Sy∣x = =
n−k −1 n−k −1

The term in the numerator of formula above makes sense because the unexplained sum of squares
(the USS) represents the total vertical distance that each observed data point falls from the
estimated sample regression function. The term in the denominator, n - k - 1, is the number of
degrees of freedom of the estimated sample regression function, a value that indicates the number of
independent pieces of information that remain after the model is estimated. As the formula value
indicates, the number of degrees of freedom depends on two factors: the number of observations in
the sample (n) and the number of variables being estimated (k - 1). In this notation, k represents the
number of independent variables included in the estimated sample regression function and + 1
represents the intercept of the estimated sample regression function.

Accordingly, in the case of simple linear regression, because we are only considering one
independent variable, it follows that k = 1 and ( n - k - 1) = n - 2. A sample regression function
cannot be estimated if the number of independent variables (k) is greater than the number of
observations (n). More generally, the greater the difference between n and k, the more pieces of
information go into estimating the sample regression function. As such, we are faced with an
important trade-off between the number of observations and the number of independent variables.

In particular, looking at formula, it should be clear that the number of degrees of freedom is a
positive function of the number of observations (n) and a negative function of the number of
independent variables included in the estimated sample regression function (k), meaning that
increases in the sample size increase the number of degrees of freedom, while increases in the
number of independent variables decrease the number of degrees of freedom. The larger the n - k - 1
term is, the smaller the standard error of the regression. This implies that the more independent
pieces of information that we use to estimate the regression function, the better we do fitting a line
through the data.

As determined earlier, we know that the larger the calculated USS, the further the observed data
points tend to fall from the estimated sample regression function, while the smaller the calculated
USS, the closer the observed data points tend to fall to the estimated sample regression function.

Accordingly, as a general rule, the larger the calculated value of (sy∣x) , the worse the fit of the

estimated sample regression function, while the smaller the calculated value of (sy∣x) , the better

the fit of the estimated sample regression function.

For our 150-observation city property crime sample, the standard error of the sample regression
function is calculated as
 326,569,153.21
sy∣x = = 2,206548.53 = 1485.45
150 − 1 − 1

This calculation indicates that, on average, observed property crime rates per 100,000 citizens in
our 150-city sample fall 1,485.45 away from the estimated sample regression function. Again,
because the calculated value depends on the chosen unit of denomination, there is no definitive
answer to whether this calculated value is relatively small or relatively large.

The Assumptions of the Classical Model

The classical model of econometrics has nothing to do with ancient Greece or even the classical
economic thinking of Adam Smith. Instead, the term classical refers to a set of fairly basic
assumptions required to hold in order for OLS to be considered the “best” estimator available for
regression models. The Classical Assumptions must be met in order for OLS estimators to be the
best available.

The Classical Assumptions

I. The regression model is linear, is correctly specified, and has an additive error term.
II. The error term has a zero population mean.
III. All explanatory variables are uncorrelated with the error term.
IV. Observations of the error term are uncorrelated with each other (no serial correlation).
V. The error term has a constant variance (no heteroskedasticity).
VI. No explanatory variable is a perfect linear function of any other explanatory variable(s) (no
perfect multicollinearity).
VII. The error term is normally distributed (this assumption is optional but usually is invoked).
An error term satisfying Assumptions I through V is called a classical error term, and if assumption
VII is added, the error term is called a classical normal error term.

I. The regression model is linear, is correctly specified, and has an additive error term. The
regression model is assumed to be linear:

yi = β0 + β1x1i + β2 x2i + ⋯ + βK xKi + ϵi

The assumption that the regression model is linear does not require the underlying theory to be
linear. For example, an exponential function:

yi = e β0 x1β1e ϵi where e is the base of the natural log, can be transformed by taking the natural log of
both sides of the equation:

ln (Yi) = β0 + β1 ln (Xi) + ϵi
Y*i
= ln (Yi)
X*i
= ln (Xi)
Y*i
= β0 + β1 X*
i
+ ϵi

In Equation, the properties of the OLS estimator of the βs still hold because the equation is linear.
Two additional properties also must hold. First, we assume that the equation is correctly specified.
If an equation has an omitted variable or an incorrect functional form, the odds are against that
equation working well. Second, we assume that a stochastic error term has been added to the
equation. This error term must be an additive one and cannot be multiplied by or divided into any of
the variables in the equation.

II. The error term has a zero population mean. Econometricians add a stochastic (random) error
term to regression equations to account for variation in the dependent variable that is not explained
by the model. The specific value of the error term for each observation is determined purely by
chance. Probably the best way to picture this concept is to think of each observation of the error
term as being drawn from a random variable distribution. Classical Assumption II says that the
mean of this distribution is zero. That is, when the entire population of possible values for the
stochastic error term is considered, the average value of that population is zero. For a small sample,
it is not likely that the mean is exactly zero, but as the size of the sample approaches infinity, the
mean of the sample approaches zero. What happens if the mean doesn’t equal zero in a sample? As
long as you have a constant term in the equation, the estimate of β0 will absorb the non-zero mean.
In essence, the constant term equals the fixed portion of Y that cannot be explained by the
independent variables, and the error term equals the stochastic portion of the unexplained value of
Y.

III. All explanatory variables are uncorrelated with the error term. It is assumed that the
observed values of the explanatory variables are independent of the values of the error term. If an
explanatory variable and the error term were instead correlated with each other, the OLS estimates
would be likely to attribute to the X some of the variation in Y that actually came from the error
term. If the error term and X were positively correlated, for example, then the estimated coefficient
would probably be higher than it would otherwise have been (biased upward), because the OLS
program would mistakenly attribute the variation in Y caused by ϵ to X instead. As a result, it’s
important to ensure that the explanatory variables are uncorrelated with the error [Link]
Assumption III is violated most frequently when a researcher omits an important independent
variable from an equation.

IV. Observations of the error term are uncorrelated with each other. The observations of the
error term are drawn independently from each other. If a systematic correlation exists between one
observation of the error term and another, then OLS estimates will be less precise than estimates
that account for the correlation. For example, if the fact that the e from one observation is positive
increases the probability that the e from another observation also is positive, then the two
observations of the error term are positively correlated. Such a correlation would violate Classical
Assumption IV. In economic applications, this assumption is most important in time-series models.
In such a context, Assumption IV says that an increase in the error term in one time period (a
random shock, for example) does not show up in or affect in any way the error term in another time
period.

V. The error term has a constant variance. The variance (or dispersion) of the distribution from
which the observations of the error term are drawn is constant. That is, the observations of the error
term are assumed to be drawn continually from identical distributions. The alternative would be for
the variance of the distribution of the error term to change for each observation or range of
observations. In figure below, for example, the variance of the error term is shown to increase as the
variable Z increases; such a pattern violates Classical Assumption V. The actual values of the error
term are not directly observable, but the lack of a constant variance for the distribution of the error
term causes OLS to generate inaccurate estimates of the standard error of the coefficients. For
example, suppose that you’re studying the amount of money that the 50 states spend on education.
New York and California are more heavily populated than New Hampshire and Nevada, so it’s
probable that the variance of the error term for big states is larger than it is for small states. The
amount of unexplained variation in educational expenditures seems likely to be larger in big states
like New York than in small states like New Hampshire. The violation of Assumption V is referred
to as heteroskedasticity.
VI. No explanatory variable is a perfect linear function of any other explanatory variable(s).
Perfect collinearity between two independent variables implies that they are really the same
variable, or that one is a multiple of the other, and/or that a constant has been added to one of the
variables. That is, the relative movements of one explanatory variable will be matched exactly by
the relative movements of the other even though the absolute size of the movements might differ.
Because every movement of one of the variables is matched exactly by a relative movement in the
other, the OLS estimation procedure will be incapable of distinguishing one variable from the other.
This problem can be corrected easily by dropping one of the perfectly collinear variables from the
equation.

VII. The error term is normally distributed. Although we have already assumed that observations
of the error term are drawn independently (Assumption IV) from a distribution that has a zero mean
(Assumption II) and that has a constant variance (Assumption V), we have said little about the
shape of that distribution. Assumption VII states that the observations of the error term are drawn
from a distribution that is normal. This assumption of normality is not required for OLS estimation.
Its major application is in hypothesis testing and confidence intervals, which use the estimated
regression coefficient to investigate hypotheses about economic behavior. Even though Assumption
VII is optional, it’s usually advisable to add the assumption of normality to the other six
assumptions for two reasons:
1. The error term ϵi can be thought of as the sum of a number of minor influences or errors. As the
number of these minor influences gets larger, the distribution of the error term tends to approach the
normal distribution.
2. The t-statistic and the F-statistic are not truly applicable unless the error term is normally
distributed.
The Gauss–Markov Theorem and the Properties of OLS Estimators

The Gauss–Markov Theorem proves two important properties of OLS estimators. This theorem is
proven in all advanced econometrics textbooks, but for a regression user, it’s more important to
know what the theorem implies than to be able to prove it. The Gauss–Markov Theorem states that
given Classical Assumptions I through VI (Assumption VII, normality, is not needed for this
theorem), the Ordinary Least Squares estimator of βk is the minimum variance estimator from
among the set of all linear unbiased estimators of βk, for k= 0, 1, 2, …, K.

If an equation’s coefficient estimation is unbiased (that is, if each of the estimated coefficients is
produced by an unbiased estimator of the true population coefficient), then:

E (βk̂ ) = βk (k = 0,1,2,… . K )

Best means that each β k̂ has the smallest variance possible (in this case, out of all the linear
unbiased estimators of βk). An unbiased estimator with the smallest variance is called efficient, and
that estimator is said to have the property of efficiency. Since the variance typically falls as the
sample size increases, larger samples almost always produce more accurate coefficient estimates
than do smaller ones. The Gauss–Markov Theorem requires that just the first six of the seven
classical assumptions be met. What happens if we add in the seventh assumption, that the error term
is normally distributed? In this case, the result of the Gauss–Markov Theorem is strengthened
because the OLS estimator can be shown to be the best (minimum variance) unbiased estimator out
of all the possible estimators, not just out of the linear estimators. In other words, if all seven
assumptions are met, OLS is “BUE.”

Given all seven classical assumptions, the OLS coefficient estimators can be shown to have the
following properties:
1. They are unbiased. That is, E( β ̂ ) is β. This means that the OLS estimates of the coefficients are
centered around the true population values of the parameters being estimated.
2. They are minimum variance. The distribution of the coefficient estimates around the true
parameter values is as tightly or narrowly distributed as is possible for an unbiased distribution. No
other unbiased estimator has a lower variance for each estimated coefficient than OLS.
3. They are consistent. As the sample size approaches infinity, the estimates converge to the true
population parameters. Put differently, as the sample size gets larger, the variance gets smaller, and
each estimate approaches the true value of the coefficient being estimated.
4. They are normally distributed. The βŝ are N(β, VAR[ β ̂ ]). Thus various statistical tests based on
the normal distribution may indeed be applied to these estimates.