CHAPTER 2: SIMPLE LINEAR

REGRESSION MODEL
Econometrics lecture summary

1. The Core Idea: What Is Regression?

Regression analysis is the study of the dependence of one variable, the dependent variable (Y), on one or more
other variables, the explanatory variable(s) (X), with a view to estimating or predicting the mean value of Y on the
basis of the known values of X.

The classic example comes from Galton's law of universal regression: a son's height depends on his father's height,
but a tall father does not guarantee a son of exactly the same height. The average height of sons born to fathers of a
given height tends to move toward the average height of the population as a whole — this tendency is the origin of
the term "regression."

In the family income and consumption example (Table 2.1), for each level of weekly income X there are many
different families, each with a different consumption expenditure Y — but on average, expenditure rises as income
rises. This is the idea of the

conditional mean — the average value of Y corresponding to each specific value of X.

2. The Population Regression Function (PRF)

We define the population regression function as:

E(Y | Xᵢ) = f(Xᵢ)

In the simple linear regression model, this function is assumed to be linear:

E(Y | Xᵢ) = β₁ + β₂Xᵢ

β₁ (the intercept): the mean value of Y when X = 0.

β₂ (the slope): the change in the mean value of Y for a one-unit increase in X.

However, any individual observation Yᵢ does not lie exactly on this mean line — it deviates by some amount, called
the stochastic disturbance uᵢ:

uᵢ = Yᵢ − E(Y | Xᵢ)

This gives the stochastic form of the PRF:

Yᵢ = β₁ + β₂Xᵢ + uᵢ

Why is uᵢ needed? Because Y depends not only on X but also on countless other factors that cannot all be included
in the model, due to: vagueness of theory, unavailability of data, the distinction between core and peripheral
variables, and limited resources when collecting data. uᵢ represents the combined effect of all these omitted factors.
The meaning of "linear": it is important to distinguish linearity in the variables from linearity in the parameters. In
econometrics, the term "linear regression" always refers to

linearity in the parameters β, not necessarily in the variable X. For example, Yᵢ = β₁ + β₂Xᵢ² + uᵢ is still a linear
regression model, because it is linear in β₁ and β₂, even though it is not linear in X.

3. The Sample Regression Function (SRF)

The core problem is that we never know the true population values β₁ and β₂ — we only have an observable sample
of data. From that sample, we estimate:

Ŷᵢ = β̂₁ + β̂₂Xᵢ

This is called the SRF, an approximation of the PRF computed from a specific sample of data. If a different sample
is drawn from the same population, β̂₁ and β̂₂ will change — this is known as

sampling fluctuation, illustrated by the fact that two different samples drawn from Table 2.1 produce two different
regression lines, SRF₁ and SRF₂.

Two terms must be carefully distinguished:

Estimator: a rule, formula, or method that tells how to estimate the population parameter from the information
provided by a sample.

Estimate: a particular numerical value obtained by applying the estimator to a specific sample.

The stochastic form of the SRF is:

Yᵢ = β̂₁ + β̂₂Xᵢ + ûᵢ

where ûᵢ (the sample residual) is the difference between the actual and estimated Y values: ûᵢ = Yᵢ − Ŷᵢ.
Conceptually, ûᵢ plays a role analogous to uᵢ, but ûᵢ is

computable from the sample, whereas uᵢ is unobservable in the population.

The central question of this chapter: how should the SRF be constructed so that β̂₁ is as close as possible to the
true β₁, and β̂₂ is as close as possible to the true β₂, even though we will never know the true values of β₁ and β₂?

4. The Method of Ordinary Least Squares (OLS)

This is the answer to the question above. Given n pairs of observations (Xᵢ, Yᵢ), we want to determine the SRF so
that it lies as close as possible to the actual Y values. The criterion adopted is:

choose β̂₁ and β̂₂ so that the sum of the squared residuals is as small as possible (rather than the simple sum of
residuals, since positive and negative deviations would cancel out and fail to reflect how well the line actually fits
the data).

Q(β̂₁, β̂₂) = Σûᵢ² = Σ(Yᵢ − β̂₁ − β̂₂Xᵢ)² → min

Taking the partial derivatives of Q with respect to β̂₁ and β̂₂ and setting them to zero yields the normal equations:
 ΣYᵢ = nβ̂₁ + β̂₂ΣXᵢ

ΣXᵢYᵢ = β̂₁ΣXᵢ + β̂₂ΣXᵢ²

Solving these equations simultaneously gives the OLS estimator formulas:

β̂₂ = [nΣXᵢYᵢ − ΣXᵢΣYᵢ] / [nΣXᵢ² − (ΣXᵢ)²] = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / Σ(Xᵢ − X̄)² = Sxy / Sxx

β̂₁ = Ȳ − β̂₂X̄

Where:

Sxy = Σ(Xᵢ − X̄ )(Yᵢ − Ȳ) = ΣXᵢYᵢ − nX̄ Ȳ

Sxx = Σ(Xᵢ − X̄ )² = ΣXᵢ² − nX̄ ²

Syy = Σ(Yᵢ − Ȳ)² = ΣYᵢ² − nȲ²

Intuition: β̂₂ is simply the ratio of the covariance between X and Y to the variance of X — it measures how much
X and Y move together relative to how much X varies on its own: β̂₂ = Cov(X, Y) / Var(X).

Example 1: Working Time and Income

With 10 observations: ΣX = 80, ΣY = 96, ΣX² = 668, ΣXY = 789, X̄ = 8, Ȳ = 9.6

Sxx = 668 − 10×8² = 28

Sxy = 789 − 10×8×9.6 = 21

Syy = 952 − 10×9.6² = 30.4

β̂₂ = 21/28 = 0.75

β̂₁ = 9.6 − 0.75×8 = 3.6

SRF: Ŷ = 3.6 + 0.75X

Interpretation: if working time increases by 1 hour, the estimated increase in income is about 0.75 units (75 cents).
This is an essential skill to practice — always interpret coefficients in the correct units and real-world context, not
just as bare numbers.

Example 2: Rice Price and Rice Demand

With ΣX = 24, ΣY = 36, ΣXY = 111, ΣX² = 120, n = 6, we obtain:

SRF: Ŷᵢ = 11.5 − 1.375Xᵢ

Interpretation: if the price of rice increases by 1 thousand dong per kg, estimated rice demand decreases by about
1.375 tons per month — consistent with the law of demand (price up, quantity demanded down).
5. Three Algebraic Properties of the OLS Line
These properties always hold for any sample — they are direct consequences of the first-order conditions from the
minimization, not additional assumptions:

(1) The sum and the sample average of the OLS residuals is always zero: Σûᵢ = 0.

(2) The sample covariance between the regressor X and the OLS residuals is always zero: ΣXᵢûᵢ = 0.

(3) The point (X̄ , Ȳ) always lies on the OLS regression line.

6. The Assumptions Underlying the Classical Linear Regression

Model (CLRM)
For the estimators β̂₁ and β̂₂ to have desirable statistical properties (unbiasedness, efficiency), OLS must rest on the
following assumptions:

Assumption 1 — Linearity in parameters: the model has the form Yᵢ = β₁ + β₂Xᵢ + uᵢ, linear in the parameters β.

Assumption 2 — X values are fixed in repeated sampling: X is treated as non-stochastic, meaning the regression
analysis is conditional on the given values of the regressor(s).

Assumption 3 — Zero mean value of the disturbance: E(uᵢ | Xᵢ) = 0. For any given Xᵢ, the positive and negative
deviations of Y around the mean line cancel out. This assumption implies Cov(uᵢ, Xᵢ) = 0 — the disturbance and the
explanatory variable are uncorrelated. If X and u were correlated, it would be impossible to separate the individual
effects of X and of the omitted factors on Y.

Assumption 4 — Homoscedasticity, or equal variance of the disturbance: Var(uᵢ | Xᵢ) = σ² for all i, meaning the
dispersion of Y around the regression line is the same at every level of X. When this is violated, the model exhibits
heteroscedasticity — a topic covered in a later chapter.

Assumption 5 — No autocorrelation between disturbances: Cov(uᵢ, uⱼ) = 0 for i ≠ j. The disturbance of one
observation is unrelated to that of another — particularly important in time-series data, where u ₜ may be correlated
with uₜ₋₁.

Assumption 6: the number of observations n must exceed the number of parameters to be estimated.

Assumption 7 — Variability in X values: the X values in the sample must not all be identical, otherwise Sxx
cannot be computed (the denominator would be zero).

Assumption 8 — The regression model is correctly specified: there is no specification bias — no important
variable is omitted, the functional form is correct (linear or nonlinear), and the probabilistic assumptions about Y, X,
and u are reasonable. A classic illustration: if the true relationship between the unemployment rate and the rate of
change of money wages is nonlinear (the Phillips curve, Yᵢ = β₁ + β₂(1/Xᵢ) + uᵢ), but a linear model Yᵢ = α₁ + α₂Xᵢ
+ uᵢ is fitted instead, the resulting predictions will be seriously wrong.

Assumption 9 — No perfect multicollinearity: no perfect linear relationship exists among the explanatory
variables — relevant to multiple regression, covered in a later chapter.
If Assumptions 1–5 hold, then by the Gauss-Markov Theorem, the OLS estimators β̂₁ and β̂₂ are BLUE (Best
Linear Unbiased Estimators) — that is, among all linear unbiased estimators, OLS has the minimum variance, i.e.,
it is the most efficient.

If we further assume that uᵢ is normally distributed, the OLS estimators have these additional properties:

β̂₁ ~ N(β₁, σ²β̂₁) β̂₂ ~ N(β₂, σ²β̂₂)

Var(β̂₂) = σ² / Σxᵢ² Var(β̂₁) = [ΣXᵢ² / (nΣxᵢ²)] σ²

σ² (the population disturbance variance) is estimated by:

σ̂² = Σûᵢ² / (n − 2)

where (n − 2) is the number of degrees of freedom — two degrees are lost because β̂₁ and β̂₂ must be estimated
before the residuals ûᵢ can be computed. The quantity Σûᵢ² is called the Residual Sum of Squares (RSS).

7. Measuring Goodness of Fit — The Coefficient of Determination

r² (R²)
Once the SRF has been obtained, the next question is: how well does this regression line fit the data? We decompose
the total variation in Y into two parts:

TSS = ESS + RSS

TSS (Total Sum of Squares): Σ(Yᵢ − Ȳ)², the total variation of the actual Y values about their sample mean.

ESS (Explained Sum of Squares): Σ(Ŷᵢ − Ȳ)² = β̂₂²Σxᵢ², the variation in Y explained by the regression model.

RSS (Residual Sum of Squares): Σûᵢ², the remaining, unexplained variation, attributable to the disturbance.

The coefficient of determination is then defined as:

r² = ESS/TSS = 1 − RSS/TSS

Interpretation: r² measures the proportion (or percentage) of the total variation in Y that is explained by X through
the regression model. r² is always bounded between 0 and 1:

r² = 1: a perfect fit, every Ŷᵢ equals the actual Yᵢ.

r² = 0: X explains none of the variation in Y (equivalently, β̂₂ = 0).

Real-world Example: CEO Salary and ROE

Regression result: salary = 963.19 + 18.50×roe, with R² = 0.0132.

Interpretation: ROE (return on equity) explains only about 1.32% of the variation in CEO salary — a very low
proportion, indicating that ROE is far from the main determinant of CEO compensation, and that many other factors,
not yet included in the model, account for the remaining variation.
Summary: The Logical Flow of the Chapter
(1) We want to know how Y depends on X at the population level → we construct the Population Regression
Function (PRF), but the entire population is generally unobservable in practice.

(2) We use sample data to estimate this relationship → we construct the Sample Regression Function (SRF).

(3) To choose the "best" set of estimated coefficients β̂ from the sample, we apply the principle of Ordinary Least
Squares (OLS) — minimizing the sum of squared residuals.

(4) For the OLS estimators to be statistically reliable (unbiased and efficient — BLUE), the assumptions of the
Classical Linear Regression Model (CLRM) must be satisfied.

(5) Finally, we assess how well the model fits the data using the coefficient of determination, R².