Econometric Methods (AE-EE-08)

Class Notes: Module 1

MA 2nd Semester (Jan-May 2026)
Course Coordinator: Gopal Krishna Roy
Email: [Link]@[Link]
What is Econometrics?
• Econometrics applies mathematical and statistical concepts to develop
theories and methods for estimating economic relationships and test
economic theories/models using empirical data (Wooldridge, 2018, Ch 1).
• Vast spectrum of applications:
▪ Hypotheses testing: understanding the relationship (association) among
economic variables;
▪ Impact evaluation: assessing the causal impact of a new product, process, or
policy on economic indicators of interest.
▪ Prediction: forecasting economic variables such as GDP, inflation, or sales, etc.
• The application of econometrics goes beyond economics and finds a wide
range of applications across all social sciences and management
disciplines.
• For that matter, the linear regression topic we will cover in this
econometrics course has much in common with supervised machine
learning and data science.
About the course
Topics Covered:
1. Simple Regression analysis
2. Multiple Regression analysis and inference
3. Models with Dummy Variables as regressors
4. Extensions of Linear Models: Regression diagnostics
5. Endogeneity (Multi-equation models): Omitted variable
and Simultaneity Bias, instrument variables and Two-SLS
Prerequisites: Linear algebra, Calculus, Probability, and Statistical inference
Evaluation:
Midterm 1: 20 % Midterm 2 (term paper): 20 % End term: 60
❖ The selection of the topic and dataset for the term paper is flexible.
You may choose to do a group or individual project, followed by
individual viva-voce. Preparing Stata codes .do file is mandatory.
From Economic Model to Econometric Model
• An economic model is conceptual in nature with a purpose to explain and
hypothesize economic relationships.
▪ E.g. Consumer theory suggests a negative relationship between the quantity
demanded of a normal good and its price (linear demand curve equation) based
on utility maximization theory and a positive relationship between income and
demand
▪ The Keynesian consumption function assumes a positive relationship between
consumption and income.
• Formal economic modeling (like demand curve equation) is occasionally the
initial stage in empirical analysis (the latter being the modeling using data)
• But more frequently, economic theory is applied in a less structured manner,
and, in some cases, entirely reliant on intuition based on economic thinking
and common sense (without specifying a precise equation, assumptions,
derivations like a formal economic model) to hypothesize a relationship. E.g.,
job training should improve workers’ productivity.
Economic Model Vs Econometric Model continued..
• Econometric models estimate, test, and validate the economic model or
hypothesized relationships among economic variables (hypothesis testing)
by applying quantitative methods to the collected data for those variables.
• So, the economic model derives and predicts a negative relationship
between a normal good’s price and quantity demanded using the consumer
choice theory of budget constraints, indifference curves, and utility
maximization in Economics.
• Econometric model estimates/predicts the relationship between price and
quantity by “fitting” a straight line using the observed data on price and
quantity for a normal good. If the slope of the estimated straight line is
negative and significantly (statistically) different from zero, the negative
price-quantity relationship predicted by the economic model is validated.
• In summary, economic and econometric models are intertwined in the
empirical economic analysis process, each playing a crucial role in shaping
our understanding of real-world economic phenomena.
 Navigating Economic Data Structures: A Prerequisite for Effective
Econometric Modeling

Data
Pooled Cross-sections
Panel
Cross- i=n
Time-series i=n
sectional t>1
i=1 t>1
i=n ‘i’ sampled from same
t>1 population over ‘t’ Observing ‘i’ over time ‘t’
t=1 without knowing the Identity of ‘i’ is known
identity of ‘i’ (tracking (tracking is possible)
not possible)

• i refers to individual entities, e.g., individuals, households, firms, countries, etc.

• t refers to time, e.g., year.
• The scope of this course is limited to the analysis of cross-sectional data only.
Types of data (Based on Scale of Measurement)
There are four types of variables/data based on the scale of
measurement:
Categorical variables:
1. Nominal – purely categories without any specified
order/rank.
• Ex: Gender, Marital status, Type of location –
Rural/Urban; Yes/No types: Spam/Not Spam, invest/not
invest.
• Often coded as numbers, e.g., 1-Yes, No-0; 1-Male, 2-
Female, 3- Third Gender.
• However, these numbers do not have numeric properties;
they are only used for representing categories.
2. Ordinal – multiple categories with meaningful order/rank.
• Ex: Likert Scale, e.g., Service ratings – 1. poor, 2. Good, 3.
Excellent
Numerical Variables
3. Interval: numerical values where addition and subtraction are
meaningful, i.e., the intervals between numbers are meaningfully
interpreted, but ratios are not. There is no absolute zero in the
interval scale.
• Interval scale examples: Temperature in degrees Fahrenheit and
Celsius.
• The difference between 10 and 20 degrees Celsius is 10 degrees
Celsius.
• But 20 degrees Celsius is not twice as hot as 10 degrees Celsius.
• Zero degrees Celsius does not mean there is no temperature.
Scales of measurement continued..
4. Ratio: numerical values with all arithmetic operations,
including ratios being meaningful. There is an absolute
zero.
• Ex: Income/price – INR 100 is twice INR 50; zero
income means there is no income at all, or zero price
means the product is free.
• Output in an economy or GDP per capita.
• In this course, the scope of the variables used in the
econometric modeling are as follows:
▪ Dependent variable: numerical dependent variable that
are continuous in nature (measured in the ratio scale)
▪ Independent variables: both numerical (ratio scale) and
categorical (nominal scale) in nature.
Goals of the course
Upon completion of this course, you will possess the ability to:
• Explain and implement simple and multiple regression models in the
context of cross-sectional data
• Understand the underlying assumptions of the linear regression
framework (Gauss-Markov conditions and other necessary
assumptions) and perform statistical inference
• Work with dummy variables as explanatory variables in a linear
regression model
• Identify reasons/scenarios and explain the consequences of the
violation of these assumptions
• Conduct tests to detect violations of these assumptions
• Propose potential remedial measures, including, when suitable, the
utilization of instrumental variables
Simple Linear Regression: Introduction
• Regression forms the basis of econometric analysis
(estimating relationships between economic variables,
e.g., price and quantity demanded).
• Regression is a statistical tool that is used across
science and social science disciplines (not just
econometrics) to estimate relationships among
variables .
Historical Origin of the term Regression
• Francis Galton coined the term "regression"
• In his 1886 study of heights of individuals in a
population, Galton observed that although there is a
tendency for children to inherit height characteristics
from their parents (tall parents often have tall children,
and short parents often have short children), the
average height of children born to parents of a certain
height tends to move or "regress" toward the average
height of the entire population (Gujarati & Porter, 2009,
Ch-1).
• Galton called this phenomenon as “regression to
mediocrity”, i.e., extreme characteristics (either tall or
short) in parents are likely to be moderated or balanced
out in their children (Gujarati & Porter, 2009, Ch-1). In
statistics, this is also called as regression to mean.
• Later, Karl Pearson in his 1903 study of heights in a
population confirmed the findings of Galton’s law of
universal regression by finding that the average height
of sons of a group of tall fathers was less than their
fathers’ height and the average height of sons of a
group of short fathers was greater than their fathers’
height.
• This indicates a "regression" phenomenon where both
tall and short sons tend to move towards the average
height of all men (Gujarati & Porter, 2009, Ch-1).
Modern Interpretation of Regression
• Estimate or predict relationships between two or more
variables. In linear regression, we fit and quantify a linear
(in parameters) relationship between two or more variables.
• The idea is to estimate and/or predict the (population)
mean or average value of one variable in terms of the
known or fixed (in repeated sampling) values of one or
more variables (Gujarati & Porter, 2009, Ch-1).
• E.g. estimating a linear relationship between sons' heights
and their fathers’ heights (the slope of the linear
relationship is positive or negative) and predicting the
average height of sons knowing the height of their fathers.
Regression Vs Correlation
• Note: Correlation coefficients give the strength/degree of
linear association between two variables (say Y and X are
highly positively associated), but they do not quantify the
relationship, i.e., we can’t predict the expected/average
value of a variable (Y) given the values of another variable
(X) or how much on an average a unit change in X is
associated with a unit change in Y. Moreover, we cannot
control for other factors (ceteris paribus condition) through
a simple correlation analysis.
• Linear regression analysis helps us overcome the above
challenges to quantify and predict linear relationships
between variables.
The Simple linear Regression Model
• The simple linear regression model can be used to study the
linear relationship between two variables (Wooldridge,
2018, Ch-2).
Definition of the simple regression model
• Premise:
▪ y and x are two variables, representing some population,
and we are interested in “explaining y in terms of x,” or
in “studying how y varies with changes in x.”
▪ E.g., y is crop yield and x is amount of fertilizer; y is
hourly wage and x is years of education; and y is
quantity demanded of a normal good and x is its price.
• We can define a linear relationship between y and x in
terms of the general equation of line: y=mx +c (i.e., y
is a linear function of x), where m is the slope and c is
the y-axis intercept.
• The general equation of line is deterministic/exact in
nature, i.e., all the combinations of y and x fall on one
straight line on the x-y plane.
• However, in practice, there is never an exact
relationship between two variables and hence, we add a
stochastic or random disturbance term to the
deterministic term to allow departure of the actual y
values from those corresponding to the straight line.
• A simple linear regression model equation in the population of
interest is defined as follows:
𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝑢 ……(1)
• Eqn (1) is also known as the the two-variable linear regression
model or bivariate linear regression model because it relates
the two variables x and y (Wooldridge, 2018, Ch-2).
• y and x have several different names used interchangeably
• y is referred as dependent variable/explained
variable/response variable/predicted variable/regressand.
• x is referred as independent variable/explanatory
variable/control variable/predictor
variable/regressor/covariates.
• The variable u in eqn (1), referred to as the error term or
disturbance in the relationship, accounts for factors
beyond x that affect y.
• In a simple regression analysis, all factors affecting y,
excluding x, are essentially considered unobserved. It is
helpful to conceptualize u as representing the
"unobserved" factors (Wooldridge, 2018, Ch-2).
• Other than the unobserved factors, there could be
several other reasons for the error terms to exist, e.g.,
measurement error, model specification, functional
misspecification, etc.
Functional relationship between y and x from eqn (1):
▪ 𝛽0 is the intercept parameter also known as the constant
term in eqn (1).
▪ Since 𝛽0 is constant ∆ 𝛽0 =0.
▪ If the other factors in u are held fixed, so that the change
in u is zero, ∆u=0, then x has a linear effect on y:
▪ ∆𝑦 = 𝛽1 ∆𝑥 if ∆u=0,i.e., ceteris paribus …..(2)
▪ Thus, the change in y is simply 𝛽1 multiplied by the change
in x. This means that 𝛽1 is the slope parameter in the
relationship between y and x, holding the other factors in u
fixed (Wooldridge, 2018, Ch-2).
▪ 𝛽1 is the parameter estimate that is of primary interest in
econometric analysis.
Examples of a simple linear regression model (Wooldridge, 2018, Ch-2).

• Relationship between soyabean yield and fertilizer:

𝑦𝑖𝑒𝑙𝑑 = 𝛽0 + 𝛽1 𝑓𝑒𝑟𝑡𝑖𝑙𝑖𝑧𝑒𝑟 + 𝑢
▪ Comparing the above eqn with eqn (1): y= soyabean
yield, x= fertilizer, u contains unobserved (measurables
but data not available and unmeasurables) factors that
may affect soyabean yield such as land quality, rainfall,
etc.
▪ The objective of the econometrician is to examine the
effect of fertilizer on yield, holding other factors fixed.
This effect is given by 𝛽1 .
▪ The coefficient 𝛽1 measures the effect of fertilizer on yield,
holding other factors fixed: ∆𝑦𝑖𝑒𝑙𝑑 = 𝛽1 ∆𝑓𝑒𝑟𝑡𝑖𝑙𝑖𝑧𝑒𝑟 if ∆u=0.
• Relationship between a person’s wage and education
(Simple Wage Equation): wage= 𝛽0 + 𝛽1 𝑒𝑑𝑢𝑐𝑎𝑡𝑖𝑜𝑛 + 𝑢
▪ If wage is expressed as dollars per hour and
education represents years of education, then 𝛽1
measures the change in hourly wage for each
additional year of education, while keeping all other
unobserved variables (u) constant.
▪ Here, u could be labor force experience, inherent
ability, tenure with the current employer, work ethic,
etc.
Assumptions about the error term
• Assumption about the distribution of the unobserved
factors in the population: E(u)=0 …..(3)
▪ Intuition: In the previous example of wage equation,
if we assume ability to be an unobserved factor
affecting wage, without loss of generality, we can
assume that things such as average ability are zero
in the population of all working people (Wooldridge,
2018, Ch-2).
▪ Eqn (3) is not restrictive, we can always redefine the
intercept in equation (1) to make equation (3) true.
Prove: In the simple linear regression model 𝑦 = 𝛽0 +
𝛽1 𝑥 + 𝑢 , suppose that E(u)≠ 0. Letting 𝛼0 = 𝐸(𝑢), show
that the model can always be rewritten with the same
slope, but a new intercept and error, where the new error
has a zero expected value.
Proof: Add and subtract 𝛼0 = 𝐸(𝑢) to the RHS of eqn (1):
y= (𝛼0 +𝛽0 ) + 𝛽1 𝑥 + (𝑢- 𝛼0 )
Let the new error be e = (𝑢- 𝛼0 ) . Then E(e)= E(u)-
E(𝛼0 )=0 (since E(𝛼0 )=E(E(u))=E(u)). Hence, the new
intercept is (𝛼0 +𝛽0 ) with the same slope 𝛽1 but the
expected value of the error term is zero.
Assumptions about the error term continued..
• Expected value of u given x:
▪ the average value of u does not depend on the value
of x, i.e., u is mean independent of x (Wooldridge,
2018, Ch-2).
▪ E(u|x)=E(u) ……..(4)
▪ Eqn (4) says that the average value of the
unobservables is the same across all slices of the
population determined by the value of x.
▪ Moreover, the common average is necessarily equal
to the average of u over the entire population.
▪ When we combine mean independence (eqn 4) with
eqn (3), we obtain the zero conditional mean
assumption, E(u|x)= 0.
Population regression Function:
• The zero conditional mean assumption gives 𝛽1 , in eqn (1):
𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝑢, another interpretation:
• Taking the expected value of eqn (1) conditional on x and using
E(u|x)= 0 gives: E(y|x)= 𝛽0 + 𝛽1 𝑥 ……………(5)
• Eqn (5) shows that the population regression function (PRF),
E(y|x) , is a linear function of x (Wooldridge, 2018, Ch-2).
• The PRF gives us a relationship between the average level of y
at different levels of x.
• The linearity means that a one-unit increase in x changes the
expected value of y by the amount 𝛽1 (Wooldridge, 2018, Ch-2).
• For any given value of x, the distribution of y is centered about E(y|x). See the fig
below:

• The regression line does not

describe individual outcomes.
It describes the average of 𝑦
for each value of 𝑥.
• For any fixed 𝑥, the
distribution of 𝑦 is centered at
𝐸(𝑦∣𝑥), and the regression line
Source: (Wooldridge, 2018, Ch-2). simply connects those centers
across different values of 𝑥.
• Given the zero conditional mean assumption E(u|x)=0, y can be seen
of having two components. The first component 𝛽0 + 𝛽1 𝑥, which
represents E(y|x) , is called the systematic part of y (the part of y
explained by x)
• The second component u is called the unsystematic part, or the part of
y not explained by x (Wooldridge, 2018, Ch-2).
• 𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝑢
y=E(y|x) + u (using eqn (5))

Unsystematic part
Systematic part

• In practice, we never know the population intercept 𝛽0 and slope 𝛽1 .

• We statistically estimate 𝛽0 and 𝛽1 using a random sample of data. The
estimator is known as ordinary least squares (OLS) estimator.
Deriving the OLS estimates
• Let { (𝑥𝑖 , 𝑦𝑖 ): i=1, . . . , n} denote a random sample of size n from the population.
• Objective: use sample data to estimate the population parameters 𝛽0 and 𝛽1 in
equation (1).
• Suppose we have data on savings and incomes for 15 families (n=15) for a
particular year, we have 𝑦𝑖 = 𝛽0 + 𝛽1 𝑥𝑖 + 𝑢𝑖 ….(6)

Source: (Wooldridge, 2018, Ch-2).

• There are several ways to motivate the estimation of population parameters.
We will follow (Wooldridge, 2018, Ch-2) approach.
• Recall assumptions about the error term u in eqn (2) and (3):
E(u)=0 …..(3)
E(u|x)=E(u) …..(4)
Eqn (4) implies that, in the population, u is uncorrelated with x, i.e., Cov(x,u)=
E(xu)=0
Cov(x,u)= E[xu -u𝜇𝑥 − 𝑥𝜇𝑢 + 𝜇𝑥 𝜇𝑢 ]
 Cov(x,u)= E(xu) -𝜇𝑥 𝐸(𝑢) − 𝜇𝑢 𝐸(𝑥) + 𝜇𝑥 𝜇𝑢
 Cov(x,u)= E(xu) -𝜇𝑥 𝜇𝑢 − 𝜇𝑥 𝜇𝑢 + 𝜇𝑥 𝜇𝑢
 Cov(x,u)= E(xu) (as 𝜇𝑢 =E(u)=0 from eqn(3))
 Cov(x,u)= E(xu)=E[E(xu∣x)] = E[xE(u∣x)] (using iterated expectations:
E(y)=E[E(y|x)], conditioning on x makes x a constant, and E(u∣x)=0, so
E(xu)=E(x⋅0)=0))
Therefore, Cov(x,u)= E(xu)=0 …(7)
• In terms of the observable variables x and y and the
unknown parameters 𝛽0 and 𝛽1 , equations (3) and (7)
respectively can be written as:
• E(y- 𝛽0 − 𝛽1 𝑥)=0 ………….(8)
• E[x(y- 𝛽0 − 𝛽1 𝑥)]=0 ……………(9)
• Given a sample of data, we choose estimates 𝛽 ෢0 and 𝛽
෢1 to
solve the sample counterparts of equations (8) and (9):
• 𝑛−1 σ𝑛𝑖=1(𝑦𝑖 −𝛽෢0 − 𝛽෢1 𝑥𝑖 ) =0 ……..(10)
෢0 − 𝛽
• 𝑛−1 σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝛽 ෢1 𝑥𝑖 ) =0 ……….(11)
• This estimation is based on method of moments approach
to estimation.
• Equations (10) and (11) can be solved for 𝛽
෢0 and 𝛽
෢1 .
• From eqn(10), we have:
• 𝑦ത = 𝛽෢0 + 𝛽෢1 𝑥ҧ ……….(12), Here 𝑦ത and 𝑥ҧ are sample averages of
y and x, respectively.
𝛽 ෢0 = 𝑦ത - 𝛽
෢1 𝑥ҧ ………..(13)
Therefore, once we have the slope estimate 𝛽 ෢1 , it is
straightforward to obtain the intercept estimate 𝛽 ෢0 , given 𝑦ത and 𝑥ҧ
Using (13) in (11), we get:
෢1 𝑥)ҧ − 𝛽
σ𝑛𝑖=1 𝑥𝑖 [𝑦𝑖 − (𝑦ത − 𝛽 ෢1 𝑥𝑖 ] =0
 σ𝑛𝑖=1 𝑥𝑖 [(𝑦𝑖 − 𝑦)ത − 𝛽 ෢1 (𝑥𝑖 − 𝑥)] ҧ =0
 σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦)ത = 𝛽 ෢1 σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ …………………(14)
From basic properties of the summation operator, it
can be proved (next two slides) that:
σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦) σ 𝑛
ത = 𝑖=1 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത …..(15),
and
𝑛 𝑛
σ𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ = σ𝑖=1(𝑥𝑖 − 𝑥)ҧ 2 ………………(16)
Proof: σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ = σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2
LHS= σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ
=σ𝑛𝑖=1 𝑥𝑖2 − 𝑥ҧ σ𝑛𝑖=1 𝑥𝑖
= σ𝑛𝑖=1 𝑥𝑖2 − 𝑥(𝑛
ҧ 𝑥) ҧ (using the formula of mean 𝑥:
ҧ  𝑥𝑖 = n𝑥ҧ )
= σ𝑛𝑖=1 𝑥𝑖2 − 𝑛𝑥ҧ 2 ………..(17)
RHS= σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2
= σ𝑛𝑖=1( 𝑥𝑖2 − 2𝑥𝑖 𝑥ҧ + 𝑥ҧ 2 )
= σ𝑛𝑖=1 𝑥𝑖2 − 2𝑥ҧ σ𝑛𝑖=1 𝑥𝑖 + 𝑛𝑥ҧ 2
= σ𝑛𝑖=1 𝑥𝑖2 − 2𝑥ҧ 𝑛𝑥ҧ + 𝑛𝑥ҧ 2
= σ𝑛𝑖=1 𝑥𝑖2 − 2𝑛𝑥ҧ 2 + 𝑛𝑥ҧ 2
=σ𝑛𝑖=1 𝑥𝑖2 − 𝑛𝑥ҧ 2 ………..(18)
LHS=RHS (from (17) and (18))
Proof: σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦) ത = σ𝑛𝑖=1(𝑥𝑖 − 𝑥)(𝑦
ҧ 𝑖 − 𝑦)
ത
LHS= σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦) ത
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑦ത σ𝑛𝑖=1 𝑥𝑖 (using the formula of mean 𝑥:
ҧ  𝑥𝑖 = n𝑥ҧ )
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑦(𝑛
ത 𝑥) ҧ
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑛( 𝑥ҧ 𝑦)
ത …………(19)
RHS= σ𝑛𝑖=1(𝑥𝑖 − 𝑥)(𝑦 ҧ 𝑖 − 𝑦) ത
= σ𝑛𝑖=1(𝑥𝑖 𝑦𝑖 − 𝑥𝑦 ҧ 𝑖 − 𝑦𝑥 ത 𝑖 + 𝑥ҧ 𝑦)
ത
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑥ҧ σ𝑛𝑖=1 𝑦𝑖 − 𝑦ത σ𝑛𝑖=1 𝑥𝑖 + 𝑛𝑥ҧ 𝑦ത
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑥(𝑛
ҧ 𝑦) ത − 𝑦ത 𝑛𝑥ҧ + n𝑥ҧ 𝑦ത
= σ𝑛𝑖=1 𝑥𝑖 𝑦𝑖 − 𝑛𝑥ҧ 𝑦ത ………………(20)
LHS=RHS (from (19) and (20))
• Using (14), (15), and (16), where
ത = 𝛽
σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦) ෢1 σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ …………………(14),
ത = σ𝑛𝑖=1 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത
σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦) …..(15),
σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ = σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2 ………………(16)
We have:
σ𝑛𝑖=1 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത = 𝛽 ෢1 σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2
σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦 ത
𝛽
෢1 =
σ𝑛 ҧ 2 (provided 𝑖=1(𝑥𝑖 − 𝑥)ҧ 2 >0) ..(21)
σ𝑛
𝑖=1(𝑥𝑖 − 𝑥)
Hence, the estimated slope is simply the sample
covariance between 𝑥𝑖 and 𝑦𝑖 divided by the sample
variance of 𝑥𝑖 .
෢1 can also be expressed in terms of correlation (first multiply
𝛽
numerator and denominator by 1/(n-1) and then multiply
numerator and denominator by standard deviation of 𝑦𝑖 ) as
follows:
ෝ𝑦
𝜎
෢1 =𝜌ො𝑥𝑦 ( ) …..(22),
𝛽 ෝ
𝜎 𝑥
Where, 𝜌ො𝑥𝑦 is the sample correlation between 𝑥𝑖 and 𝑦𝑖 and 𝜎ො𝑥 , 𝜎ො𝑦
denote the sample standard deviations.
• Implication of eqns (22) or (21) is that if 𝑥𝑖 and 𝑦𝑖 are positively
correlated in the sample then 𝛽 ෢1 (slope) is positive; if 𝑥𝑖 and
𝑦𝑖 are negatively correlated then 𝛽෢1 < 0.
• Caution: It can be noted that the slope 𝛽 ෢1 (or 𝛽1 𝑖𝑛 𝑡ℎ𝑒 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛)
is a scaled version of correlation between x and y.
• This highlights an important limitation of simple
regression that it is an analysis of correlation between
two variables, and so one must be careful in inferring
causality (when we do not have experimental data)
(Wooldridge, 2018, Ch-2).
• The estimates given in eqn (13) for slope 𝛽 ෢𝑜 and eqn
(21) for intercept 𝛽෢1 are called the ordinary least
squares (OLS) estimates of 𝛽0 and 𝛽1 .
෢0 = 𝑦ത - 𝛽
𝛽 ෢1 𝑥ҧ ………..(13)
σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦ത
෢1 =
𝛽 …..(21)
σ𝑛 ҧ 2
𝑖=1(𝑥𝑖 −𝑥)

• Next, we consider the derivation of OLS estimates 𝛽

෢𝑜
and 𝛽
෢1 using the least-square estimation technique.
Least Squares Regression with One Explanatory Variable
• For any 𝛽 ෢𝑜 and 𝛽 ෢1 in a sample define a fitted value for y (𝑦)
ො for a
given x=𝑥𝑖 .
෢0 + 𝛽
𝑦ෝ𝑖 = 𝛽 ෢1 𝑥𝑖 ……………(23)
• 𝑦ෝ𝑖 is the value we predict for y when x=𝑥𝑖 for the given intercept and
slope.
• There is a fitted value for each observation in the sample.
• The residual for observation i is the difference between the actual/
observed 𝑦𝑖 and its fitted value 𝑦ෝ𝑖 (based on the estimated
coefficients):
𝑢ෝ𝑖 = 𝑦𝑖 − 𝑦ෝ𝑖
𝑢ෝ𝑖 = 𝑦𝑖 − 𝛽 ෢0 − 𝛽෢1 𝑥𝑖 ……………(24)
• The residuals above are not the same as the error terms.
Fitted values and residuals
Minimizing the Sum of Squared Residuals
• From eqn (24): 𝑢ෝ𝑖 = 𝑦𝑖 − 𝛽 ෢0 − 𝛽෢1 𝑥𝑖 , we get sum of squared
residuals for a sample of size n as:
σ𝑛𝑖=1 𝑢ො 𝑖2 = σ𝑛𝑖=1(𝑦𝑖 −𝛽෢0 − 𝛽
෢1 𝑥𝑖 )2 ……(25)
• Objective: we choose 𝛽 ෢0 and 𝛽 ෢1 to minimize the sum of
squared residuals in (25).
• The name “ordinary least squares” comes from the fact that
the estimates 𝛽 ෢0 and 𝛽 ෢1 are obtained through minimizing the
sum of squared residuals (Wooldridge, 2018, Ch-2).
• Minimization Problem: min σ𝑛𝑖=1(𝑦𝑖 −𝑏0 − 𝑏1 𝑥𝑖 )2 ….(26),
𝑏0,𝑏1
where 𝑏0 and 𝑏1 are the dummy arguments for the optimization
problem, i.e, they represent any possible candidate values of
the intercept and slope
The first order partial derivatives of eqn (26) w.r.t. 𝑏0 and
𝑏1 are given by:
𝜕(σ𝑛 (𝑦 −𝑏 −𝑏 𝑥 )2)
𝑖=1 𝑖 0 1 𝑖
= −2 σ𝑛𝑖=1( 𝑦𝑖 − 𝑏0 − 𝑏1 𝑥𝑖 ) …..(27)
𝜕𝑏0
𝜕(σ𝑛 (𝑦 −𝑏 −𝑏 𝑥 ) 2)
𝑖=1 𝑖 0 1 𝑖
= −2 σ𝑛𝑖=1 𝑥𝑖 ( 𝑦𝑖 − 𝑏0 − 𝑏1 𝑥𝑖 ) .… (28)
𝜕𝑏1

let 𝛽෢0 and 𝛽 ෢1 are the values of 𝑏0 and 𝑏1 that satisfies

the first-order conditions of the minimisation
problem, i.e., once the minimization is solved, the
values of 𝑏0 and 𝑏1 that minimize the function are
𝛽መ0 and 𝛽መ1 .
• A necessary condition for β ෡ and β෡ to solve the minimization
0 1
problem is that the partial derivatives of (25) with respect to 𝑏0
[i.e, eqn(27)] and 𝑏1 [i.e., eqn(28)] must be zero when evaluated
෡ and β
at β ෡ :
0 1

-2 σ𝑛 ( y - β෡ -β෡ x ) =0 ……(29),
𝑖=1 i 0 1 i

෡ -β
-2 σ𝑛𝑖=1 xi( yi - β ෡ x ) =0 ………(30)
0 1 i

• Eqn (29) and (30) are referred as first order conditions.

෡ from (29). Multiplying both sides of
• Now, let us solve for β 0
eqn(29) by –1/2n, we get:
෢1 𝑥𝑖 ) =0 …..(31)
෢0 − 𝛽
𝑛−1 σ𝑛𝑖=1(𝑦𝑖 −𝛽
𝑦ത = 𝛽
෢0 + 𝛽
෢1 𝑥ҧ …..(32)
(same estimate that we got from the method of moments earlier)
Using (32) in (30), we get:
σ𝑛𝑖=1 𝑥𝑖 [𝑦𝑖 − (𝑦ത − 𝛽 ෢1 𝑥𝑖 ] =0
෢1 𝑥)ҧ − 𝛽
 σ𝑛𝑖=1 𝑥𝑖 [(𝑦𝑖 − 𝑦)
ത − 𝛽 ෢1 (𝑥𝑖 − 𝑥)] ҧ =0
 σ𝑛𝑖=1 𝑥𝑖 (𝑦𝑖 − 𝑦)
ത = 𝛽 ෢1 σ𝑛𝑖=1 𝑥𝑖 (𝑥𝑖 − 𝑥)ҧ
σ𝑛𝑖=1 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത = 𝛽෢1 σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2 (using (15) and (16))
σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦ത
𝛽
෢1 =
σ𝑛 ҧ 2
(provided σ𝑛𝑖=1(𝑥𝑖 − 𝑥)ҧ 2 >0) ……(33)
𝑖=1(𝑥𝑖 −𝑥)

(same estimate that we got from the method of moments earlier)

• The first order conditions are necessary but not sufficient
conditions.
• Next step is to verify that we have minimized the sum of squared
residuals.
• If we add and subtract β෠ 0 and β෠ 1xi to eqn (26), we get:
σ𝑛𝑖=1( yi – b0- b1xi )2 = σ𝑛𝑖=1 [( yi - β෠ 0 - β෠ 1xi +(β෠ 0- b0 ) +(β෠ 1 -b1)xi )]2
= σ𝑛𝑖=1 [ (𝑢ො i +(β෠ 0- b0 ) +(β෠ 1 -b1)xi )]2
= σ𝑛
𝑖=1 𝑢
ෝ 𝑖
2 + n(β ෠ 0 - b 0 ) ෠
2 + (β -b
1 1 ) 2 σ𝑛 x
𝑖=1 i
෠
2 + 2(β
0 - b 0 )( ෠
β 1 -b 1 ) σ𝑛
𝑖=1 xi
(we use the formula for (𝑎 + 𝑏 + 𝑐)2 and the OLS first order conditions
from (29) and (30) ensure elimination of the remaining terms)
• The first term does not depend on b0 and b1, and the sum of the last three
terms can be written as σ𝑛𝑖=1 [(β෠ 0- b0 ) + (β෠ 1 -b1) xi ] 2
• Because this is a sum of squared terms, the minimum it can be is zero.
• Therefore, it is minimized when 𝑏0 = β෠ 0 and 𝑏1 = β෠ 1. Sum of Squared
Residuals (SSR) using any b values is always larger than (or equal to) the
SSR using our OLS estimates since something squared is always >=0.
Sample regression function
• Once we have determined the OLS intercept and slope estimates, we
form the OLS regression line:
෢0 + 𝛽
𝑦ො = 𝛽 ෢1 𝑥 ……….(31)
Equation (31) is also called the sample regression function (SRF)
because it is the estimated version of the population regression function
eqn (5): E(y|x)= 𝛽0 + 𝛽1 𝑥
• each fitted value of 𝑦ො𝑖 is on the OLS regression line (31).
• It is important to remember that the PRF is something fixed, but
unknown, in the population (Wooldridge, 2018, Ch-2).
• Because the SRF is obtained for a given sample of data, a new sample
will generate a different slope and intercept in equation (31).
Δ𝑦ො
• the slope estimate in (31): 𝛽1 = , the amount by which 𝑦ො changes
෢
Δ𝑥
when x increases by one unit.
OLS regression equation interpretation:
Example- CEO Salary and Return on Equity (Wooldridge, 2018, Ch-2).
• Suppose we are interested in studying the relationship between return on equity (a
measure of firm performance) and CEO compensation.
• For the population of chief executive officers, let y be annual salary (salary) in
thousands of dollars.
• Let x be the average return on equity (roe) measured in percentage for the CEO’s
firm for the previous three years.
• we postulate the simple model:
𝑠𝑎𝑙𝑎𝑟𝑦 = 𝛽0 + 𝛽1 𝑟𝑜𝑒 + 𝑢
The slope parameter 𝛽1 measures the change in annual salary, in thousands of
dollars, when return on equity increases by one percentage point. Because a higher
roe is good for the company, we think 𝛽1 > 0.
• Next we use a sample data for US firms during 1990 to estimate the OLS
regression line relating salary to roe. Suppose the estimated OLS regression is:
෣
𝑠𝑎𝑙𝑎𝑟𝑦= 963.191 + 18.501 roe, n=209
How do we interpret the above equation? 𝛽
෢0 , 𝛽
෢1 , what is the predicted salary at roe
=30?
• The estimated OLS regression line is graphed in figure below, along with the
population regression function.
• We will never know the PRF, so we cannot tell how close the SRF is to the PRF.
• Another sample of data will give a different regression line, which may or may
not be closer to the population regression line.
Algebraic Properties of OLS Statistics
[Link] sum, and therefore the sample average of the OLS
residuals, is zero. Mathematically,
σ𝑛𝑖=1 𝑢ෝ𝑖 = 0 ….(32),
and hence, 𝑢=0 തො ……….(33)
This property needs no proof; it follows immediately from
the OLS first order condition (eqn(29)).
2. The sample covariance between the regressors and the
OLS residuals is zero.
Cov(𝑥𝑖 , 𝑢ෝ𝑖 )= 0 …..(34) (using the first algebraic
property above & using first order condition from (30))
Algebraic Properties continued..

3. The point (𝑥,ҧ 𝑦)

ത is always on the OLS regression line
(implied from eqn (32): 𝑦ത = 𝛽෢0 + 𝛽
෢1 𝑥)
ҧ
4. The sample average of the fitted/predicted values, 𝑦ෝ𝑖 , is
the same as the sample average of the 𝑦𝑖 , i.e., 𝑦തො = 𝑦ത ..(35)
Proof: since 𝑦𝑖 = 𝑦ෝ𝑖 + 𝑢ෝ𝑖 (writing each 𝑦𝑖 as its fitted value,
plus its residual)
 n-1σni=1 𝑦𝑖 = n-1σni=1 𝑦ෝ𝑖 + n−1 σni=1 𝑢ෝ𝑖
ത 𝑦തො (since σ𝑛𝑖=1 𝑢ෝ𝑖 = 0 from property 1)
𝑦=
5. The sample covariance between the fitted value 𝑦ෝ𝑖 𝑎𝑛𝑑 𝑢ෝ𝑖 is zero:
Cov(𝑦ෝ𝑖 , 𝑢ෝ𝑖 )= 0 ……………..(36)
തො 𝑢ෝ𝑖 − 𝑢)
Proof: Cov(𝑦ෝ𝑖 , 𝑢ෝ𝑖 )= (n-1)-1σni=1(𝑦ෝ𝑖 − 𝑦)( തො
= (n-1)-1σni=1[(𝑦ෝ𝑖 𝑢ෝ𝑖 ) − (𝑦തො 𝑢ෝ𝑖 ) − (𝑢തො 𝑦ෝ𝑖 ) + (𝑦തො𝑢)]
തො
= (n-1)-1σni=1(𝑦ෝ𝑖 𝑢ෝ𝑖 ) (all other terms above are zero due to property 1)
෡ +β
= (n-1)-1 σni=1(β ෡ x )ොu (as 𝑦ෝ𝑖 = β
෡ +β෡ x)
0 1 i i 0 1 i

෡ σn uො +(n−1)−1β
= (n-1)-1 β ෡ σn x uො
0 i=1 i 1 i=1 i i

= 0 (as σni=1 uො i =0 and σni=1 xi uො i =0 from the first-order conditions)

Thus, OLS decomposes each yi into two uncorrelated parts, a fitted
value and a residual.
SST, SSE, and SSR
Total sum of squares (SST) = σni=1 (yi - y
ഥ)2
• SST is a measure of the total sample variation in the yi ;
• SST measures how spread out the yi are in the sample. If we divide
SST by n-1, we obtain the sample variance of y.
Explained sum of squares (SSE) = σni=1 (ොyi - y
ഥ)2
• SSE measures the sample variation in the yො i (using the property 𝑦തො = 𝑦)
ത
Residual sum of squares (SSR) = σni=1 𝑢ො i2
• SSR measures the sample variation in the 𝑢ො I
Algebraic property 6: The total variation in y can always be expressed
as the sum of the explained variation and the unexplained variation SSR.
Thus,
SST = SSE + SSR …..(37)
Proof: LHS =SST =σni=1 (yi - y ഥ)2
= σni=1 (yi -ොyi +ොyi- y
ഥ)2
= i=1 [𝑢ො i+(ොyi- y
σ n ഥ)]2
= σni=1 𝑢ො i2 +2σni=1 𝑢ො i(ොyi- y
ഥ) + σni=1 (ොyi - y
ഥ)2
തො + σni=1 (ොyi - y
= σni=1 𝑢ො i2 +2σni=1 (𝑢ො i − 0)(ොyi- 𝑦) ഥ)2 (as
തො
𝑢=0 as per property 1 and 𝑦തො = 𝑦ത as per property 4)
= σni=1 𝑢ො i2 + σni=1 (ොyi - yഥ)2 = SSR +SSE (RHS).
തො
Note: 2 σni=1 (𝑢ො i − 0)(ොyi− 𝑦)=0 as sample covariance
between the fitted values and residuals is 0 (see
property 5).
Goodness-of-Fit: how well the explanatory variable, x, explains the
explained variable, y
• R-squared is a numeric indicator of how well the OLS
regression line fits the data
• The R-squared of the regression, sometimes called
2 SSE
the coefficient of determination, is defined as: R 
SST
SSR
=1− …(38) (using eqn (37)) (assuming SST0)
SST
• R2 is the ratio of the explained variation compared to
the total variation
• R2 is interpreted as the fraction of the sample
variation in y that is explained by x.
• the value of R2 is always between zero (poor fit) and one
(perfect fit), because SSE can be no greater than SST (from
eqn(37)).
• When interpreting R2 , we usually multiply it by 100 to change it
into a percent: percentage of the sample variation in y that is
explained by x.
• We can express 𝑅2 as the ratio of var(𝑦ෝ𝑖 ) and var(𝑦𝑖 )
𝑆𝑆𝐸 σn (𝑦ෞ𝑖 − y)
ഥ 2 𝑣𝑎𝑟(ෞ
𝑦𝑖 ) 𝑣𝑎𝑟(ෞ
𝑢𝑖 )
Proof: 𝑅 
2
= ni=1
= =1- ..(39) (using
𝑆𝑆𝑇 σi=1 (yi − y) ഥ 2 𝑣𝑎𝑟(𝑦𝑖 ) 𝑣𝑎𝑟(𝑦𝑖 )
(35): 𝑦തො = 𝑦ത and dividing by n-1 in numerator and denominator)
What is the value of 𝑅2 when the slope coefficient 𝛽
෢1 is zero?
(hint: use 𝑦ෝ𝑖 = 𝛽 ෢1 𝑥𝑖 in (39)).
෢0 + 𝛽
Goodness-of-Fit continued..

In fact, it can be shown that 𝑅2 is equal to the square

of the sample correlation coefficient between 𝑦𝑖 and 𝑦ෝ𝑖 .
𝐶𝑜𝑣 𝑦𝑖,ෞ
𝑦𝑖 𝐶𝑜𝑣 [ෞ
𝑦𝑖 +ෞ
𝑢𝑖 ],ෞ
𝑦𝑖
Proof: 𝑟𝑦𝑦
ො = = (using 𝑦𝑖 = 𝑦ෝ𝑖 + 𝑢ෝ𝑖 )
𝑣𝑎𝑟 𝑦𝑖 𝑣𝑎𝑟(ෞ
𝑦𝑖 ) 𝑣𝑎𝑟 𝑦𝑖 𝑣𝑎𝑟(ෞ
𝑦𝑖 )
ො 𝑦ො +𝐶𝑜𝑣(𝑦,ෞ
𝐶𝑜𝑣 𝑦, ො 𝑢𝑖 )
= (using
ො
𝑣𝑎𝑟 𝑦 𝑣𝑎𝑟(𝑦)
Cov(X+Y,Z)=Cov(X,Z)+Cov(Y,Z))
𝑣𝑎𝑟 𝑦ො
= (using Cov(X,X)=Var(X) and 𝐶𝑜𝑣(𝑦,
ො 𝑢ෝ𝑖 ) =0
ො
𝑣𝑎𝑟 𝑦 𝑣𝑎𝑟(𝑦)
from (36))
𝑣𝑎𝑟 𝑦ො
= = 𝑅2 (using (39)). Hence, 𝑅2 =𝑟𝑦𝑦
2
ො
𝑣𝑎𝑟 𝑦
The Effects of Changing Units of Measurement on OLS Statistics

• OLS estimates values are affected by changing the

units of measurement of the dependent and/or
independent variables but the interpretation remains
the same.
• Case 1: The dependent variable is multiplied by a
constant c: the OLS intercept and slope estimates are
also multiplied by c.
Proof: From (33) and (32) we know that OLS estimates
are calculated as:
σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦ത
෢1 =
𝛽 𝑎𝑛𝑑 ෢0 = 𝑦ത − 𝛽
𝛽 ෢1 𝑥ҧ
σ𝑛 ҧ 2
𝑖=1(𝑥𝑖 −𝑥)
(Original Model)
Now consider the modified model where the units of
measurement of the dependent variable changes where
each value of the dependent variable in the sample is
multiplied by c:
The OLS estimates for the modified slope and intercept
are:
෢1 = σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑐𝑦𝑖 −𝑐𝑦 ത σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦 ത
𝛽′ σ𝑛 ҧ 2 =c σ𝑛 ҧ 2 =c𝛽
෢1 ,
𝑖=1 (𝑥 𝑖 − 𝑥) 𝑖=1(𝑥𝑖 − 𝑥)
෢0 = 𝑐 𝑦ത − 𝛽′
𝛽′ ෢1 𝑥ҧ = 𝑐 𝑦ത − c𝛽
෢1 𝑥ҧ = c (𝑦ത − 𝛽
෢1 𝑥ҧ ) = c 𝛽
෢0
• Case 2: The independent variable is multiplied by a
constant c: the OLS intercept remains unchanged and
slope estimates are multiplied by 1/c.
෢1 = σ𝑛
𝑖=1 𝑐𝑥𝑖 −𝑐 𝑥ҧ 𝑦𝑖 −𝑦ത
Proof: 𝛽′
σ𝑛
𝑖=1(𝑐𝑥𝑖 −𝑐 ҧ
𝑥) 2

𝑐 σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦ത
= σ𝑛
𝑐2 𝑖=1(𝑥𝑖 − ҧ
𝑥) 2

1
= ෢1
𝛽 ,
𝑐
෢ ෢ 𝑐
𝛽′0 = 𝑦ത − 𝛽′1 c𝑥ҧ = 𝑦ത − 𝛽෢1 𝑥ҧ = 𝑦ത − 𝛽
෢1 𝑥ҧ = 𝛽
෢0
𝑐
• Example: In the previous example from (Wooldridge, 2018,
Ch-2) on the regression of roe (measured as percentage
e.g. 30 means roe= 30 %) on CEO’s salary (measured in
$1000 e.g. 1.123 means $1123), we estimated the
following OLS regression: 𝑠𝑎𝑙𝑎𝑟𝑦=
෣ 963.191 + 18.501 roe,
n=209
• Suppose now the unit of salary is changed from $1000 to $.
Let salardol be salary in dollars (salardol = 845,761 would
be interpreted as $845,761). Here, c=1000 is multiplied to
each observation on salary (dependent variable).
• So, the new estimated OLS regression equation is:
෣
𝑠𝑎𝑙𝑎𝑟𝑑𝑜𝑙= 963191 + 18501 roe
• Note: The interpretation remains the same.
• R-squared is invariant to changes in the units of y or x.
σi=1 (𝑦
ෞ𝑖 − y) 2
n 2
𝑆𝑆𝐸 ഥ 𝑣𝑎𝑟(ෞ
𝑦𝑖 ) ෢ ෢
𝑣𝑎𝑟(𝛽0 + 𝛽1 𝑥𝑖 ) ෢
𝛽1 var(𝑥𝑖 )
Proof: 𝑅 
2 = n = = =
𝑆𝑆𝑇 σi=1 (yi − y)ഥ 2 𝑣𝑎𝑟(𝑦 𝑖 ) 𝑣𝑎𝑟(𝑦 𝑖 ) 𝑣𝑎𝑟(𝑦𝑖 )
Suppose each y is multiplied by a constant c. Then:
෢0 + 𝛽′
𝑣𝑎𝑟(𝛽 ෢1 𝑥𝑖 )
෢1 = σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑐𝑦𝑖 −𝑐 𝑦
ത σ𝑛
𝑖=1 𝑥𝑖 −𝑥ҧ 𝑦𝑖 −𝑦ത
𝑅2 = , where 𝛽′ σ𝑛 ҧ 2
=c σ𝑛 ҧ 2
𝑣𝑎𝑟(𝑐𝑦𝑖 ) 𝑖=1(𝑥𝑖 −𝑥) 𝑖=1(𝑥𝑖 −𝑥)
=c𝛽
෢1
෢1 2 var(𝑥𝑖 )
𝛽′
= (using properties of variance)
𝑐 2 𝑣𝑎𝑟(𝑦𝑖 )
2
෢1 var(𝑥𝑖 )
𝑐2𝛽
= = 𝑅2 (𝑐 2 cancels out, then see above formula for 𝑅2 )
𝑐 2 𝑣𝑎𝑟(𝑦𝑖 )
Incorporating popular (Nonlinearities) functional forms

• So far we have seen the use cases where both

dependent variable and independent variable are in
level terms.
• It is common to use log-transformed variables in
economics and econometrics analysis.
• How do we interpret the regression coefficients
when the dependent or/and independent variables
are not in level terms?
Log-level form: The dependent variable is in log-term
and the independent variable is in level term.
E.g: log (wage)= β0+ β1educ+u
If ∆u = 0, then %∆wage ≈(100 . 𝛽1 ) ∆educ
Hence, 𝛽1 multiplied by 100 gives the percentage change
in wage (y) given one additional year of education (one
additional unit of x). The model gives a constant
percentage effect.
If the model estimated using OLS is:
෣
log(wage)= 0.584+ 0.083 educ , then wage (not
log(wage)) increases by 8.3% for every additional year
of education. 100*β1 is also called semi-elasticity of y with
respect to x.
Why is this done?
Intuition:
• If we regress hourly wage on years of education, we obtain a slope
estimate of x, which means that each additional year of education is
predicted to increase hourly wage by x units.
• Because of the linear nature of the model, x units is the increase for
either the first year of education or the twentieth year; this may not
be reasonable.
• A better characterization of how wage changes with education is that
each year of education increases wage by a constant percentage.
• For example, an increase in education from 5 years to 6 years
increases wage by, say, 8% (ceteris paribus), and an increase in
education from 11 to 12 years also increases wage by 8%
(Wooldridge, 2018, Ch-2).
Log-log form:
• Both the dependent and independent variables are log-transformed.
• This gives a constant elasticity model.
• E.g., We can estimate a constant elasticity model relating CEO
salary to firm sales. Let sales be annual firm sales, measured in
millions of dollars. A constant elasticity model is
log (salary)= β0+ β1log(sales) +u ,
where β1 is the elasticity of salary (y) with respect to sales (x).
• Suppose the estimated OLS eqn is:
෣
log (salary)=4.822+ 0.257 log(sales)
The coefficient of log(sales) is the estimated elasticity of salary with
respect to sales. It implies that a 1% increase in firm sales is
associated with an increase in CEO salary by about 0.257%.
Source: Wooldridge(2018, Ch-2)
• Slope are not affected if we change the units of
measurement of the dependent variable or independent
variable when it appears in logarithmic form.
Proof: l𝑜𝑔 𝑦𝑖 = 𝛽0 + 𝛽1 𝑥𝑖 + 𝑢𝑖
Add log (c) to both sides:
log c + l𝑜𝑔 𝑦𝑖 = log 𝑐 + 𝛽0 + 𝛽1 𝑥𝑖 + 𝑢𝑖
l𝑜𝑔 𝑐𝑦𝑖 = [log 𝑐 + 𝛽0 ] + 𝛽1 𝑥𝑖 + 𝑢𝑖 (using
log(ab)=log(a) +log(b))
• Hence, the slope remains 𝛽1 and the intercept changes
to [log 𝑐 + 𝛽0 ]
• Similarly, the slope remains the same and the intercept
changes when we multiply each term of the
independent variable x with a constant c before taking
its log.
• 𝑦𝑖 = 𝛽0 + 𝛽1 log 𝑥𝑖 + 𝑢𝑖 (original equation)
𝑦𝑖 = 𝛽0 − 𝛽1 log 𝑐 + 𝛽1 log 𝑐 + 𝛽1 log 𝑥𝑖 + 𝑢𝑖 (adding and
subtracting 𝛽1 log 𝑐 )
𝑦𝑖 = 𝛽0 − 𝛽1 log 𝑐 + 𝛽1 (log 𝑐 + log 𝑥𝑖 ) + 𝑢𝑖
𝑦𝑖 = [𝛽0 −𝛽1 log 𝑐 ] + 𝛽1 log 𝑐𝑥𝑖 + 𝑢𝑖 (slope remains 𝛽1 and
the intercept changes to [𝛽0 −𝛽1 log 𝑐 ])
What does “linear” mean in simple linear regression?
• The discussion of the regression model involving various log-
transformed variables show that the general regression
framework also allows certain non-linear relationships.
• The general regression equation: 𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝑢, is linear in
parameters 𝛽0 and 𝛽1 .
• There are no restrictions on how y and x relate to the
original explained and explanatory variables of interest, e.g.
log-transformed y and x, square-root or square of y and x,
etc.
• The mechanics of simple regression do not depend on how y
and x are defined.
• However, the interpretation of the coefficients does depend
on the functional form of y and x (Wooldridge, 2018, Ch-2).
Statistical Properties of OLS: Expected Values and Variances of the OLS
estimators
• 𝛽መ0 and 𝛽መ1 are estimators for the population
parameters 𝛽0 and 𝛽1 .
• Our focus here is to study the properties of the
distributions of 𝛽መ0 and 𝛽መ1 over different random
samples from the population.
• We study the statistical properties of OLS
estimators under a set of assumptions as follows:
• Assumption SLR 1: Linear in parameters:
Population model is given by: y = β0 + β1x + u ,
where β1 is the slope parameter and β0 is the
intercept parameter.
• Assumption SLR 2: Random Sampling: We have a
random sample of size n, {(xi,yi); i=1, . . . , n},
following the population model. We can write the
population equation in terms of the random sample
as yi = β0 + β1xi + ui …….(40), i =1,2,….,n.
Here, ui is the error or disturbance for observation i
(not the residuals).
• Assumption SLR.3 – Sample variation in the
explanatory variable: the OLS slope and intercept
estimates are not defined unless we have some
sample variation in the explanatory variable (see
𝛽መ1 estimation formula).
• Assumption SLR.4- Zero Conditional Mean: The error u
has an expected value of zero given any value of the
explanatory variable , i.e. E(u|x)=0.
• For a random sample (SLR.2), this assumption (SLR.4)
implies that E(ui|xi)= 0, for all i= 1, 2,.., n.
• From SL4.2 and SLR.4, we can derive the statistical
properties of the OLS estimators as conditional on the
values of the xi in our sample.
• Once we assume the E(u|x)=0 and we have random
sampling, we can treat xi as non-random in deriving the
statistical properties of the OLS estimators.
Data generating process

• Technically, in statistical derivations, conditioning on

the sample values of the independent variable is the
same as treating the xi as fixed in repeated samples
(this is the assumed data generating process for
technical simplification).
• Intuitively, we first choose n sample values for
𝑥1 , 𝑥2 , … , 𝑥𝑛 . (These can be repeated) Given these
values, we then obtain a sample on y. Next, another
sample of y is obtained, using the same values for
𝑥1 , 𝑥2 , … , 𝑥𝑛 , and so on.
Expressing 𝜷
෡ 𝟏 in terms of 𝜷𝟏

• Since σni=1 (xi − xത) (yi – yത ) = σni=1 yi (xi − xത) (Same as eqn. 19)
෠ σni=1 yi (xi − x ത)
• We can write, β1= σn 2
i=1 ( xi − ത
x )
• β෠ 1 is a random variable in this context as we are interested in the
behaviour of β෠ 1 across all possible random samples from the
population.
• Now, we write β෠ 1 in terms of the population coefficient and errors
෠ i=1 (xi − x
σn ത ) (β0 + β1xi + 𝑢𝑖 )
as β1= SST
(here we have defined
X
σni=1 (xi − xത)2 as SSTX to simplify the notation.)
σni=1 (xi − xത) (β0 + β1xi +𝑢𝑖 )
n n n
= β0 i=1 (xi − xത) +β1 i=1 (xi − xത)xi+ i=1 ui (xi − xത)
σ σ σ
n
= β1SSTX + σi=1 ui (xi − xത)
[Because σni=1 (xi − xത) =0 and
σni=1 (xi − xത)xi = σni=1 (xi − xത)2 = SSTX ]
β෠ 1 = β1+ (1/𝑆𝑆𝑇𝑋 )(σ𝑛𝑖=1 diui) (where di = xi − xത ) ….(41)
Statistical Property 1: Unbiasedness of OLS estimators
• Using assumptions SLR.1 to SLR.4, E(β ෡ ) = β and
0 0
෡ ) = β for any values of β and β .
E(β
1 1 0 1

• In other words, β෡ is unbiased for β , and β

෡ is
0 0 1
unbiased for β1.
• Proof: In this proof, the expected values are
conditional on the sample values of the independent
variable. Because SSTx and di are functions only of
the xi, they are non-random.
• We have β ෡ = β + 1/SST (σn d u )
1 1 X i=1 i i
• Keeping the conditioning on {x1, x2 , x3 …. }
implicit, we have
• E(β෡ ) = β + E [1/SST (σn d u )]
1 1 X i=1 i i
= β1+ 1/SSTX E( i=1 diui)
σ n

= β1+ 1/SSTX i=1 di .0 = β1 [as E(σni=1 diui)= σni=1 diE(ui)]

σn

• where we have used the fact that the expected

value of each ui (conditional on {x1, x2 , x3 …. }
is zero under Assumptions SLR.2 and SLR.4.
Since unbiasedness of the OLS slope estimator
holds for any outcome on {x1, x2 , x3 …. },
unbiasedness also holds without conditioning on
{x1, x2 , x3 …. }
• Now, β ෢1 xത …(42)
෢0 = yത − β
• From (40): yi = β0 + β1xi + ui , taking average gives:
𝑦ത = 𝛽0 + 𝛽1 𝑥ҧ + 𝑢ത …(43)
Using (43) in (42), we get:
β෢0 = 𝛽0 + 𝛽1 𝑥ҧ + 𝑢ത − β
෢1 xത
෢0 = 𝛽0 + 𝑥(𝛽
β ෢1 ) +ത
ҧ 1− β 𝑢
Then, conditional on the values of the xi:
E(β
෢0 ) = 𝛽0 + 𝐸[𝑥(𝛽
ҧ 1−β ෢1 )] +E(ത
𝑢)
E(β
෢0 ) = 𝛽0 (as E(β෢1 )=E(𝛽1 ) and E(𝑢ത )=0 from slr 2 and 4)
• Unbiasedness generally fails if any of our four SLR
assumptions fail (Wooldridge, 2018, Ch-2).
Variances of the OLS Estimators
• Variance or its square root (standard deviation) of
the OLS estimators measures the spread in the
෡ (an𝑑 β
distribution of β ෡ ), i.e., how far we can
1 0
expect β෡ (and β෡ ) to be away from β (and β ) on an
1 0 1 0
average.
• Ideally, we want the estimators to have minimum
variance.
• We add another assumption, SLR 5, that is
traditional for cross-sectional analysis to compute
the variance of OLS estimators.
• SLR.5 Homoskedasticity: The error u has the same
variance given any value of the explanatory variable. In
other words, Var(u|x) = 𝜎2
• This assumption states that the variance of the
unobservable, u, conditional on x, is constant. This is
known as the homoskedasticity or “constant variance”
assumption.
• Var(u|x) =E(u2|x)- [E(u|x)]2
E(u2|x) = 𝜎2 [as E(u|x)=0 from SLR4 ]
 E(u2)= 𝜎2 ……(43) [as E(u2|x)= E(u2) see discussion
on unbiasedness]
which means 𝜎2 is also the unconditional expectation of u2
Now, Var(u)= E(u2)- [E(u)]2
Var(u)= E(u2) [as E(u)=0] ….(44)
So, from (43) and (44): Var(u)= 𝜎2
• In other words, σ2 is the unconditional variance of u,
and so σ2 is often called the error variance or
disturbance variance. The square root of σ2, σ, is the
standard deviation of the error.
• When Var(u|x) depends on x, the error term is said to
exhibit heteroskedasticity (or nonconstant variance).
• SLR.5 can be written in terms of the conditional
variance of y as Var(y|x)= 𝜎2 (use u =y- β0- β1x)
• Variance of the slope estimate: Under
Assumptions SLR.1 to SLR.5
• Var (β෡ ) = 𝜎 2 / σ𝑛 (x − 𝑥)ҧ 2 = 𝜎2/ 𝑆𝑆𝑇 , where it is
1 𝑖=1 i 𝑋
conditional on the sample values of x. This formula
is invalid in the presence of heteroskedasticity
• Proof of Var (β ෡ ) = 𝜎2/ SST
1 X
෡ = β + 1/SST (σn d u )
• From (41): β 1 1 X i=1 i i
Here, β1 is just a constant, and we are conditioning
on the xi, SSTx and di= xi − xത are also non-random.
Furthermore, because the ui are independent random
variables across i (by random sampling), the variance of
the sum is the sum of the variances.
෡ )
• Using these facts, we have, LHS=Var (β 1
= Var (β1)+ Var [1/SSTX (σni=1 diui)]
= (1/SSTX)2 σni=1 di2 Var(ui)
= (1/SSTX)2 σni=1 di2 𝜎2 [since Var(ui) = 𝜎2 for all 𝑖]
= SSTX /SSTX2 𝜎2
= 𝜎2 /SSTX =RHS …(45)
• From (45), the variance of β෠ 1 depends directly on the error variance
and inversely on the total variation in the xi
• The larger the error variance, the larger is Var (β෠ 1). This suggests that
more variation in the unobservable affecting y makes it more difficult
to precisely estimate β1.
• On the other hand, more variability in the independent variable is
preferred: as the variability in the xi increases, the variance of β෠ 1
decreases. This means that the more spread out is the sample of
independent variables, the easier it is to trace out the relationship
between E(y|x) and x. That is, the easier it is to estimate β1.
• If there is little variation in the xi, then it can be hard to pinpoint how
E(y|x) varies with x. As the sample size increases, so does the total
variation in the xi. Therefore, a larger sample size results in a smaller
variance for β෠ 1
Estimating the Error Variance
• The variance of the estimators depend on the
variance of the error term. The errors are never
observed, while the residuals are computed from
the data. we can use the data to estimate error
variance, which then allows us to estimate variance
of the estimators.
• 𝜎2 =Var(u) = E(u2)-E(u)2 = E(u2) [as E(u)=0]
• Replacing the errors with the OLS residuals, an
1 𝑛
estimator of 𝜎 = σ𝑖=1 𝑢ෝ𝑖 2 =SSR/n (here, 𝑢ෞ𝑖 serve
2
𝑛
as the estimates of 𝑢𝑖 since 𝑢𝑖 are not observed)
• The estimator SSR/n is biased because it does not
account for two restrictions that must be satisfied by the
OLS residuals. These restrictions are given by the two
OLS first order conditions: σ𝑛𝑖=1 𝑢ෝ𝑖 =0 and σ𝑛𝑖=1 xi 𝑢ෝ𝑖 =0
• One way to view these restrictions is this: if we know n-2
of the residuals, we can always get the other two
residuals by using the restrictions implied by the first
order conditions above. Thus, there are only n-2 degrees
of freedom in the OLS residuals, as opposed to n degrees
of freedom in the errors.
• The unbiased estimator of 𝜎2 is 𝜎ො 2
1
= σ𝑛𝑖=1 𝑢ෝ𝑖 2
𝑛−2
= SSR/n-2
• If 𝜎ො 2 is plugged into the variance formula
෡
Var (β ) = 𝜎 / 2 σ 𝑛
(x − 𝑥)ҧ 2
1 𝑖=1 i
= 𝜎2/ 𝑆𝑆𝑇𝑋
෡ )
then we have unbiased estimators of Var (β 1
• Later in hypothesis testing, we will need
estimators of the standard deviations of β෡ , and
1
this requires estimating 𝜎.
• The natural estimator of 𝜎 is √𝜎ො 2 = 𝜎ො and is
called the standard error of the regression
(SER). (Other names for 𝜎ො are the standard
error of the estimate and the root mean squared
error)
• The estimate 𝜎ො is an estimate of the standard
deviation in the unobservables affecting y;
equivalently, it estimates the standard deviation
in y after the effect of x has been taken out.
• Standard error of β ෡ se(β ෡ ) = 𝜎ො /√ 𝑆𝑆𝑇
1 1 𝑋
Reference Books
• Gujarati and Porter, Basic Econometrics, Fifth Edition, McGraw
Hill/Irwin, 2009.
• Wooldridge, J., Introductory Econometrics: A Modern Approach 7th
Edition, 2018, Cengage Learning.
• Dougherty, C. (2011). Introduction to econometrics. Oxford university
press, USA.