Tutorial Guide Week 5: Linear Models
DON’T FORGET TO FILL UP YOUR ESSFS
Let’s START with a summary of the Steps in building a Linear Model
Exercises start at page 12.
How is the linear model postulated?
1. As a summary of the scatterplot, a pattern that describes the data.
(𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ),…, (𝑥𝑛 , 𝑦𝑛 )
(𝑥1 , 𝑦1 )
(𝑥1 , 𝑦1 )
, (regression coefficients)
Note: without X, we summarize (𝑦1 , … . , 𝑦𝑛 ) in terms of 𝑦̅
2. As a conditional expectation of 𝒚 given 𝒙
𝐸 (𝑦|𝑥 ) = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + ⋯ + +𝛽𝑝 𝑥𝑝
Example in Econometrics: Economic Growth conditional on Interest Rate
Example in Business Analytics:
Sales conditional on Expenses on Digital Marketing Campaigns
What are the statistical inference elements associated with linear models?
Describe the population⟺Characterize Patterns
Estimate Population Characteristics ⟺ Estimate Parameters/Slopes
Test Hypothesis (parameters/characteristics) ⟺ Test Hypothesis
(regression coefficients/slopes)
1
Steps in building a linear model or regression model?
STEP 1: Translate layman’s problem into a linear model
Layman:
What is the effect of inflation (INF) on GDP growth (GDP)?
What is the effect of interest rate (INT) on GDP growth (GDP)?
Linear Model:
𝐺𝐷𝑃𝑖 = 𝛽0 + 𝛽1 𝐼𝑁𝐹𝑖 + 𝛽2 𝐼𝑁𝑇𝑖 + 𝜖𝑖
Interpretation:
𝛽1 change in GDP growth for a unit change in INF holding the effect of
INT constant.
𝛽2 change in GDP growth for a unit change in INT holding the effect of
INF constant.
𝜖𝑖 remaining variability in GDP growth not explained by INF and INT.
STEP 2: Collect the Data
With 1 predictor, random sample is the tuple:
(𝑦1 , 𝑥1 ), (𝑦2 , 𝑥2 ), … , (𝑦𝑛 , 𝑥𝑛 )
With more than 1 predictor, random sample is:
(𝑦1 , 𝑥11 , 𝑥21 , … , 𝑥𝑝1 )
(𝑦2 , 𝑥12 , 𝑥22 , … , 𝑥𝑝2 )
… … … … …
(𝑦𝑛 , 𝑥1𝑛 , 𝑥2𝑛 , … , 𝑥𝑝𝑛 )
In reality, we don’t really know how the data (sample) is linked to the
population. Consider simulated data!
Example: Simulated Data
X1, X2, X3 are structurally uncorrelated (DATA1)
2
X3 is Computed from X2 (correlated) (DATA2)
Messy Data, Error with Large Variance (DATA3)
This is how DATA1 looks like!
3
STEP 3: Explore the Data
o Explore data without any clear ideas of what to expect
– Let the data speak for itself!
o Before you use the most sophisticated methods and models
– Understand the data first
– Explore for patterns and anomalies
o Methods and Models capturing what the data is all about
=>more sensible analytics strategies!
Summary Measures
4
Histogram
Pointing out: Shape, Central tendency, Spread, Extreme values
5
Data 2: Few Small Values for Y
Data 3: More Spread Values for Y
Scatterplots
6
DATA1
7
DATA2
8
DATA3
STEP 4: Postulate a model
𝑦𝑖 = 𝛽0 + 𝛽1 𝑥1𝑖 + 𝛽2 𝑥2𝑖 + ⋯ + 𝛽𝑝 𝑥𝑝𝑖 + 𝜖𝑖
Assumptions:
- 𝜖𝑖 ~𝑖𝑖𝑑 𝑁(0, 𝜎 2 )
o normality, constant variance, independent errors
- 𝑥𝑗𝑖 and 𝜖𝑖 are uncorrelated
o misspecification error
′
- 𝑥𝑗𝑖 𝑠 and independent (weakly correlated at most)
o Multicollinearity
Summary of the data as conditional expectation of 𝑦 given 𝑥′𝑠
⇒ 𝐸(𝑦𝑖 |𝑥1𝑖 , … , 𝑥𝑝𝑖 ) = 𝛽0 + 𝛽1 𝑥1𝑖 + 𝛽2 𝑥2𝑖 + ⋯ + 𝛽𝑝 𝑥𝑝𝑖 𝑠𝑖𝑛𝑐𝑒 𝐸(𝜖𝑖 ) = 0
9
STEP 5: Estimate the model
a. Ordinary Least Squares (OLS).
2
Minimize ∑𝑛𝑖=1 (𝑦𝑖 − 𝛽0 − 𝛽1 𝑥1𝑖 − 𝛽2 𝑥2𝑖 − ⋯ − 𝛽𝑝 𝑥𝑝𝑖 )
b. Maximum Likelihood Estimation (MLE): Maximize
𝐿(𝛽0 , 𝛽1 , … , 𝛽𝑝 |𝑥1 , … , 𝑥𝑝 )
𝐼𝑓 𝜖𝑖 ~𝑁𝑜𝑟𝑚𝑎𝑙
𝑛
(𝑦𝑖 − 𝜇 )2
𝑙(𝛽0 , 𝛽1 , … , 𝛽𝑝 ) = ∑ [− log(√2𝜋𝜎) − ]
2𝜎 2
𝑖=1
𝑛 2
(𝑦𝑖 − 𝛽0 − 𝛽1 𝑥1𝑖 − 𝛽2 𝑥2𝑖 − ⋯ − 𝛽𝑝 𝑥𝑝𝑖 )
= ∑ [− log(√2𝜋𝜎) − ]
2𝜎 2
𝑖=1
It can be easily observed that 𝐎𝐋𝐒 = 𝐌𝐋𝐄
c. Gauss-Markov Theorem (Optimality of OLS)
10
STEP 6: Test certain hypothesis
a. Test for the existence of a linear model (ANOVA)
𝑦𝑖 = 𝛽0 + 𝛽1 𝑥1𝑖 + 𝛽2 𝑥2𝑖 + ⋯ + 𝛽𝑝 𝑥𝑝𝑖 + 𝜖𝑖
How much variability in 𝑦 is explained by the PINK relative to the RED?
Too much of RED⟹ No LINEAR MODEL EXIST!
𝐻0 : 𝛽1 = 𝛽2 … = 𝛽𝑝 = 0
⟺No Linear Model Exist PINK is dominated by RED
Summary of Analysis of Variance
Sources DF SS MS F-Ratio
Regression p SSR SSR/p MSR/MSE
Error n-p-1 SSE SSE/n-p-1
Total n-1 TSS
b. Significance of Individual Parameters (variable selection)
Are both INF and INT needed to explain GDP growth?
𝐻0 : 𝛽𝐼𝑁𝐹 = 0 𝑣𝑠 𝐻𝑎 : 𝛽𝐼𝑁𝐹 ≠ 0
𝐻0 : 𝛽𝐼𝑁𝑇 = 0 𝑣𝑠 𝐻𝑎 : 𝛽𝐼𝑁𝑇 ≠ 0
𝑡 − 𝑡𝑒𝑠𝑡
c. Assess usefulness of the model (Predictive Ability)
Can the model be used to characterize (or predict) 𝑦 in terms of the 𝑥′𝑠
through the linear model?
𝑅 2 , MAPE
11
Activities:
1. Simulate data (DATA1) from the following (TRUE Model):
𝑦 = 2 + 3𝑥1 − 0.2𝑥2 + 1.5𝑥3 + 𝜖
Where 𝜖~𝑁(0,1)
a. Construct scatterplots. Identify insights it will give you.
b. Estimate the model 𝑦 = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽3 𝑥3 using DATA1. What
insights can you extract on building a linear model?
2. Simulate data (DATA2) from the same model as DATA1 (TRUE Model):
𝑦 = 2 + 3𝑥1 − 0.2𝑥2 + 1.5𝑥3 + 𝜖
Where 𝜖~𝑁(0,1)
but, 𝑥3 = 2𝑥2 + 𝑤 where 𝑤~𝑁(0,0.25)
𝑥2 and 𝑥3 are structurally related, the multicollinearity problem exists.
a. Estimate the model with DATA2. What insights it will give you?
b. Remove the insignificant 𝑥2 from the model? What have you observed?
3. Simulate DATA3 as follows:
𝑦 = 2 + 3𝑥1 − 0.2𝑥2 + 1.5𝑥3 + 𝜖
Where 𝜖~𝑁(0,49)
a. Construct scatterplots. Identify insights it will give you.
b. Estimate a linear model using DATA3. What insights have you observed?
4. Data on annual GDP growth at constant prices, Inflation (annual change in
consumer prices) and Interest Rates charged by private banks on loans for
Malaysia was obtained from the World Bank Development Indicators for the
period 1970 until 2022. The policy team would like to focus on these 2
Instruments (Inflation and Interest Rate) in planning for GDP growth. Construct
a model that link GDP to Inflation and Interest Rate. The data is in the file
“GDP_Malaysia.csv”.
12
5. Data on productivity of a garment industry is obtained from University of
California Irvine Data Repository ([Link] given in the file
“[Link]”. The dataset includes the following indicators:
Develop/build a productivity model (dependent variable: actual_productivity)
that can advise the teams to achieve their production targets.
6. Try GDP Growth and Inflation+Interest rate in any country, you can download the
data here: [Link]
13