Lecture 5

Statistical Inference

Yexiao Xu

Jindal School of Management

The University of Texas at Dallas
1
Outline
• The goodness of fit
- R2 versus adjusted R2
• Statistical inference
- Hypothesis
- Size versus Power
• Inference under the normality assumption
- Testing single restriction
- One-side versus two-side test
• Testing multiple restriction
- Univariate versus joint tests
- Imposing restrictions
2
 5.1 Goodness of Fit
The Goodness of Fit
• An overall measure of a model’s explanatory power
- How much variation in y can be explained by predicted value "! !
• Defined:
- TSS = ∑$!"# 𝑦! − 𝑦$ %
: Total Sum of Square
q Total variation we want to explain
%
- ESS =∑$!"# 𝑦%! − 𝑦$% : Explained Sum of Square
q The variation explained by our model
- RSS = ∑$!"# 𝑒! %
: Residual Sum of Square
q The variation cannot be explained by our model
• Decomposition: TSS = ESS+RSS
- This is only true when there is an intercept in regression model
• Define a Coefficient of Determination, denoted as R2
- R2 = ESS/TSS
- It is a Goodness of Fit measure, and is between zero and one
3
 5.1 Goodness of Fit
Alternative Definition of R2
• There are other ways to define R2
- The following three definitions are only equivalent
when there is an intercept term in the regression
• Definition 1: (original)
%
"## ∑$ & ! (')&
!"# '
- 𝑅! = $##
= ∑$ ) %
!"# '! ('

• Definition 2:
$##(*## ∑$!"# +!
%
- 𝑅! = $##
= 1 − ∑$ ' (') %
!"# !

• Definition 3: %
∑$ )&
) ('& ! (')
!"# '! ('
- 𝑅! = 𝐶𝑜𝑟𝑟 ! 𝑦, 𝑦* = %
∑$ )
!"# '! ('
% ∑$ & ! (')&
!"# '
4
 5.1 Goodness of Fit
Compare Different Definitions of R2
• They only measure linear goodness of fit
• When there is no intercept term in the regression
model
- R2 under the first definition could exceed one
- R2 under the second definition could be less than zero
• R2 under the Definition 3 is always between zero
and one
- It does not necessarily have the same meaning as the
original one
- But it can be used in many other cases
5
 5.1 Goodness of Fit
The Adjusted R2
• What happens when the number of regressors increases?
- ESS will increase! (why?)
q Consequently, R2 will increase
- But, there is a loss of degree of freedom at the same time
q Degree of freedom measures the amount of independent information
q The efficiency will drop
• The adjusted coefficient of determination
- To balance the above two factors, we can define an adjusted R2
$&#
$
- 𝑅 = 1 − $&' 1 − 𝑅%
#

- The adjusted R2 only increase when the additional included x

has sufficient explanatory power
- It can be used to compare different models
• Note:
- The adjusted R2 could be less than one 6
 5.2 Statistical Inference
What Is Statistical Inference?
• An estimate is just one possible realization
- Is it close to the true parameter value!
• Statistical inference
- It is the process of drawing conclusions on certain relations
implied by the model using data that are subject to random
variation (random sample)
• Due to the stochastic nature of the sample, distributional
assumptions about the sample are critical
• There are two Statistical Inference methods
- The Test of Significance approach
q Testing whether a relationship involving the model parameters holds
with certain likelihood
- The Confidence Interval approach
q A region that a relationship might holds with certain confidence
7
 5.2 Statistical Inference
Forming Hypothesis
• Statistical inference starts with “guessing” a relationship
- A hypothesis is a statement about a model’s parameters
• Hypotheses should be based on a model’s implications
- Directly derived from a mathematical model
q e.g. the CAPM model
- Implications of a model
q e.g. including other variables in testing the CAPM
• Or, based on a logical reasoning
- The trade-off theory of capital structure
q Cash flow volatility, tangibility of assets, profitability
q New variable: volatility of the firm cash flow variance
• A joint-test issue: when rejecting a hypothesis we may not
reject a theory we intend to reject because of other
possibilities
- Reject the CAPM model or the efficiency of a market proxy? 8
 5.2 Statistical Inference
Null vs. Alternative Hypotheses
• Null hypothesis (Yes)
- Assertion that a relation holds or what a theory predicts
q For example, H0: β = 3
• Alternative hypothesis (No)
- Assertion that a relation is false (HA : β ≠ 3)
• A different way to specify the Null hypothesis
- Sometimes theory is silent on a particular value of a relationship
q But the alternative hypothesis can only take one value
- Instead of testing whether a theory holds or not, we specify the
Null being Theory Doesn’t Hold (e.g. H0 : β = 0 )
q This is called testing the significance of a variable
• When the null is not rejected,
- It is wrong to say that we accept the hypothesis!!!
- Instead, we should say “fail to reject” a hypothesis since we
don’t enough evidence 9
 5.2 Statistical Inference
Statistical Tests
• We cannot make an “exact” judgment on a hypothesis
- This is because we only have estimates not parameters
- But we can use statistical tests to see how likely a hypothesis holds
• A test is a decision rule
- Used to verify the truth or falsity of a null hypothesis
• A test statistic is a single measure of some attributes of a
sample (about the null hypothesis)
- It allows us to make a decision based on the value of the test statistic
• A good test statistic needs to be invariant to the underlying
model parameters, or model structure
- So that we do not need different decision rules for different H0
• The significance of a test
- It is the probability of observing a certain value of a test statistic
(that leads to the conclusion), or the likelihood for H0 to be true
10
 5.2 Statistical Inference
Introducing a Statistical Test
Type II error
Area
represents
significance

𝛽!
H0 c 𝛽!(")
• Question: is H0: 𝛽 = 𝑏 true? (i.e. b is a hypothesized value)
• Since we do not observe 𝛽, we can only rely on 𝛽,
- Assume: 𝛽! ~ 𝑁(𝑏, 𝜎!" )
• Test: establishing the following rule under H0: 𝛽 = 𝑏
- Reject H0 if the estimate 𝛽!($) exceeds a critical value c , i.e. 𝛽!($) > 𝑐
- The dark green area represents the likelihood of wrongly reject H0
• Issue:
- c dependents on the distribution of 𝛽, ! i.e. b and 𝜎!"
- Solution: use a test statistic z = (𝛽! − 𝑏)/𝜎! instead
q In this case, z ~ N(0, 1), we can compute c value given the likelihood 11
 5.2 Statistical Inference
Mistakes in Hypothesis Testing

Type II error
Type I error

𝛽
H0 c HA
• It is impossible to conclude with 100% confidence
- We are dealing with a particular sample & using a specific rule
• Type I error:
- The likelihood of rejecting the null hypothesis when it is true
- It is also called the significance (or confidence) level
• Type II error:
- The likelihood of failing to reject the null when it is false
12
 5.2 Statistical Inference
Size versus Power of a Test
• We cannot reduce both Type I and Type II errors at the
same time
- Reducing Type I error means increasing Type II error
• The size of a test: (the significance level)
- the probability of rejecting a null hypothesis when it is true
- that is, Size = Type I error (the green area)
• The power of a test:
- the probability of rejecting a null hypothesis when is false
- that is, Power = 1 – Type II error (the purple area)
• Researcher can choose the size (a) of a test
- Choice of a small size could lead to a low power of a test
- When fail to reject, it could be due to the fact that the power
of a test is too low
13
 5.3 Statistical Inference Under Normality
OLS Estimator Under Normality
• To do a test, we first need the distribution of 𝛽!
• Consider a linear model:
- 𝑦& = 𝛽' + 𝛽( 𝑥&,( + 𝛽" 𝑥&," + ⋯ + 𝛽* 𝑥&,* + 𝜀&
- or yi = b’xi + εi,
Where 𝜷 = [𝛽$ , 𝛽% , 𝛽& , ⋯ , 𝛽' ]′ and 𝒙( = [1, 𝑥(,% , 𝑥(,& , ⋯ 𝑥(,' ]′
q

• Assume normal residuals, i.e., εi ~ N(0,σ2 )

• The estimator will have a normal distribution too
!' = # !" !! +* !" " = # !" !! +* !" # !! + "!
-
()$$$$!' & ! #!%$$! , # !" !! +* !
• Testing a simple hypothesis H0: βi = bi
• How can we reach a conclusion?
- Again, we can establish a rule (test) by comparing the
estimate with a critical value
- Under H0, we have 𝛽!& ~ 𝑁(𝑏& , 𝜎!"! )
14
 5.3 Statistical Inference Under Normality
Sample Distribution of A Test Statistic
• But a test statistic needs to be invariant to model
parameters, which can be done by “standardizing” it!
• Case I: σ2 is known
- We can construct a test statistics Z based on the hypothesis
% !+- !*)( - !
,=
" ! ' &%$#"!
-

- This is known as the Z test, and is an exact test

q Rule: comparing z value to c=1.96 with a significance level of 5%
• Case II: σ2 has to be estimated
- s2 is an estimator of σ2, which has a Chi-square distribution
- We can construct a t test statistics
, "+# )*)( # '
$= % $ " !!
& "#
- Rule: comparing t value to c with a significance level of 5%
q The value of c depends on the sample size n
15
 5.3 Statistical Inference Under Normality
Two-Side Test
• The significance test: a variable xi is significant (in
explaining y) if the following hypothesis is rejected
- H0 : βi = 0 versus HA : βi ≠ 0

• Pr( - ta/2, n-(K+1) < t < ta/2, n-(K+1) ) = 1- a

• Example: a = 5%, n-(K+1) = 20
- Critical value: c º ta/2, 20 = 2.086
- Reject if |t| > 2.086
- What does fail to reject mean?
• When n-(K+1) > 100,
- Critical value: c5% » 1.960, c1% » 2.576
16
 5.3 Statistical Inference Under Normality
One-Side Test
• The variable xi has a positive effect if the following
hypothesis is rejected
- H0 : βi ≤ 0 versus HA : βi > 0

• Pr( t < ta, n-(K+1) ) = 1- a

• Example: a = 5%, n-(K+1) = 20
- Critical value: c º ta, 20 = 1.725
- Reject if t > 1.725
- If you reject at 5% level, you also reject at a 10% level
• When n-(K+1) > 100,
- Critical value: c5% » 1.645, c1% » 2.326
17
 5.3 Statistical Inference Under Normality
One-Side Test
• The variable xi has a negative effect if the following
hypothesis is rejected
- H0 : βi ≥ 0 versus HA : βi < 0

• Pr( t > - ta/2, n-(K+1) ) = 1- a

• Example: a = 5%, n-(K+1) = 20
- Critical value: c º -ta, 20 = -1.725
- Reject if t < -1.725
• When n-(K+1) > 100,
- Critical value: c5% » -1.645, c1% » -2.326
18
 5.3 Statistical Inference Under Normality
Inference on Confidence Interval
• Construct a range that the true value bi is likely to fall in
- This is an equivalent to inference based on statistical test
) !% #& &
'
K!/0*#- # $ " 4 P2/ # 1 < ! !
< $ " 4 P2/ # 1 $ = , # "
' " !! $
( %
+*#'()*###!%! # $ " 4 P2/ # 1 #" ! ! < & ! < !%! + $ " 4 P2/ # 1 #" ! !

• The confidence interval (C.I.) is:

% !#! # #" ' &$% # $ ""! ! $""!#! + #" ' &$% # $ ""! ! !
• For a hypothesis H0 : βi = bi, we reject the hypothesis if
the hypothesized value bi lies outside the interval
• This might be a more flexible approach when dealing
with complicated distributions
19
 5.3 Statistical Inference Under Normality
Restrictions on Parameters
• A significance test only involves one variable
- Simple relation: e.g., β1 = 0
• A more complicated hypothesis involves several βs
- Example: β2 + 2 β4 = 3
- Note: covariances among parameter estimators are important
• A general linear relation
- H0 : r0β0 + r1β1 + r2β2 + · · · + rKβK = r β = q
• Conclusions about such a hypotheses (relation) can only
be drawn based on estimates
- When plugging in estimators, such a relation becomes a
random variable with a particular distribution
- If we consider r β as an “aggregate” parameter, the same test
procedure can be applied
20
 5.3 Statistical Inference Under Normality
Test a General Relation w/ Normality
• Again, consider a linear model:
yi = xi’b + εi
- with normal residuals, i.e., εi ~ N(0,σ2 )
• Since the OLS estimator has a normal distribution
!( ' &#!%$$! + # !" !! )* !
- Under H0: r β = q, the distribution of the restribution is:
! & !) &"&! ( # $ ! & !&"&!'&&!%! , $ !# !! *+ "!" !
• When σ2 has to be estimated
- We can construct a similar t test statistics
% ! ( '&'% #
!
#= + *)
! #"! !
!& $ % !$ !# "!"
- and use the same rule to make a decision
21
 5.4 Testing Multiple Restriction
Multiple Relations
• Consider a linear model: yi = xi’b + εi
- With a normally distributed residual, εi ~ N(0, σ2)
• Multiple relations
- H0 : β1 = β2 = · · · = βK = 0
- More complicated example:
β2 =1 and 2β1 + β3 = 1
• General linear restriction
𝑟#,) 𝛽) + 𝑟#,# 𝛽# + ⋯ +𝑟#,' 𝛽' = 𝑞#
- H0 : - ⋮
𝑟*,) 𝛽) + 𝑟*,# 𝛽# + ⋯ +𝑟*,' 𝛽' = 𝑞*

- H0 : R β = q
22
 5.4 Testing Multiple Restriction
The Classical ANOV Analysis
• The classical problem
- H0 : β1 = β2 = · · · = βK = 0
• What is the idea?
- Exam the difference in residual variance
• The F test
- Compute the sum of the estimated residuals (e’e), and,
- the total sum of variation in the dependent variable
(𝑦′
+ 𝑦)
+
- Then, construct the following F test statistics
'"
'$ "
' *&*%) %$&' ! ! !$ $ " # ' ! ! !$
- #= = "
" #! !!!" ! !
%) %&'( % #$ '! % $ $&'( % #$

23
 5.4 Testing Multiple Restriction
Testing General Multiple Relations
• If the J linear relations on β hold simultaneously
- H0 : Rβ = q
• The idea
- Sum of squared residuals with restrictions imposed (e∗’e∗) should be
equal to that w/o imposing the restriction (e’e) if H0 is true
• Implementation
- Step 1: compute the sum of the estimated residuals (e’e) for the
original model, and,
- Step 2: compute the sum of the estimated residuals (e∗’e∗) for the
constrained model
- The F test statistics
%&" ( &+ )')&( &" $ # % %"* ! % * " $ #
$= = *
# $# !" ! !
&( &$%& # !" %' # % "$%& # !"
• This is an exact test too under the normality assumption
24
 5.4 Testing Multiple Restriction
Univariate versus Joint Tests

• Rejection in univariate tests does not necessarily

mean rejection in a joint test
25
 5.4 Testing Multiple Restriction
How to Impose Restrictions
• Method I: the substitution method
- Example
• Method II: use formula
- under H0 : Rβ = q
- The constrained OLS estimator 𝒃
%) = !) ! # "! "" +* ! ! # !* #!!) ! $"
q 5 is the OLS estimator
%&'('$$$$$# = ! # "! "" +* ! ! and 𝜷
• An alternative expression for the F statistics
- We can also directly compute the F test statistics:
*"!K -&-!&/,"%#/ #$!# " / (!# *"!K -&-!& ) #
$= $ $# !" ! !
,*" ! ! & % ()*+ ' %&
"

26
 5.4 Testing Multiple Restriction
Applications: Testing Structural Changes
• The economic environment might change over time
- Econometric models may not be stable over time
- Changes due to government policy
q Tax regime
q The Lucas critique
- Changes due to taste and technology
q Risk aversion in different phases of business cycle
q Introduction of new technology
- Changes due to natural shock
q Oil shock
• How to model changes?
- Regime switch
- Time varying
- Structure break
• The last one can apply the Chow's test, which is a F test
27