Chapter 2
Chapter 2
CHAPTER TWO
OVERVIEW OF THE
CLASSICAL LINEAR REGRESSION MODEL
By [Link] M.
Regression
2
Some Notation
• Denote the dependent variable by y and the independent variable(s) by x1, x2,
... , xk where there are k independent variables.
• Note that there can be many x variables but we will limit ourselves to the
case where there is only one x variable to start with. In our set-up, there is
only one y variable.
3
Regression is different from Correlation
4
Simple Regression
• For simplicity, say k=1. This is the situation where y depends on only one x
variable.
5
Simple Regression: An Example
• Suppose that we have the following data on the excess returns on a fund
manager’s portfolio (“fund XXX”) together with the excess returns on a
market index:
Year, t Excess return Excess return on market index
= rXXX,t – rft = rmt - rft
1 17.8 13.7
2 39.0 23.2
3 12.8 6.9
4 24.2 16.8
5 17.2 12.3
• We have some intuition that the beta on this fund is positive, and we
therefore want to find whether there appears to be a relationship between
x and y given the data that we have. The first stage would be to form a
scatter plot of the two variables.
6
Graph (Scatter Diagram)
45
Excess return on fund XXX
40
35
30
25
20
15
10
5
0
0 5 10 15 20 25
Excess return on market portfolio
7
Finding a Line of Best Fit
8
Why do we include a Disturbance term?
9
Determining the Regression Coefficients
x
10
Ordinary Least Squares
• The most common method used to fit a line to the data is known as
OLS (ordinary least squares).
• What we actually do is take each distance and square it (i.e. take the
area of each of the squares in the diagram) and minimise the total sum
of the squares (hence least squares).
11
Actual and Fitted Value
yi
û i
ŷi
xi x
12
How OLS Works
• But what was ût ? It was the difference between the actual point and
the line, yt - ŷt .
• So minimising y ˆ
y
t t is equivalent to minimising
2
t
ˆ
u 2
13
Deriving the OLS Estimator
ˆ t ˆ ˆxt , so let
• But y L ( yt yˆ t ) 2 ( yt ˆ ˆxt ) 2
t i
L
2 xt ( yt ˆ ˆxt ) 0 (2)
ˆ t
• But y t Ty and xt Tx .
14
Deriving the OLS Estimator (cont’d)
t t t
x y y x ˆ
x t
x ˆ
t 0
x
2
t t
x y T y x ˆ
T x 2
ˆ
t 0
x
2
15
Deriving the OLS Estimator (cont’d)
• Rearranging for $ ,
• So overall we have
ˆ
xt yt Tx y
andˆ y ˆx
xt2 Tx 2
16
What do We Use $ and $ For?
yˆ t 1.74 1.64 x t
• Question: If an analyst tells you that she expects the market to yield a return
20% higher than the risk-free rate next year, what would you expect the return
on fund XXX to be?
• Solution: We can say that the expected value of y = “-1.74 + 1.64 * value of x”,
so plug x = 20 into the equation to get the expected value for y:
yˆ i 1.74 1.64 20 31.06
17
Accuracy of Intercept Estimate
0 x
18
The Population and the Sample
19
The DGP and the PRF
• Linear in the parameters means that the parameters are not multiplied
together, divided, squared or cubed etc.
Yt e X t e ut ln Yt ln X t ut
23
The Assumptions Underlying the
Classical Linear Regression Model (CLRM)
• The model which we have used is known as the classical linear regression model.
• We observe data for xt, but since yt also depends on ut, we must be specific about
how the ut are generated.
• We usually make the following set of assumptions about the ut’s (the
unobservable error terms):
• Technical Notation Interpretation
1. E(ut) = 0 The errors have zero mean
2. Var (ut) = 2 The variance of the errors is constant and finite
over all values of xt
3. Cov (ui,uj)=0 The errors are statistically independent of
one another
4. Cov (ut,xt)=0 No relationship between the error and
corresponding x variate
24
The Assumptions Underlying the
CLRM Again
• Additional Assumption
5. ut is normally distributed
25
Properties of the OLS Estimator
26
Consistency/Unbiasedness/Efficiency
• Consistent
The least squares estimators $ and $ are consistent. That is, the estimates will
converge to their true values as the sample size increases to infinity. Need the
assumptions E(xtut)=0 and Var(ut)=2 < to prove this. Consistency implies that
lim Pr ˆ 0 0
T
• Unbiased
The least squares estimates of $ and $ are unbiased. That is E($)= and E($ )=
Thus on average the estimated value will be equal to the true values. To prove
this also requires the assumption that E(ut)=0. Unbiasedness is a stronger
condition than consistency.
• Efficiency
An estimator $ of parameter is said to be efficient if it is unbiased and no other
unbiased estimator has a smaller variance. If the estimator is efficient, we are
minimising the probability that it is a long way off from the true value of .
27
Precision and Standard Errors
• Any set of regression estimates of $ and $ are specific to the sample used in
their estimation.
• Recall that the estimators of and from the sample parameters ($ and $) are
ˆ t 2 t
given by x y Tx y
andˆ y ˆx
xt Tx 2
T
uˆ t
29
Estimating the Variance of the Disturbance Term
(cont’d)
T 2
where uˆ
2
t is the residual sum of squares and T is the sample size.
2. The sum of the squares of x about their mean appears in both formulae.
The larger the sum of squares, the smaller the coefficient variances.
30
Some Comments on the Standard Error Estimators
y
y
y
y
x x
0 x 0 x
31
Some Comments on the Standard Error Estimators
(cont’d)
3. The larger the sample size, T, the smaller will be the coefficient
variances. T appears explicitly in SE($ ) and implicitly in SE( $ ).
The reason is that xt measures how far the points are away from the
2
y-axis.
32
Example: How to Calculate the Parameters and
Standard Errors
• Assume we have the following data calculated from a regression of y on a
single variable x and a constant over 22 observations.
• Data:
xt yt 830102, T 22, x 416.5, y 86.65,
x 2
t 3919654, RSS 130.6
• SE(regression), s uˆ t2
130.6
2.55
T 2 20
3919654
SE ( ) 2.55 * 3.35
22 3919654 22 416.52
1
SE ( ) 2.55 * 0.0079
3919654 22 416.5 2
• We now write the results as
yˆ t 59.12 0.35 xt
(3.35) (0.0079)
34
An Introduction to Statistical Inference
35
Hypothesis Testing: Some Concepts
• We can use the information in the sample to make inferences about the
population.
• We will always have two hypotheses that go together, the null hypothesis
(denoted H0) and the alternative hypothesis (denoted H1).
• The null hypothesis is the statement or the statistical hypothesis that is actually
being tested. The alternative hypothesis represents the remaining outcomes of
interest.
• For example, suppose given the regression results above, we are interested in
the hypothesis that the true value of is in fact 0.5. We would use the notation
H0 : = 0.5
H1 : 0.5
This would be known as a two sided test.
36
One-Sided Hypothesis Tests
• There are two ways to conduct a hypothesis test: via the test of
significance approach or via the confidence interval approach.
37
The Probability Distribution of the
Least Squares Estimators
• Since the least squares estimators are linear combinations of the random
variables
i.e. $ wt yt
• What if the errors are not normally distributed? Will the parameter estimates
still be normally distributed?
• Yes, if the other assumptions of the CLRM hold, and the sample size is
sufficiently large.
38
The Probability Distribution of the
Least Squares Estimators (cont’d)
ˆ ˆ
~ N 0,1 and ~ N 0,1
var var
ˆ ˆ
~ tT 2 and ~ tT 2
SE (ˆ ) ˆ
SE ( )
39
Testing Hypotheses:
The Test of Significance Approach
40
The Test of Significance Approach (cont’d)
42
The Rejection Region for a 1-Sided Test (Upper Tail)
f(x)
95% non-rejection
region 5% rejection region
43
The Rejection Region for a 1-Sided Test (Lower Tail)
f(x)
44
The Test of Significance Approach: Drawing
Conclusions
7. Finally perform the test. If the test statistic lies in the rejection
region then reject the null hypothesis (H0), else do not reject H0.
45
A Note on the t and the Normal Distribution
• You should all be familiar with the normal distribution and its
characteristic “bell” shape.
• We can scale a normal variate to have zero mean and unit variance by
subtracting its mean and dividing by its standard deviation.
46
What Does the t-Distribution Look Like?
normal distribution
t-distribution
47
Comparing the t and the Normal Distribution
• The reason for using the t-distribution rather than the standard normal is that
we had to estimate 2, the variance of the disturbances.
48
The Confidence Interval Approach
to Hypothesis Testing
49
How to Carry out a Hypothesis Test
Using Confidence Intervals
3. Use the t-tables to find the appropriate critical value, which will again have T-2
degrees of freedom.
5. Perform the test: If the hypothesised value of (*) lies outside the confidence
interval, then reject the null hypothesis that = *, otherwise do not reject the null.
50
Confidence Intervals Versus Tests of Significance
• The first step is to obtain the critical value. We want tcrit = t20;5%
52
Determining the Rejection Region
f(x)
-2.086 +2.086
53
Performing the Test
• Note that we can test these with the confidence interval approach.
For interest (!), test
H0 : = 0
vs. H1 : 0
H0 : = 2
vs. H1 : 2
55
Changing the Size of the Test
• For example, say we wanted to use a 10% size of test. Using the test of
significance approach, $ *
test stat
SE ( $ )
05091
. 1
1917
.
0.2561
as above. The only thing that changes is the critical t-value.
56
Changing the Size of the Test:
The New Rejection Regions
f(x)
-1.725 +1.725
57
Changing the Size of the Test:
The Conclusion
• t20;10% = 1.725. So now, as the test statistic lies in the rejection region,
we would reject H0.
58
Some More Terminology
• If we reject the null hypothesis at the 5% level, we say that the result
of the test is statistically significant.
59
The Errors That We Can Make
Using Hypothesis Tests
• The probability of a type I error is just , the significance level or size of test we
chose. To see this, recall what we said significance at the 5% level meant: it is only
5% likely that a result as or more extreme as this could have occurred purely by
chance.
• Note that there is no chance for a free lunch here! What happens if we reduce the size
of the test (e.g. from a 5% test to a 1% test)? We reduce the chances of making a type
I error ... but we also reduce the probability that we will reject the null hypothesis at
all, so we increase the probability of a type II error: less likely
to falsely reject
Reduce size more strict reject null
of test criterion for hypothesis more likely to
rejection less often incorrectly not
reject
• So there is always a trade off between type I and type II errors when choosing a
significance level. The only way we can reduce the chances of both is to increase the
sample size.
61
A Special Type of Hypothesis Test: The t-ratio
$i
Since i* = 0, test stat
SE ( $i )
63
What Does the t-ratio tell us?
• If we reject H0, we say that the result is significant. If the coefficient is not
“significant” (e.g. the intercept coefficient in the last regression above), then
it means that the variable is not helping to explain variations in y. Variables
that are not significant are usually removed from the regression model.
•y In practice there are good statistical reasons for always having a constant
t
even if it is not significant. Look at what happens if no intercept is included:
xt
64
An Example of the Use of a Simple t-test to Test a
Theory in Finance
• The Data: Annual Returns on the portfolios of 115 mutual funds from
1945-1964.
65
Frequency Distribution of t-ratios of Mutual Fund
Alphas (gross of transactions costs)
66
Frequency Distribution of t-ratios of Mutual Fund
Alphas (net of transactions costs)
67
Can UK Unit Trust Managers “Beat the Market”?
68
Can UK Unit Trust Managers “Beat the Market”?
: Results
69
The Overreaction Hypothesis and
the UK Stock Market
• Motivation
Two studies by DeBondt and Thaler (1985, 1987) showed that stocks which
experience a poor performance over a 3 to 5 year period tend to outperform
stocks which had previously performed relatively well.
• How Can This be Explained?
2 suggestions
1. A manifestation of the size effect
DeBondt & Thaler did not believe this a sufficient explanation, but Zarowin
(1990) found that allowing for firm size did reduce the subsequent return on
the losers.
2. Reversals reflect changes in equilibrium required returns
Ball & Kothari (1989) find the CAPM beta of losers to be considerably
higher than that of winners.
70
The Overreaction Hypothesis and
the UK Stock Market (cont’d)
71
Methodology
• Calculate the monthly excess return of the stock over the market over a 12,
24 or 36 month period for each stock i:
• Calculate the average monthly return for the stock i over the first 12, 24, or
36 month period:
1 n
Ri U it
n t 1
72
Portfolio Formation
• Then rank the stocks from highest average return to lowest and from 5
portfolios:
73
Portfolio Formation and
Portfolio Tracking Periods
• If n = 1year:
Estimate for year 1
Monitor portfolios for year 2
Estimate for year 3
...
Monitor portfolios for year 36
L
• Define the difference between these as RDt = Rp - RpW .
• Solution: Allow for risk differences by regressing against the market risk
premium:
where
Rmt is the return on the FTA All-share
Rft is the return on a UK government 3 month t-bill.
76
Is there an Overreaction Effect in the
UK Stock Market? Results
• Is there evidence that losers out-perform winners more at one time of the
year than another?
• To test this, calculate the difference between the winner & loser portfolios
as previously, RDt , and regress this on 12 month-of-the-year dummies:
12
RDt i Mi t
i 1
• Significant out-performance of losers over winners in,
– June (for the 24-month horizon), and
– January, April and October (for the 36-month horizon)
– winners appear to stay significantly as winners in
• March (for the 12-month horizon).
78
Conclusions
Comments
• Small samples
• No diagnostic checks of model adequacy
79
The Exact Significance Level or p-value
• If the test statistic is large in absolute value, the p-value will be small, and
vice versa. The p-value gives the plausibility of the null hypothesis.