MST3124 Applied Time

Series Econometrics

Topic 1: Stationary Models

Dr. Jason Ng

1
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

2
Introduction

The univariate time series models that we study in Topic 1 are

primarily used for short term forecasting.
The models are based on the assumption that the observed data is
being generated by an underlying statistical model or data
generating process (dgp).
The forecasting strategy, which was first proposed by Box and
Jenkins in the 1970s, can be summarized in the following steps:

3
Introduction

1 Use the data to identify the underlying statistical model that is

generating the data, or at least a model which is a good
approximation to the dgp.
2 Estimate the parameters of the chosen model.
3 Diagnostic checking and model revision, if necessary.
4 Use the estimated model to forecast future values of the time
series.

4
Introduction

In order to implement this strategy it is necessary to be familiar

with certain basic properties of a time series and the sample
analogues of these properties, as these properties play a crucial
role at S1.
We begin our discussion with a formal definition of a univariate
time series.

5
Introduction

Definition (Univariate Time Series)

1.1: A univariate time series is an ordered sequence of random
variables indexed by time. Symbolically we denote a time series by
{yt : t = ........... − 2, −1, 0, 1, 2, ......}

A realization of the time series occurs when {yt } takes on a

particular value for each possible value of t.
Since each yt is a random variable, there is an infinite number of
possible realizations of the time series.

6
Introduction

When we observe a particular realization of a time series consisting

of
{yt : t = 1, 2, ......T},
we are in fact observing a sequence of outcomes of these random
variables.
The set of all possible realizations is called the ensemble.
In order to conduct statistical inference on a time series
(i.e. estimate it’s parameters and test hypotheses about it’s
properties) it is necessary to make some simplifying assumptions
regarding it’s statistical behavior.

7
Introduction

For example, even if we restricted our attention to the fragment of

the time series corresponding to
{yt : t = 1, ......T},
in the most general case, this fragment would contain:
T means,(µ1 , µ2 , ...., µT ),
T variances, (δ12 , δ22 , ...., δT2 ),
and T(T − 1)/2 covariances.
It is impossible to accurately estimate this number of parameters
from T observations.
Need to impose some restriction on the time series which reduces
the number of parameters to be estimated.

8
Introduction

A commonly imposed restriction: the time series is covariance

stationary (also called weakly or second order stationary).
This restriction dramatically simplifies the structure of the time
series and greatly reduces the number of parameters to be
estimated.
The definition of a covariance stationary process is given below.

9
Introduction

Definition (Covariance Stationary)

1.2 The time series
{yt : t = ..... − 2, −1, 0, 1, 2, ........}
is covariance stationary if
a) E(yt ) = µ < ∞ for all t.
b) Var(yt ) = E[(yt − µ)2 ] = γ0 < ∞ for all t.
c) Cov(yt , yt−j ) = E[(yt − µ)(yt−j − µ)] = γj < ∞ for all t and j
More succinctly, the time series
{yt : t = ..... − 2, −1, 0, 1, 2, ........}
is covariance stationary if it’s first and second moments are both finite
and time invariant.

10
Introduction

Each random variable yt in the sequence now has:

the same mean;
the same variance and;
the covariances between the random variables are assumed to
depend only on the time interval separating them j and not on time
t itself. For example,

cov(y1 , y10 ) = cov(y21 , y30 ) = cov(y101 , y110 ).

11
Introduction

4000

3900
Dow Jones Index

3800

3700

3600

0 50 100 150 200 250 300

Day

12
Introduction

50
Change in Dow Jones Index

−50

−100
0 50 100 150 200 250 300
Day

13
Introduction

There are other concepts of stationarity used in the time series

literature, but they are not of concern in this course.
Therefore, we shall use the term stationary as an abbreviation for
the term covariance stationary.
In Topic 2, we will consider in some detail the conditions required
for a time series to be stationary.

14
Introduction

Although it is an abuse of mathematical notation, in the interest of

notational economy, we shall henceforth denote both the time
series and a particular realization of the time series by yt . It should
be clear from the context which concept is intended.

15
Introduction

When the time series is stationary, it makes sense to speak of ‘the

mean,’ ‘the variance,’ and ‘the jth autocovariance,’ since these
values do not depend on t.
µ, γ0 , and γj are unknown population parameters. Given a sample
of T observations,
y1 , y2 , . . . , yT−1 , yT
the sample analogues of these population parameters are defined
as follows:

16
Introduction

Definition (Sample mean)

_
1.3: The sample mean y is defined as
T
1X
y= yt
T t=1

Definition (Sample variance)

ˆ
1.4: The sample variance γ 0 is defined as
T
1X
γ̂0 = (yt − y)2
T t=1

Definition (Sample autocovariance)

ˆ
1.5: The sample jth autocovariance γ j is defined as
T
1X
γ̂j = (yt − y)(yt−j − y)
T t=j+1 17
Introduction

For example, setting j = 1 in Definition 1.5, we obtain the sample

covariance at lag 1:

T
1X
γ̂1 = (yt − y)(yt−1 − y)
T t=2
Setting j = 2 in Definition 1.5, we obtain the sample covariance at
lag 2:

T
1X
γ̂2 = (yt − y)(yt−2 − y)
T t=3

18
Introduction

The jth autocovariance, γj , measures the strength of the linear

relationship between yt and yt−j .
It is more common to use the autocorrelation coefficient to
measure the strength of the linear relationship between yt and yt−j
as it is unit-free.

19
Introduction

Definition (Population autocorrelation)

1.6: If yt is a stationary time series, the autocorrelation between yt and
yt−j is defined as

Cov(yt , yt−j ) Cov(yt , yt−j ) γj

ρj = √ q = √ √ =
Var(yt ) Var(yt−j ) γ0 γ0 γ0

20
Introduction

For a stationary time series, ρj possesses the following properties:

γ0
1 ρ0 = γ0 =1
2 −1 ≤ ρj ≤ 1 for all j.
3 ρj = ρ−j .
ρj has two major advantages over γj as a measure of linear
association:
1 Unlike γj , ρj is unit-free.
2 Unlike γj , ρj is easy to interpret.

21
Introduction

The autocorrelation function (ACF) or theoretical correlogram is

obtained by plotting ρj against non-negative values of j.
A concept closely related to the ACF is the partial autocorrelation
function (PACF).
The partial autocorrelation at lag j, denoted by ϕjj , measures the
correlation between yt and yt−j , when the intermediate values
yt−1 , . . . , yt−j+1 are considered fixed.

22
Introduction

The ϕ′jj s are given by the appropriate coefficients in the following

system of equations:

ϕ11 : yt = ϕ11 yt−1 + εt (E1)

ϕ22 : yt = ϕ21 yt−1 + ϕ22 yt−2 + εt (E2)
ϕ33 : yt = ϕ31 yt−1 + ϕ32 yt−2 + ϕ33 yt−3 + εt (E3)
...
ϕjj : yt = ϕj1 yt−1 + ϕj2 yt−2 + . . . + ϕjj yt−j + εt (Ej)
For example, the partial autocorrelation coefficient at lag 1 is ϕ11 ,
and the partial autocorrelation at lag 2 is ϕ22 .
The PACF, or partial correlogram, is obtained by plotting ϕjj against
j.
23
Introduction

As ρj and ϕjj are unobserved parameters, in practice, we replace

these unknown parameters with estimators based on our observed
sample.

Definition (Sample autocorrelation)

1.7: The sample correlation coefficient at lag j, which we denote by ρ̂j ,
is given by
PT
γ̂j t=j+1 (yt − y)(yt−j − y)
ρ̂j = = PT 2
γ̂0 t=1 (yt − y)

24
Introduction

For example,
PT
t=2 (yt − y)(yt−1 − y)
ρ̂1 = PT 2
t=1 (yt − y)
PT
t=3 (yt − y)(yt−2 − y)
ρ̂2 = PT 2
t=1 (yt − y)

25
Introduction

When we plot ρ̂j against j, we obtain the sample correlogram or

sample ACF.
The sample autocorrelations can be used as estimates of the
corresponding theoretical autocorrelations for the stochastic
process assumed to be generating the data.
The sample ACF is subject to sampling variability: different
samples will lead to different sample ACFs.
Consequently, while the sample ACF will tend to mirror the
properties of the theoretical ACF, it will not reproduce them exactly.

26
Introduction

Example
Series: h02

0.8

0.6

0.4
ACF

0.2

0.0

6 12 18 24
Lag

27
Introduction

The sample partial correlogram or sample PACF is obtained by

plotting ϕ̂jj against non-negative values of j, where

ϕ̂11 is the estimated coefficient attached to yt−1

when (E1) is estimated by OLS.
ϕ̂22 is the estimated coefficient attached to yt−2
when (E2) is estimated by OLS.
...
ϕ̂jj is the estimated coefficient attached to yt−j
when (Ej) is estimated by OLS.

28
Introduction

Example
Series: h02
0.8

0.4
PACF

0.0

−0.4

6 12 18 24
Lag

29
Introduction

Some important population parameters and their sample

analogues for the stochastic process yt are summarized below:

Population parameter Sample statistic

R∞
y = T −1 Tt=1 yt
P
E(yt ) = µ = −∞ yt f(yt )dy

PT
γ0 = E[(yt − µ)2 ] γ̂0 = T −1 t=1 (yt − y)2

PT
γj = E[(yt − µ)(yt−j − µ)] γ̂j = T −1 t=j+1 (yt − y)(yt−j − y)
PT
Cov(yt ,yt−j ) γ̂j (yt −y)(yt−j −y)
ρj = √ √ ρ̂j = γ̂0 = t=j+1
PT
Var(yt ) Var(yt−j ) t=1
(yt −y)2

30
Introduction

Both the sample ACF and the sample PACF are commonly used as
tools to aid in identifying an appropriate statistical model for the
time series yt .
Rationale: Stochastic processes can be differentiated based on the
properties of their ACF and PACF.
Therefore, knowledge of the ACF and PACF properties may help us
to identify the nature of the stochastic process that is generating
the observed values of yt .
However, to exploit this fact, we need to be familiar with the
properties of the ACF and PACF of different stochastic processes.
In the next few subsections, we describe the ACF and PACF of
several stochastic processes.

31
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

32
I.I.D. Process

Definition (I.I.D. Process)

1.8: A time series, et , is independently and identically distributed,
which we denote by
et ∼ i.i.d(0, σ 2 ),
if
E(et ) = 0 for all t.
Var(et ) = σ 2 for all t.
and et and et−j are independent random variables for all j and all t.

33
I.I.D. Process

That is,

et ∼ i.i.d(0, σ 2 ),
if et and et−j are independently distributed random variables for all j ̸= 0.

34
White Noise

Definition (White Noise)

1.9: A time series, et , is white noise, which we denote by
et ∼ WN(0, σ 2 ),
if
E(et ) = 0 for all t.
Var(et ) = σ 2 for all t.
Cov(et , et−j ) = 0 for all j ̸= 0.

35
I.I.D. vs White Noise

Thus, a time series is white noise if it has a zero constant mean, a

constant variance, and is serially uncorrelated. (Some authors do
not restrict the mean to be zero). Note the following points with
respect to white noise processes:
1 By definition, every white noise process is stationary, but not every
stationary series is white noise. Assuming that a series is white noise
is a much stronger assumption than assuming that it is merely
stationary.
2 If et has the above properties and is also normally distributed, then
et is referred to as Gaussian white noise.
3 If et is white noise, and therefore uncorrelated, observing the value
of the series in one period is of no assistance in forecasting its value
in any future period, using a linear forecasting rule.

36
I.I.D. vs White Noise

The assumption that et ∼ i.i.d(0, σ 2 ) is a much stronger

assumption than the assumption that et ∼ WN(0, σ 2 ).

37
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

38
Stationary AR(1) Processes

The time series yt is an AR(1) process if

yt = c + ϕ1 yt−1 + et , (1.1)
where
et ∼ WN(0, σ 2 ).
Notice that this model has three unknown parameters, c, ϕ1 , and
σ2.

39
Stationary AR(1) Processes

If yt is a stationary AR(1) process given by (1.1) then:

c
E(yt ) = µ = ∀t (1.2)
(1 − ϕ1 )
σ2
Var(yt ) = γ0 = ∀t (1.3)
(1 − ϕ21 )
σ2
Cov(yt , yt−j ) = γj = 2 ϕj1 , ∀ t and j ∈ N (1.4)
(1 − ϕ1 )
γj
ρj = = ϕj1 , ∀ j ∈ N (1.5)
γ0
where

N = {0, 1, 2, .......}.

40
Stationary AR(1) Processes

Note the following implications of the above formulas:

1 The mean, variance, and autocovariances of yt are expressed in
terms of the three unknown parameters of the model c, ϕ1 , and σ 2 .
2 It is evident from (1.2) that yt has a mean of zero if and only if c = 0
in (1.1). Thus, assuming that there is no intercept in (1.1) is
equivalent to assuming that the mean of yt = 0.
3 It is evident from (1.5) that yt and yt−j are positively correlated for
all j if ϕ1 > 0.

41
Stationary AR(1) Processes

4 It is evident from (1.5) that the “memory of yt ” depends on |ϕ1 |. yt

will have a “short memory” in the sense that
ρj → 0
quickly as j increases if |ϕ1 | is small. Conversely, yt will have a “long
memory” in the sense that
ρj → 0
slowly as j increases if |ϕ1 | is close to 1. For example:
ϕ1 = 0.9 ϕ1 = 0.2
ρ0 = (0.9)0 =1 ρ0 = (0.2)0 =1
ρ1 = (0.9)1 = 0.90 ρ1 = (0.2)1 = 0.20
ρ2 = (0.9)2 = 0.81 ρ2 = (0.2)2 = 0.04
ρ3 = (0.9)3 = 0.73 ρ3 = (0.2)3 = 0.01
ρ4 = (0.9)4 = 0.66 ρ4 = (0.2)4 = 0.00
42
Stationary AR(1) Processes

5 In general, for a stationary AR(1) process, ρj will decline

exponentially since, from (1.5),
ρj = ϕj1 ∀j

For a stationary AR(1) process,

yt = c + ϕ1 yt−1 + et
the PACF cuts off after one lag, since
ϕ11 = ϕ1
and
ϕjj = 0 ∀j > 1

43
Stationary AR(1) Processes

Note carefully that for a stationary AR(1) process it is the partial

autocorrelation coefficients that go to zero after one lag, not the
autocorrelation coefficients. From (1.5),
ρj = ϕj1
Therefore,
{ρ1 , ρ2 , ρ3 , . . .} = {ϕ1 , ϕ21 , ϕ31 , . . .}
implying that if yt is an AR(1) process, it is correlated with
yt−1 , yt−2 , yt−3 , . . ., not just with yt−1 . How quickly the
autocorrelations die off depends on the absolute value of ϕ1 .

44
Stationary AR(1) Processes

An AR(1) process would be a good model for a time series which

has an ACF which declines exponentially, and a PACF which goes
to zero after one lag.
Since the ACF or the PACF are unobservable, we have to choose a
model based on the behavior of the SACF and the SPACF.

45
Stationary AR(1) Processes

Example:
yt = 0.5 + 0.7yt−1 + et

5.0
2.5
0.0
−2.5
0 200 400 600 800 1000

0.50 0.50

PACF
ACF

0.25 0.25

0.00 0.00

0 5 10 15 20 25 30 0 5 10 15 20 25 30
Lag Lag

46
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

47
Stationary AR(p) Processes

The time series yt is an AR(p) process if

yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et , (1.6)
where
et ∼ WN(0, σ 2 ).
Notice that this model has p + 2 unknown parameters:

c, ϕ1 , ϕ2 , . . . , ϕp , σ 2

48
Stationary AR(p) Processes

The AR(p) model is a generalization of the AR(1) which permits

non-zero partial autocorrelation coefficients beyond the first lag.
If yt is a stationary AR(p) process, then

c
E(yt ) = µ = for all t. (1.7)
(1 − ϕ1 − ϕ2 − . . . − ϕp )
Equation (1.7) implies that

E(yt ) = 0, iff c = 0.
Therefore, as in the AR(1) case, omitting the intercept from the
model is appropriate only if the time series has a mean of zero.

49
Stationary AR(p) Processes

The next few slides leading to the Yule-Walker equations are

optional reading.
For an AR(1) model, equation (1.5) expresses the jth
autocorrelation coefficient in terms of the parameter ϕ1 . For a
stationary AR(p) process, expressing ρj in terms of the underlying
model parameters
c, ϕ1 ϕ2 , . . . , ϕp , σ 2
is slightly more difficult.
Thus, the Yule-Walker equations can be used to estimate the
population correlation coefficients.

50
Stationary AR(p) Processes

We write
yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et (1.6)
in mean deviation form.
From (1.7) we have
c = µ(1 − ϕ1 − ϕ2 − . . . − ϕp ). (1.8)
Substituting (1.8) into (1.6) we obtain
yt = µ(1 − ϕ1 − ϕ2 − . . . − ϕp ) + ϕ1 yt−1 + ϕ2 yt−2 +
. . . + ϕp yt−p + et
yt = µ − µϕ1 − µϕ2 − . . . − µϕp + ϕ1 yt−1 + ϕ2 yt−2 +
. . . + ϕp yt−p + et
yt − µ = ϕ1 (yt−1 − µ) + ϕ2 (yt−2 − µ) + . . . + ϕp (yt−p − µ) + et .
(1.9)
51
Stationary AR(p) Processes

Equation (1.9) expresses the time series in mean deviation form.

Notice from (1.9) that when it is expressed in mean deviation form,
the time series necessarily has a mean of zero.
In equation (1.5) we expressed the jth autocorrelation coefficient
of an AR(1) process in terms of the parameter ϕ1 . For a stationary
AR(p) process, expressing ρj in terms of the underlying model
parameters
c, ϕ1 , ϕ2 , . . . , ϕp , σ 2
is slightly more difficult.

52
Stationary AR(p) Processes

We next show that if

yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et , (1.6)

the autocorrelation coefficients of yt also follow an AR(p). That is,

ρj = ϕ1 ρj−1 + ϕ2 ρj−2 + . . . + ϕp ρj−p ∀j ̸= 0.

53
Stationary AR(p) Processes

When we multiply both sides of

yt − µ = ϕ1 (yt−1 − µ) + ϕ2 (yt−2 − µ) + . . . + ϕp (yt−p − µ) + et (1.9)

by (yt−j − µ), we obtain

(yt − µ)(yt−j − µ) = ϕ1 (yt−1 − µ)(yt−j − µ) + ϕ2 (yt−2 − µ)(yt−j − µ)+

. . . + ϕp (yt−p − µ)(yt−j − µ) + et (yt−j − µ). (1.10)

54
Stationary AR(p) Processes

Taking expectations on both sides of (1.10), we obtain

E[(yt − µ)(yt−j − µ)] = ϕ1 E[(yt−1 − µ)(yt−j − µ)]+

ϕ2 E[(yt−2 − µ)(yt−j − µ)] + . . . +
ϕp E[(yt−p − µ)(yt−j − µ)] + E[et (yt−j − µ)].
(1.10a)
It can be shown that if yt is a stationary time series, then

E[et (yt−j − µ)] = E(et yt−j − et µ)

= E(et yt−j ) − µE(et )
=0 ∀j ̸= 0. (1.11)

55
Stationary AR(p) Processes

Using (1.11) and the fact that, by definition,

E[(yt − µ)(yt−j − µ)] = Cov(yt , yt−j ) = γj ,
equation (1.10a) may be written as
γj = ϕ1 γj−1 + ϕ2 γj−2 + . . . + ϕp γj−p ∀j ̸= 0. (1.12)
Finally, dividing on both sides of (1.12) by γ0 , we obtain
γj γj−1 γj−2 γj−p
= ϕ1 + ϕ2 + . . . + ϕp ,
γ0 γ0 γ0 γ0
or

ρj = ϕ1 ρj−1 + ϕ2 ρj−2 + . . . + ϕp ρj−p ∀j ̸= 0, (1.13)

where
γj
ρj = .
γ0

56
Stationary AR(p) Processes

Equation (1.13) demonstrates that if yt is a stationary AR(p)

process, then ρj is also an AR(p) process with the same
autoregressive coefficients for yt . That is, when
yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et ,
it necessarily follows that

ρj = ϕ1 ρj−1 + ϕ2 ρj−2 + . . . + ϕp ρj−p ∀j ̸= 0. (1.13)

57
Stationary AR(p) Processes

Sequentially setting j = 1, 2, 3, .... in equation (1.13) we obtain the

following system of equations:

ρ1 = ϕ1 ρ0 + ϕ2 ρ−1 + . . . + ϕp ρ1−p
ρ2 = ϕ1 ρ1 + ϕ2 ρ0 + . . . + ϕp ρ2−p
... (1.14)
ρp = ϕ1 ρp−1 + ϕ2 ρp−2 + . . . + ϕp ρ0
Since, for a stationary process,

ρ0 = 1
ρ−j = ρj

58
Stationary AR(p) Processes

The set of equations given by (1.14) may be rewritten as:

ρ1 = ϕ1 + ϕ2 ρ1 + . . . + ϕp ρp−1
ρ2 = ϕ1 ρ1 + ϕ2 + . . . + ϕp ρp−2
... (1.15)
ρp = ϕ1 ρp−1 + ϕ2 ρp−2 + . . . + ϕp
The set of equations given by (1.15) is known as the Yule-Walker
equations.
The Yule-Walker equations can be used to estimate the population
correlation coefficients by executing the following steps:

59
Stationary AR(p) Processes

1 Estimate
yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et (1.6)
and obtain

ϕb1 , ϕb2 , . . . , ϕbp

2 Substitute the estimated coefficients into the Yule-Walker
equations (1.15):

ρ1 = ϕb1 + ϕb2 ρ1 + . . . + ϕbp ρp−1

ρ2 = ϕb1 ρ1 + ϕb2 + . . . + ϕbp ρp−2
...
ρp = ϕb1 ρp−1 + ϕb2 ρp−2 + . . . + ϕbp

60
Stationary AR(p) Processes

The Yule-Walker equations transform into a system of p equations with

p unknowns, which we can solve to obtain:

ρ1 , ρ2 , ρ3 , . . . , ρp
in terms of

ϕb1 , ϕb2 , . . . , ϕbp

61
Stationary AR(p) Processes

Once we have obtained

ρb1 , ρb2 , . . . , ρbp

we can use

ρj = ϕ1 ρj−1 + ϕ2 ρj−2 + . . . + ϕp ρj−p ∀j ̸= 0 (1.13)

to obtain ρbj , for j > p.

62
Stationary AR(p) Processes

For example,

ρbp+1 = ϕb1 ρbp + ϕb2 ρbp−1 + . . . + ϕbp ρb1

Conversely, if we have estimates of
ρ1 , ρ2 , ρ3 , . . . , ρp ,
say
ρb1 , ρb2 , . . . , ρbp ,
we could solve the Yule-Walker equations to derive estimates of
ϕ1 , ϕ2 , . . . , ϕp .

63
Stationary AR(p) Processes

For a stationary AR(p) process

yt = c + ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et (1.6)
it can be shown that:
The ACF declines exponentially
The PACF is given by

ϕjj ∀ j≤p
ϕjj = 0 ∀ j>p
Therefore an AR(p) process may provide a suitable model for the
data generating process (DGP) of yt if the ACF shows exponential
decay, and the PACF cuts off (i.e., ceases) after p lags.

64
Stationary AR(p) Processes

Example: AR(2)

yt = 1.3yt−1 − 0.7yt−2 + et

−5

0 200 400 600 800 1000

0.8 0.8

0.4 0.4
PACF
ACF

0.0 0.0

−0.4 −0.4

0 5 10 15 20 25 30 0 5 10 15 20 25 30
Lag Lag

65
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

66
MA(q) Processes

A time series yt follows an MA(q) process if

yt = c + et + θ1 et−1 + θ2 et−2 + . . . + θq et−q (1.16)
where et ∼ WN(0, σ 2 ).
If yt is generated by (1.16), then

E(yt ) = c.
Var(yt ) = σ 2 (1 + θ12 + θ22 + . . . + θq2 ).
γj = σ 2 (θj + θ1 θj+1 + θ2 θj+2 + . . . + θq−j θq ) for j ≤ q.
γj = 0 ∀ j > q.
ρj = 0 ∀ j > q. 1.17

67
MA(q) Processes

Since t does not appear explicitly in (1.17), yt is stationary.

Therefore, every MA(q) process is stationary.
It is evident from (1.17) that the ACF for an MA(q) process cuts off
after q lags. It can also be shown that the PACF for an MA(q)
process declines exponentially.
Assuming that c = 0 in (1.16) is equivalent to assuming that the
process has a mean of zero.
An MA(q) process may be a good model for the Data Generating
Process (DGP) of yt if the ACF appears to cut off completely after q
lags, and the PACF declines exponentially.

68
MA(q) Processes

Example: MA(2)

yt = 0.5et−1 − 0.5et−2 + et

4
2
0
−2
−4
0 200 400 600 800 1000

0.0 0.0
PACF
ACF

−0.2 −0.2

−0.4 −0.4
0 5 10 15 20 25 30 0 5 10 15 20 25 30
Lag Lag

69
MA(q) Processes

Notice the difference between the behavior of an AR(p) and an

MA(q) process:

AR(p) MA(q)
ACF Declines exponentially Cuts off after q lags
PACF Cuts off after p lags Declines exponentially

70
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

71
Stationary ARMA(p,q) Processes

AR(p) and MA(q) process are both special cases of a more general
process known as an ARMA(p,q) process.
The time series yt follows an ARMA(p,q) process if
yt = c+ϕ1 yt−1 +ϕ2 yt−2 +. . .+ϕp yt−p +et +θ1 et−1 +θ2 et−2 +. . .+θq et−q ,
(1.18)
where et ∼ WN(0, σ ). 2

If yt is an ARMA(p,q) process, rather than an AR(p) or an MA(q)

process, it is generally difficult to identify the underlying ARMA
model because of the complicated structure of both the ACF and
PACF.

72
Stationary ARMA(p,q) Processes

Example: ARMA(1,1)

yt = 0.5yt−1 − 0.8et−1 + et

−2

0 200 400 600 800 1000

0.0 0.0

PACF
ACF

−0.1 −0.1

−0.2 −0.2

0 5 10 15 20 25 30 0 5 10 15 20 25 30
Lag Lag

73
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

74
Properties of Sample Mean and Sample Correlogram

When we observe a realization of a time series, we don’t observe

the underlying DGP, nor do we observe the mean, variance, ACF
and PACF of the process.
In practice, we often use the sample mean, sample variance and
sample ACF as estimators of the population mean, population
variance and population ACF.
In light of this practice, it is appropriate to consider the statistical
properties of these estimators. The following results have been
shown to hold for the case in which yt is a stationary process:

75
Properties of Sample Mean and Sample Correlogram

1 The sample mean is an unbiased estimator of the population

mean. That is,
E(y) = µ
2 The sample mean is a consistent estimator of the population mean.
That is,
Plim(y) = µ
3 The sample autocovariance and autocorrelation coefficients are
biased estimators of the population analogues. That is,
E(γ̂j ) ̸= γj
E(ρ̂j ) ̸= ρj

76
Properties of Sample Mean and Sample Correlogram

4 In general, the sample autocovariance and autocorrelation

coefficients are consistent estimators of the population analogues.
That is,
Plim(γ̂j ) = γj
Plim(ρ̂j ) = ρj
5
asy
ρ̂j ∼ N(ρj , T −1 )

77
Properties of Sample Mean and Sample Correlogram

6 The sample partial autocorrelation coefficients are biased but

consistent estimators of the corresponding population partial
correlation coefficients. That is,
E(ϕˆjj ) ̸= ϕjj
Plim(ϕˆjj ) = ϕjj

78
Properties of Sample Mean and Sample Correlogram

Properties 4 and 6 provide the justification for using the SACF and
SPACF for model identification: In a large sample, the SACF and
SPACF should provide a “good” approximation to the population
ACF and PACF respectively.
As we discuss next, a test of the statistical significance of the
sample correlation coefficients can be based on Property 5.

79
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

80
Testing for Autocorrelation in y

We often wish to test the null hypothesis that a time series is white
noise (or, more generally, for the presence of autocorrelation in a
time series). One test of the white noise hypothesis is based on
Property 5 above.
Note that since
asy
ρ̂j ∼ N(ρj , T −1 ),
it follows that
ρ̂j − ρj asy
√ ∼ N(0, 1).
T −1
Therefore, asymptotically,
ρ̂j − ρj
 
P −Zcrit < √ < Zcrit = 0.95,
T −1
where Zcrit has the property that for any standard normal variable,
Z,
P[Z > Zcrit ] = 0.025.
81
Testing for Autocorrelation in y

At the 5% significance level,

Zcrit ≈ 2
and

ρ̂j − ρj
 
P −2 < √ < 2 = 0.95,
T −1
or
h √ √ i
P −2 T −1 < ρ̂j − ρj < 2 T −1 = 0.95.
Under the null hypothesis that yt is white noise, ρj = 0, it follows
that,

2 2
 
P − √ < ρ̂j < √ = 0.95 ∀j ̸= 0.
T T
82
Testing for Autocorrelation in y

That is, if
ρj = 0,
then a 95% confidence interval for ρ̂j is given by
2 2
− √ < ρ̂j < √ .
T T
If ρ̂j lies outside this confidence interval, we reject the null
hypothesis that ρj = 0.
Another popular test of the white noise hypothesis is that
proposed by Ljung and Box (1978).

83
Testing for Autocorrelation in y

The null hypothesis for the test is that there is no autocorrelation in

the time series at any lag up to and including lag s. That is,

H0 : ρ1 = ρ2 = . . . = ρs = 0.

H1 : ρj ̸= 0 for at least one j = 1, 2, ...s.

The test statistic is based on the sample correlation coefficients and
is given by
s
X ρ̂2j
Qs = T(T + 2) .
j=1
(T − j)
For example, the test statistic for testing
H0 : ρ1 = ρ2 = 0,
is
2
X ρ̂2j
Q2 = T(T + 2) . 84
j=1
(T − j)
Testing for Autocorrelation in y

Under the null hypothesis

H0 : ρ1 = ρ2 = . . . = ρs = 0,

asy
Qs ∼ χ2 (s).
At the 5% significance level we reject the null hypothesis if
Qs > χ2 (s)crit ,
where χ2 (s)crit is the 95th percentile of a chi-square variable with s
degrees of freedom.
If we reject the null, we conclude that there is non-zero
autocorrelation at at least one lag ≤ s.
Many software packages routinely produce these Q statistics
together with the sample autocorrelation function of the time
series and a set of p-values. 85
Testing for Autocorrelation in y

Reject the null hypothesis that

ρ1 = ρ2 = . . . = ρs = 0
if the p-value associated with Qj is less than 0.05.
The Ljung-Box test is an asymptotic test and may have low power
in small samples.
When the Ljung-Box is applied to the residuals from an estimated
ARMA(p,q) model to determine if the underlying errors are
autocorrelated, the asymptotic distribution of the test statistic is
slightly different. Specifically,

asy
Qs ∼ χ2 (s − p − q).

86
Testing for Autocorrelation in y

R produces Ljung-Box Q statistics when we issue the appropriate

command: LjungBox(). Below we reproduce the output for lags 1 to
5 for a series rcan.

Lag AC PAC Q-Stat Prob

1 0.942 0.942 172.10 0.000
2 0.905 0.162 331.97 0.000
3 0.884 0.157 485.33 0.000
4 0.858 0.005 630.59 0.000
5 0.834 0.015 768.28 0.000

87
Testing for Autocorrelation in y

In the table above, the number 172.10 is the sample value of the
test statistic for testing

H0 : ρ1 = 0.
The number 331.97 is the sample value of the test statistic for
testing

H0 : ρ1 = ρ2 = 0.
On the basis of the output, we would conclude that rcan is
autocorrelated.

88
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

89
Estimating Stationary ARMA Processes

ARMA models are typically estimated by either the method of

(conditional) ordinary least squares estimation or the method of
maximum likelihood estimation.
Since not all students in the unit are familiar with maximum
likelihood estimation, below we focus on estimation of ARMA
models using the conditional least squares estimator.

90
Estimating Stationary ARMA Processes

Consider the AR(2) model

yt = ϕ1 yt−1 + ϕ2 yt−2 + et (1.19)
where

et ∼ WN(0, σ 2 ).
Without loss of generality, and to economize on notation, we have
assumed that there is no intercept in the process.

91
Estimating Stationary ARMA Processes

Given a sample of T observations

(y1 , y2 , . . . , yT )
we obtain the OLS estimates of the parameters in (1.19) by
minimizing the residual sum of squares
T
X T
X
e2t = (yt − ϕ1 yt−1 − ϕ2 yt−2 )2
t=3 t=3
by choice of ϕ1 and ϕ2 .

92
Estimating Stationary ARMA Processes

Notice that the lower limit of summation is 3 rather than 1. This is

because we cannot construct e21 and e22 since

e21 = (y1 − ϕ1 y0 − ϕ2 y−1 )2

(1.20)
e22 = (y2 − ϕ1 y1 − ϕ2 y0 )2
and y0 and y−1 are not observed since they are not part of our sample.
The first squared residual that we can construct is

e23 = (y3 − ϕ1 y2 − ϕ2 y1 )2 .

93
Estimating Stationary ARMA Processes

In effect, when we construct

T
X T
X
e2t = (yt − ϕ1 yt−1 − ϕ2 yt−2 )2
t=3 t=3
we treat y1 and y2 as fixed numbers (rather than random variables,
which is what they really are), and minimize
T
X T
X
e2t = (yt − ϕ1 yt−1 − ϕ2 yt−2 )2 (1.21)
t=3 t=3
conditional on y1 and y2 .
The estimator we obtain when we apply this procedure is referred
to as a conditional least squares estimator, since it is conditional
on y1 and y2 .

94
Estimating Stationary ARMA Processes

Minimizing (1.21) by choice of ϕ1 and ϕ2 is a standard calculus

problem. The first-order conditions for a solution are obtained by
setting the following partial derivatives equal to zero and solving
the resulting equations for ϕ1 and ϕ2 :

PT 2 T
∂ t=3 et
X
= −2yt−1 (yt − ϕ1 yt−1 − ϕ2 yt−2 ) = 0 (1.22)
∂ϕ1 t=3
PT 2 T
∂ t=3 et
X
= −2yt−2 (yt − ϕ1 yt−1 − ϕ2 yt−2 ) = 0 (1.23)
∂ϕ2 t=3

The values of ϕ1 and ϕ2 that satisfy (1.22) and (1.23) are the
conditional least squares estimates of ϕ1 and ϕ2 .

95
Estimating Stationary ARMA Processes

Equations (1.22) and (1.23) are a system of two equations in the

two unknowns ϕ1 and ϕ2 . Because these first-order conditions are
linear in the parameters, equations (1.22) and (1.23) are easy to
solve.
This procedure is easily extended to an arbitrary AR(p) process.
Since for an AR(p)
yp+1 = ϕ1 yp + ϕ2 yp−1 + . . . + ϕp y1 + ep+1 ,
the first squared residual we can construct is
e2p+1 = (yp+1 − ϕ1 yp − ϕ2 yp−1 − . . . − ϕp y1 )2 ,
we obtain the conditional least squares estimates of
ϕ1 , . . . , ϕ p

96
Estimating Stationary ARMA Processes

by conditioning on
(y1 , y2 , . . . , yp )
and minimizing

T
X T
X
e2t = (yt − ϕ1 yt−1 − . . . − ϕp yt−p )2
t=p+1 t=p+1
by choice of
ϕ1 , . . . , ϕ p .

97
Estimating Stationary ARMA Processes

The conditional least squares estimates of

ϕ1 , . . . , ϕ p
are obtained by solving the p first-order conditions:

PT 2 T
∂ t=p+1 et
X
= −2yt−1 (yt − ϕ1 yt−1 − . . . − ϕp yt−p ) = 0
∂ϕ1 t=p+1
..
.
PT 2 T
∂ t=p+1 et
X
= −2yt−p (yt − ϕ1 yt−1 − . . . − ϕp yt−p ) = 0
∂ϕp t=p+1

98
Estimating Stationary ARMA Processes

In the classical linear regression model, it is assumed that the

regressors are nonrandom. However, this assumption is clearly
inappropriate in the case of the AR(p) process
yt = ϕ1 yt−1 + ϕ2 yt−2 + . . . + ϕp yt−p + et
since, for example,

yt−1 = ϕ1 yt−2 + ϕ2 yt−3 + . . . + ϕp yt−p−1 + et−1 .

The assumption of strict exogeneity is violated since
corr(yt , et−1 ) ̸= 0.

99
Estimating Stationary ARMA Processes

As a consequence of the fact that the regressors are random

variables, the OLS estimator of the parameters of an AR(p) process
does not have the usual optimal properties in finite samples. The
following results can be shown to hold when an AR(p) process is
estimated by OLS:
R1: The OLS estimator is biased.
R2: Provided that the error term is not autocorrelated, the OLS
estimator is consistent.
R3: The standard t and F tests are valid only asymptotically.
R4: In the case of economic and financial data, lags of y are usually
highly correlated. Consequently, near multicollinearity is frequently
a problem when estimating an AR(p) process for economic or
financial time series. It is common to obtain large standard errors
and insignificant t-statistics when the model is estimated.

100
Estimating Stationary ARMA Processes

It is important to note that the OLS estimator of the parameters of

an AR(p) process is inconsistent if the error term is autocorrelated.
Consequently, it is important to check for evidence of
autocorrelation in the error term after estimating an AR(p) process.
The presence of autocorrelation in the errors is an indication that
an insufficient number of lags of y has been included in the
regression equation.

101
Estimating Stationary ARMA Processes

Estimating by OLS a model with a MA component is more

complicated in two respects than estimating a pure AR process.
We need to use arbitrary starting values for one or more of the error
terms.
The first-order conditions for obtaining the OLS estimator are highly
non-linear in the parameters and cannot be solved analytically.
For example, suppose that we have a sample of T observations
(y1 , y2 , . . . , yT ),
and we wish to estimate the parameters of the ARMA(1,1) model
yt = ϕ1 yt−1 + et + θ1 et−1 . (1.24)

102
Estimating Stationary ARMA Processes

To estimate (1.24) by OLS, we need to construct

T
X
RSS = e2t .
t=1
Equation (1.24) implies that
et = yt − ϕ1 yt−1 − θ1 et−1 . (1.25)

103
Estimating Stationary ARMA Processes

Therefore, using (1.25), we obtain

e1 = y1 − ϕ1 y0 − θ1 e0
e2 = y2 − ϕ1 y1 − θ1 e1
e3 = y3 − ϕ1 y2 − θ1 e2
..
.
eT = yT − ϕ1 yT−1 − θ1 eT−1 .

Clearly, we cannot construct

e1 = y1 − ϕ1 y0 − θ1 e0 ,
since it depends on y0 which is not part of your sample.

104
Estimating Stationary ARMA Processes

However, given a value for e1 , we can construct the remaining

error terms
e2 , e3 , . . . , eT .
We can estimate ϕ1 and θ1 in (1.24) by assigning an arbitrary value,
usually zero, to e1 and minimizing
T
X T
X
e2t = (yt − ϕ1 yt−1 − θ1 et−1 )2
t=2 t=2
by choice of ϕ1 and θ1 .

105
Estimating Stationary ARMA Processes

This is a standard calculus problem, and the first-order conditions

for a solution are

PT 2 T
∂ t=2 et
X
= −2 yt−1 (yt − ϕ1 yt−1 − θ1 et−1 ) = 0. (1.26)
∂ϕ1 t=2
PT 2 T
∂ t=2 et
X
= −2 et−1 (yt − ϕ1 yt−1 − θ1 et−1 ) = 0. (1.27)
∂θ1 t=2
When we substitute recursively for the error terms in (1.26) and
(1.27), we obtain two equations that are highly non-linear in the
parameters ϕ1 and θ1 and which cannot be solved analytically. The
first-order conditions can only be solved by numerical methods.

106
Estimating Stationary ARMA Processes

Recall that in order to estimate ϕ1 and θ1 in (1.24) by OLS we had

to assign an arbitrary value to e1 . We can see the effect of this
arbitrary choice on the subsequent errors by recursively
substituting for et−1 in

et = yt − ϕ1 yt−1 − θ1 et−1 . (1.25)

When we do so, we obtain

et = yt − ϕ1 yt−1 − θ1 et−1
= yt − ϕ1 yt−1 − θ1 (yt−1 − ϕ1 yt−2 − θ1 et−2 )
=.
=.
= yt − (ϕ1 + θ1 )yt−1 + ..... − (−θ1 )t−2 ϕ1 y1 +
107
(−θ1 )t−1 e1 . (1.28)
Estimating Stationary ARMA Processes

It is clear from (1.28) that the effect of e1 on et depends on the

parameter θ1 .
If
|θ1 | < 1, ((1.29))
the effect of e1 on et declines exponentially as t increases.
However, if
|θ1 | ≥ 1,
the influence of the arbitrary starting value increases with t.
To estimate an ARMA(p, q) model by conditional OLS, it is
necessary to assign arbitrary values, usually zero, to

ep , ep−1 , . . . , ep−q+1 .

108
Estimating Stationary ARMA Processes

Box and Jenkins (1970) developed a technique for estimating

stationary ARMA(p, q) models that avoids assigning arbitrary
values to
ep , ep−1 , . . . , ep−q+1 .
The details of their procedure are beyond the scope of this unit.
Essentially, they use the sample values
(y1 , y2 , . . . , yT )
to “backcast” the values of y in periods before the start of the
sample. That is, they backcast
(y0 , y−1 , ...)
and use these backcasts together with the ARMA equation to
calculate
e0 , e−1 , ..., e−q+1 .
109
Estimating Stationary ARMA Processes

Many software packages use this backcasting approach when

estimating ARMA models.

110
Outline

1 Introduction
2 I.I.D. Processes and White Noise
3 Stationary AR(1) Processes
4 Stationary AR(p) Processes
5 MA(q) Processes
6 Stationary ARMA(p,q) Processes
7 Properties of Sample Mean and Sample Correlogram
8 Autocorrelation test for y
9 Estimating Stationary ARMA Processes
10 ARMA vs Structural Econometric Models

111
ARMA Models vs Structural Econometric Models

Suppose that we are interested in modeling the aggregate quantity

demanded of some good. One approach is to use a structural
econometric model.
A structural econometric model would use economic theory and/or
common sense to identify the variables that influence demand. An
example of a simple linear structural model for quantity demanded
is
Qt = β0 + β1 Pt + β2 It + et (1.62)
where Q denotes quantity demanded, P denotes the price of the
good, I denotes aggregate income.

112
ARMA Models vs Structural Econometric Models

Notice the following features of (1.62):

Its structure, or more specifically the regressors, is suggested by
economic theory.
The parameters have an interesting economic interpretation. For
example, β1 is the marginal effect of price on demand.
It has behavioral content in the sense that it describes the behavior
of consumers in the economy.
Since
QT+1 = β0 + β1 PT+1 + β2 IT+1 + eT+1 (1.63)
if we wish to forecast QT+1 , we need to be able to forecast PT+1 and
IT+1 , since QT+1 depends on PT+1 and IT+1 . Knowledge of past values
of Q does not enable us to forecast future values of Q.

113
ARMA Models vs Structural Econometric Models

Instead of using a structural model, we could use an ARMA model

to model quantity demanded. For example, on the basis of an
inspection of the SACF and the SPACF, we might decide to model
demand as an AR(2) process and estimate
Qt = c + ϕ1 Qt−1 + ϕ2 Qt−2 + et . (1.64)

114
ARMA Models vs Structural Econometric Models

Note the following feature of (1.64):

It is not based on economic theory.
It has no behavioral content. Equation (1.64) does not describe the
behavior of consumers.
The parameters have no interesting economic interpretation. For
example,
∂Qt
ϕ1 =
∂Qt−1
does not tell us anything interesting about the behavior of
consumers.
We can use (1.64) to forecast demand as far into the future as we
wish using only our estimates of the parameters in (1.64) and the
sample values
Q1 , Q2 , . . . , QT .
In particular, we don’t need forecasts of the future values of any
other variables such as P and I in order to forecast Q.
115
ARMA Models vs Structural Econometric Models

Whether a structural or ARMA model is best for modelling demand

depends in part on the purpose of the model.
If we wish to investigate economic behavior, then clearly structural
models are superior to ARMA models. However, for short term
forecasting an ARMA model may be superior to a structural model.

116
ARMA Models vs Structural Econometric Models

Finally, we could combine structural and ARMA modelling to

forecast future values of Q. For example, we could use two
separate ARMA models to forecast PT+1 and IT+1 and then use
QT+1 = β0 + β1 PT+1 + β2 IT+1 + eT+1 (1.63)
to forecast QT+1 . Then
QFT+1 = β̂0 + β̂1 PFT+1 + β̂2 IFT+1
where QFT+1 denotes the forecast value of QT+1 and PFT+1 and IFT+1
respectively denote the forecast values of PT+1 and IT+1 obtained
from the ARMA models.

117