STA457: Time Series Analysis
Lecture 9
Lijia Wang
Department of Statistical Sciences
University of Toronto
Lijia Wang (UofT) STA457: Time Series Analysis 1 / 32
Overview
Last Time:
1 Partial Auto-correlation
2 Forecasting
3 Estimation
Today:
1 Partial Auto-correlation Function
2 Forecasting
3 Estimation
Lijia Wang (UofT) STA457: Time Series Analysis 2 / 32
Outline
1 Partial Autocorrelation
2 Forecasting
Best Linear Prediction (BLP)
One-step Ahead BLP
One-step Ahead BLP of AR(p) Processes
3 Estimation
The Method of Moment (MOM) estimate
The Maximum Likelihood Estimate (MLE)
Lijia Wang (UofT) STA457: Time Series Analysis 3 / 32
The Partial Autocorrelation (PACF)
Definition: The PACF, ωhh , is the correlation between xt+h and xt with
the linear dependence of xt+1 , xt+2 , · · · , xt+h→1 on each, removed. In
particular, The partial autocorrelation function (PACF) of a stationary
process, xt , denoted ωhh , for h = 1, 2, · · · , is
ω11 = corr (xt+1 , xt ) = ε(1)
and
ωhh = corr (xt+h → x̂t+h , xt → x̂t ) .
Lijia Wang (UofT) STA457: Time Series Analysis 4 / 32
More on PACF
Note:
1 The term regression here refers to regression in the population sense.
That is, x̂t+h is the linear combination of xt+h→1 , x!
t+h→2 , . . . , xt+1
that minimizes the mean squared error E [(xt+h → h→1 ϑ x
j=1 j t+j ) 2 ].
2 In other words, we calculate the correlation of xt and xt+h after
removing the linear e!ects of the variables xt+1 , . . . , xt+h→1 on both
xt and xt+h .
3 If the series is Gaussian, then ωhh = Cor(xt+h , xt |xt+1 , . . . , xt+h→1 );
that is ωhh is the correlation coe”cient between xt+h and xt in the
bivariate distribution of (xt+h , xt ) conditional on {xt+1 , . . . , xt+h→1 }
Lijia Wang (UofT) STA457: Time Series Analysis 5 / 32
Example: PACF
Consider the AR(1) process given by xt = ωxt→1 + wt , with |ω| < 1.
Calculate the PACF ωhh
For a general AR(p) process, the way to calculate PACF ωhh will be given
after the forecasting session.
Lijia Wang (UofT) STA457: Time Series Analysis 6 / 32
Outline
1 Partial Autocorrelation
2 Forecasting
Best Linear Prediction (BLP)
One-step Ahead BLP
One-step Ahead BLP of AR(p) Processes
3 Estimation
The Method of Moment (MOM) estimate
The Maximum Likelihood Estimate (MLE)
Lijia Wang (UofT) STA457: Time Series Analysis 7 / 32
Introduction
The next topic: Forecasting using Time-series Models
In forecasting, the goal is to predict future values of a time series,
xn+m , m = 1, 2, · · · , based on the data collected to the present,
x1:n {x1 , x2 , · · · , xn }.
Throughout this section, we will assume xt is stationary and the
model parameters are known.
Lijia Wang (UofT) STA457: Time Series Analysis 8 / 32
Forecasting
We denote xn+m n the m step ahead forecast of xt from xn (i.e. xn+m )
given {x1 , . . . , xn }.
We will restrict attention to predictors that are linear functions of the
data, that is, predictors of the form
n
"
n
xn+m = ϑ0 + ϑ k xk
k=1
where ϑ0 , ϑ1 , · · · , ϑn are real numbers.
We want to find ϑ0 , ϑ1 , · · · , ϑn minimize the mean square error
# n
$2
E xn+m → xn+m ,
Lijia Wang (UofT) STA457: Time Series Analysis 9 / 32
Forecasting
We note that the ϑ s depend on n and m, but for now we drop the
dependence from the notation.
For example, if n = m = 1, then x21 is the one-step-ahead linear
forecast of x2 given x1 . That is, x21 = ϑ0 + ϑ1 x1 .
But if n = 2, x32 is the one-step-ahead linear forecast of x3 given x1
and x2 . That is, x32 = ϑ0 + ϑ1 x1 + ϑ2 x2 .
Definition: Linear predictors of this form that minimizes the mean
square prediction error are called best linear predictors (BLPs).
Lijia Wang (UofT) STA457: Time Series Analysis 10 / 32
Best Linear Prediction
Given data x1 , x2 , · · · , xn , the best linear predictor,
n
"
n
xn+m = ϑ0 + ϑ k xk ,
k=1
for m ↑ 1, is found by solving
#% n
& $
E xn+m → xn+m xk = 0, k = 0, 1, · · · , n,
where x0 = 1, for ϑ0 , ϑ1 , · · · , ϑn . These equations are called the
prediction equations.
Lijia Wang (UofT) STA457: Time Series Analysis 11 / 32
Best Linear Prediction (BLP) non-zero mean
If the series xk has mean µ (i.e. E (xt ) = µ), then solving equation
involves x0 implies
"n
µ = ϑ0 + ϑk µ,
k=1
therefore ' (
n
"
ϑ0 = 1→ ϑk µ.
k=1
For the following discussion, without loss of generality, we assume µ = 0.
Lijia Wang (UofT) STA457: Time Series Analysis 12 / 32
One-step Ahead BLP
We first consider one-step-ahead prediction. That is, given {x1 , . . . , xn },
we wish to forecast the value of the time series at the next time point,
xn+1 . The BLP of xn+1 is of the form
n
xn+1 = ωn1 xn + ωn2 xn→1 + · · · + ωnn x1 .
Note:
In this case, the coe”cient ϑk is ωn,n→k+1 , for k = 1, . . . , n.
The last coe”cient ωn,n , is the PACF of lag h = n.
Lijia Wang (UofT) STA457: Time Series Analysis 13 / 32
One-step Ahead BLP of Stationary Processes
How to find the coe”cients ωn1 , . . . , ωnn ?
Since ϑk ’s are found by solving
)' n
( *
#% & $ "
n
E xn+1 → xn+1 xk = E xn+1 → ωn,n→k+1 xk xk = 0,
k=1
for k = 0, 1, · · · , n, this implies
n
"
ωnj ϖ(k → j) = ϖ(k).
j=1
And ωn,n→k+1 can be solved from the equations.
Lijia Wang (UofT) STA457: Time Series Analysis 14 / 32
One-step Ahead BLP: Matrix notation
The prediction equations can also be written in matrix notation as
!n ωn = ε n ,
where !n = {ϖ(k → j)}nj,k=1 is an n ↓ n matrix, ωn = (ωn1 , . . . , ωnn )↑ is an
n ↓ 1 vector and ε n = (ϖ(1), . . . , ϖ(n))↑ is an n ↓ 1 vector.
The coe”cients ωn is solved by
ωn = !→1
n εn
It is sometimes convenient to write the one-step-ahead forecast in vector
notation
n
xn+1 = ω↑n xn .
Lijia Wang (UofT) STA457: Time Series Analysis 15 / 32
Example: Best Linear Prediction for Stationary Processes
Example 1: Consider the AR(1) series
xt = ωxt→1 + wt
where wt is white noise with variance ϱw2 and the model parameters are
known. Suppose that we have observed x1 and x2 , and we would like to
estimate x3 . Find the best linear predictor of x3 .
Lijia Wang (UofT) STA457: Time Series Analysis 16 / 32
One-step Ahead BLP of AR(p) Processes
If the time series is a causal AR(p) process, then, for n ↑ p,
n
xn+1 = ω1 xn + ω2 xn→1 + · · · + ωp xn→p+1 .
In other words, for a causal AR(p) process, the BLP coe”cients equal
the AR coe”cients, i.e. ωn,n→k+1 = ωn→k+1
For ARMA models in general, the prediction equations will not be as
simple as the pure AR case.
for n large, inverting !n will be hard and one may turn to the
Durbin–Levinson Algorithm to solve the inversion iteratively.
Lijia Wang (UofT) STA457: Time Series Analysis 17 / 32
Iterative Solution for the PACF
As mentioned in previous slides, the last coe”cient of BLP ωnn is the
PACF at lag h = n
This implies that the PACF can be calculated by estimating the
one-step ahead BLP as ωnn , for n = 1, 2, . . . .
If we let n = p, for an AR(p) model
p
xp+1 = ωp1 xp + ωp2 xp→1 + · · · + ωpp x1
= ω1 xp + ω2 xp→1 + · · · + ωp x1 ,
which implies the partial autocorrelation coe”cient at lag p, ωpp , is
also the last coe”cient in the model, ωp .
Lijia Wang (UofT) STA457: Time Series Analysis 18 / 32
m-step Ahead BLP
Given data, {x, . . . , xn }, the m-step-ahead predictor is
n (m) (m) (m)
xn+m = ωn1 xn + ωn2 xn→1 + · · · + ωnn x1 ,
(m) (m)
where {ωn1 , . . . , ωnn } can be solved by solving equations
n
" (m)
ωnj ϖ(k → j) = ϖ(m + k → 1).
j=1
Similarly, the prediction equations can again be written in matrix notation
as
(m) (m)
!n ωn = ε n ,
and the solutions can be derived in matrix notations as well.
Lijia Wang (UofT) STA457: Time Series Analysis 19 / 32
Prediction Error
Definition: The mean square m-step-ahead prediction error is
n
# n
$2
Pn+m = E xn+m → xn+m
Lijia Wang (UofT) STA457: Time Series Analysis 20 / 32
Prediction Interval
To assess the precision of the forecasts, prediction intervals are typically
calculated along with the forecasts.
In general, 100(1 → ϑ)% prediction intervals are of the form
+
n n
xn+m ± Cω/2 Pn+m ,
where Cω/2 is chosen to get the desired degree of confidence.
For example, if the process is Gaussian, then choosing Cω/2 = 2 will yield
an approximate 95% prediction interval for xn+m .
Lijia Wang (UofT) STA457: Time Series Analysis 21 / 32
Outline
1 Partial Autocorrelation
2 Forecasting
Best Linear Prediction (BLP)
One-step Ahead BLP
One-step Ahead BLP of AR(p) Processes
3 Estimation
The Method of Moment (MOM) estimate
The Maximum Likelihood Estimate (MLE)
Lijia Wang (UofT) STA457: Time Series Analysis 22 / 32
Estimation
The next topic: Estimation of ARMA(p,q) Models
Throughout this section, we assume we have n observations,
x1 , x2 , · · · , xn , from a causal and invertible Gaussian ARMA(p, q)
process in which, initially, the order parameters, p and q, are known.
Our goal is to estimate the parameters, ω1 , ω2 , · · · , ωp , ς1 , ς2 , · · · , qq ,
and ϱw2 .
Lijia Wang (UofT) STA457: Time Series Analysis 23 / 32
Yule-Walker equations for AR(p) process
We first consider the method of moment (MOM) estimate:
Consider AR(p) models
xt = ω1 xt→1 + ω2 xt→2 + · · · + ωp xt→p + wt .
The Yule-Walker equations are given by
ϖ(h) = ω1 ϖ(h → 1) + ω2 ϖ(h → 2) + · · · + ωp ϖ(h → p); h = 1, 2, · · · , p,
ϱw2 = ϖ(0) → ω1 ϖ(1) → ω2 ϖ(2) → · · · → ωp ϖ(p).
Lijia Wang (UofT) STA457: Time Series Analysis 24 / 32
Yule-Walker equations for AR(p) process
In matrix notation, the Yule-Walker equations are
!p ω = ε p , ϱw2 = ϖ(0) → ω↑ ε p ,
where
!p = {ϖ(k → j)}pj,k=1 ,
ω = (ω1 , ω2 , · · · , ωp )↑ ,
and
ε p = (ϖ(1), ϖ(2), · · · , ϖ(p))↑ .
Lijia Wang (UofT) STA457: Time Series Analysis 25 / 32
Yule-Walker equations for AR(p) process
Using the method of moments, we replace ϖ(h) by ϖ̂(h) and solve
→1
ˆ
ω̂ = !p ε̂ p ,
and
↑ ˆ →1
ϱ̂w2 = ϖ̂(0) → ε̂ p !p ε̂ p .
These estimators are typically called the Yule-Walker estimators.
Lijia Wang (UofT) STA457: Time Series Analysis 26 / 32
Yule-Walker equations for AR(p) process
For calculation purposes, it is sometimes more convenient to work with the
sample ACF. We can write the Yule-Walker estimates as
→1
ω̂ = R̂ p ϑ̂p
and
↑ →1
ϱ̂w2 = ϖ̂(0)[1 → ϑ̂p R̂ p ϑ̂p ],
where R̂ p = {ε̂(k → j)}pj,k=1 and ϑ̂p = (ε̂(1), ε̂(2), · · · , ε̂(p))↑ .
Lijia Wang (UofT) STA457: Time Series Analysis 27 / 32
Large Sample Results for Yule–Walker Estimators
The asymptotic n ↔ ↗ behavior of the Yule-Walker estimators in the case
of causal AR(p) processes is as follows:
↘ d % &
n(ω̂ → ω) →
↔N 0, ϱw2 !→1
p ,
2 P
ϱ̂w →
↔ ϱw2 .
Lijia Wang (UofT) STA457: Time Series Analysis 28 / 32
Large Sample Distribution of the PACF
For a causal AR(p) process, asymptotically n ↔ ↗,
↘ d
nω̂hh →
↔ N(0, 1),
for h > p.
Lijia Wang (UofT) STA457: Time Series Analysis 29 / 32
The Maximum Likelihood Estimate (MLE)
iid
If we assume the innovations wt ≃ N(0, ϱ 2 ), the parameters can also be
estimated via the Maximum Likelihood Estimate (MLE) approach.
For a normal ARMA(p, q) model, let φ = (µ, ω1 , . . . , ωp , ς1 , . . . , ςq )↑ be
the (p + q + 1)-dimensional vector of the model parameters. The
likelihood can be written as
n
,
L(φ, ϱw2 ) = f (xt | xt→1 , . . . , x1 ).
t=1
The conditional distribution of xt given xt→1 , . .-
. , x1 is Gaussian with mean
xtt→1 and variance Pt→1 such that Ptt→1 = ϖ(0) t→1 j=1 (1 → ω 2 ).
jj
Lijia Wang (UofT) STA457: Time Series Analysis 30 / 32
The Maximum Likelihood Estimate (MLE)
Note that:
Optimizing the Likelihood function analytically is usually infeasible
and one may need to turn to the New-Raphson’s method or other
numerical approach.
In particular, as n ↔ ↗,
↘ . /
d % 2 →1
&
n φ̂ → φ →↔ N 0, ϱw #p,q .
The asymptotic variance–covariance matrix of the estimator φ̂ is the
inverse of the information matrix.
Lijia Wang (UofT) STA457: Time Series Analysis 31 / 32
Summary on MOM and MLE
The Method of Moment method may give suboptimal estimator.
An improved approach is the use the estimates obtained from the
Method of Moment approach as initial values for the Maximum
Likelihood approach to get an optimal estimate.
Under appropriate conditions, for causal and invertible ARMA
processes, the maximum likelihood estimator, initialized by the
method of moments estimator, provide optimal estimators of ϱw2 and
φ, in the sense that ϱ̂w2 is consistent, and the asymptotic distribution
of φ̂ is the best asymptotic normal distribution.
Lijia Wang (UofT) STA457: Time Series Analysis 32 / 32
Forecasting
The minimum mean square error predictor of xn+m is:
n
xn+m = E (xn+m | x1:n )
Because the conditional expectation minimizes the mean square error
E [xn+m → g (x1:n )]2 ,
where g (x1:n ) is a function of the observations x1:n .
We will restrict attention to predictors that are linear functions of the
data, that is, predictors of the form
n
!
n
xn+m = ω0 + ω k xk
k=1
where ω0 , ω1 , · · · , ωn are real numbers.
Lijia Wang (UofT) STA457: Time Series Analysis 22 / 35