4CSDS3021: Time Series Analysis

Module 1: Univariate Stationary Time-Series Models

Lecture Notes | Hours allotted: 9

Topics covered
● Introduction to stochastic processes
● Stationary processes (strict & weak/covariance stationarity)
● Wold Representation Theorem
● Autocovariance, autocorrelation (ACF) and partial autocorrelation (PACF) functions
● Autoregressive (AR) and Moving Average (MA) models
● Conditions for stationarity and invertibility
● Box–Jenkins approach to model building
● Forecasting

1. Introduction to Stochastic Processes

A stochastic process (or random process) is a collection of random variables ordered in time, {Xt}, t = 0, ±1, ±2, ... A
time series is simply one observed realisation (sample path) of the underlying stochastic process — the actual data we
see is just one possible outcome out of infinitely many the process could have generated.

Key idea
The data-generating process (DGP) is the stochastic process; the time series we observe and analyse is a single
realisation of that process. Statistical inference in time series is about using this one realisation to learn about the
properties of the underlying DGP.

1.1 Why this matters

● Unlike cross-sectional data (many independent draws from one population), here we have one draw from a
sequence of related, evolving distributions.
● To estimate properties (mean, variance, covariance structure) from a single realisation, the process must exhibit
some regularity over time — this is where stationarity becomes essential.

2. Stationary Processes
2.1 Strict (strong) stationarity
A process {Xt} is strictly stationary if the joint distribution of (Xt1, Xt2, ..., Xtk) is identical to the joint distribution of
(Xt1+h, Xt2+h, ..., Xtk+h) for any set of time points and any time shift h. In other words, shifting the time origin has no
effect on the joint distribution of the process.

2.2 Weak (covariance / second-order) stationarity

Because strict stationarity is hard to verify in practice, we typically work with the weaker condition. A process {Xt} is
covariance (weakly) stationary if all three hold:
● Constant mean: E(Xt) = μ for all t
 ● Constant variance: Var(Xt) = σ² < ∞ for all t (finite and time-invariant)
● Covariance depends only on lag: Cov(Xt, Xt+k) = γk, a function of k alone, not of t

Note
Strict stationarity + finite second moments ⇒ weak stationarity. The converse is not generally true unless the process
is Gaussian, in which case weak and strict stationarity coincide.

2.3 Why stationarity matters

● Classical estimation (sample mean, sample autocovariances) is only meaningful if the underlying moments are
stable over time.
● Standard asymptotic theory (LLN, CLT) used for inference in AR/MA models requires stationarity.
● Non-stationary series can produce spurious regression results — a problem addressed in Module 2.

3. Wold Representation Theorem

The Wold Decomposition Theorem states that any covariance-stationary process {Xt} can be expressed as the sum of
two mutually uncorrelated components:
Xt = Σ (j=0 to ∞) ψj εt−j + κt , ψ0 = 1, Σ ψj² < ∞
● The first term is a linear (possibly infinite-order) moving-average process in white noise shocks εt (E(εt)=0,
Var(εt)=σ², uncorrelated over time).
● κt is a deterministic component (perfectly predictable from its own past, e.g. a constant or deterministic trend)
— often assumed to be zero for practical modelling.

Significance
Wold's theorem justifies representing any stationary process as an infinite-order MA(∞) process. This underlies the
Box–Jenkins strategy of approximating that infinite representation with a parsimonious finite ARMA(p, q) model.

4. Autocovariance, Autocorrelation and Partial Autocorrelation

4.1 Autocovariance function (ACVF)
γk = Cov(Xt , Xt+k) = E[(Xt − μ)(Xt+k − μ)]
● γ0 = Var(Xt)
● γk = γ−k (symmetric in k for a stationary process)

4.2 Autocorrelation function (ACF)

ρk = γk / γ0 , −1 ≤ ρk ≤ 1, ρ0 = 1
The ACF measures the linear correlation between Xt and Xt+k, including any indirect correlation transmitted through the
intervening observations Xt+1, ..., Xt+k−1.

4.3 Partial autocorrelation function (PACF)

The PACF, φkk, measures the correlation between Xt and Xt+k after removing (partialling out) the linear effect of the
intermediate observations Xt+1, ..., Xt+k−1. It isolates the direct relationship at lag k.
4.4 Using ACF/PACF for identification
Process ACF pattern PACF pattern

AR(p) Tails off (decays gradually — Cuts off after lag p

exponentially or in a damped sine
wave)

MA(q) Cuts off after lag q Tails off (decays gradually)

ARMA(p,q) Tails off Tails off

White noise ≈ 0 at all lags ≠ 0 ≈ 0 at all lags ≠ 0

5. Autoregressive (AR) Models

An AR(p) process expresses the current value as a linear combination of its own p past values plus a white-noise error
term:
Xt = c + φ1 Xt−1 + φ2 Xt−2 + ... + φp Xt−p + εt
Using the lag operator L (LXt = Xt−1), this is written compactly as φ(L) Xt = c + εt, where φ(L) = 1 − φ1L − φ2L² − ... −
φpLᵖ.

5.1 The AR(1) process

Xt = c + φ Xt−1 + εt
● Mean: μ = c / (1 − φ), provided |φ| < 1
● Variance: γ0 = σ² / (1 − φ²)
● ACF: ρk = φᵏ (geometric decay; alternating sign if φ < 0)

5.2 Estimation and forecasting

● Parameters typically estimated by OLS (conditional) or maximum likelihood (exact).
● h-step-ahead forecasts are generated recursively by iterating the fitted equation forward, replacing future error
terms with their expectation of zero.

6. Moving Average (MA) Models

An MA(q) process expresses the current value as a linear combination of the current and past q white-noise shocks:
Xt = μ + εt + θ1 εt−1 + θ2 εt−2 + ... + θq εt−q
In lag-operator form: Xt = μ + θ(L) εt, where θ(L) = 1 + θ1L + θ2L² + ... + θqLᵠ.

6.1 The MA(1) process

Xt = μ + εt + θ εt−1
● Variance: γ0 = σ²(1 + θ²)
● ACF: ρ1 = θ / (1 + θ²); ρk = 0 for k ≥ 2 (abrupt cut-off after lag 1)
6.2 Key property
Every finite-order MA(q) process is stationary by construction, for any values of the θ parameters, since it is a finite
linear combination of a stationary white-noise sequence.

7. Conditions for Stationarity and Invertibility

7.1 Stationarity condition (AR models)
An AR(p) process is stationary if and only if all roots of the characteristic equation φ(z) = 1 − φ1z − φ2z² − ... − φpzᵖ = 0
lie strictly outside the unit circle (|z| > 1). Equivalently, in terms of the inverse roots (eigenvalues), all must lie strictly
inside the unit circle.
● AR(1): stationary iff |φ| < 1
● AR(2): stationary iff φ1 + φ2 < 1, φ2 − φ1 < 1, and |φ2| < 1

7.2 Invertibility condition (MA models)

An MA(q) process is invertible if it can be re-written as a convergent infinite-order AR process — i.e., the current shock
εt can be expressed as a convergent function of present and past Xt's. This requires all roots of θ(z) = 1 + θ1z + ... + θqzᵠ
= 0 to lie strictly outside the unit circle.
● MA(1): invertible iff |θ| < 1

Why invertibility matters

Invertibility ensures a unique, one-to-one mapping between the MA parameters and the observed autocorrelation
structure, and allows the process to be approximated by a finite AR model — essential for practical estimation and
forecasting.

8. The Box–Jenkins Approach

The Box–Jenkins methodology is an iterative, three-stage procedure for building an appropriate ARMA model for a
given stationary series.

Stage 1 — Identification
● Check for stationarity (plot the series; examine ACF/PACF; formal unit-root tests, covered in Module 2).
● Difference or transform the series if necessary to achieve stationarity.
● Examine sample ACF and PACF plots (correlograms) to tentatively identify candidate orders p and q, using the
patterns in Section 4.4.

Stage 2 — Estimation
● Estimate the parameters of the tentative ARMA(p,q) model(s), usually via maximum likelihood.
● Compare candidate models using information criteria (AIC, BIC/SIC, HQIC) — lower values indicate a better
fit relative to complexity.

Stage 3 — Diagnostic checking

● Examine the residuals: they should resemble white noise (no remaining autocorrelation).
● Use the Ljung–Box (portmanteau) test to formally test for residual autocorrelation.
● Check ACF/PACF of residuals; look for significant spikes indicating model inadequacy.
● If diagnostics fail, return to Stage 1 and revise the model specification — hence the process is iterative.
9. Forecasting
Once a satisfactory ARMA(p,q) model is fitted, it can be used to generate minimum mean-square-error (MMSE)
forecasts.

9.1 Principle
The optimal h-step-ahead forecast is the conditional expectation of Xt+h given information up to time t: X̂ t+h|t = E(Xt+h
| Xt, Xt−1, ...). Future error terms are replaced by zero (their expected value); past/current errors and observations are
replaced by their known/estimated values.

9.2 Forecast error and interval

● Forecast error: et+h|t = Xt+h − X̂ t+h|t, with E(et+h|t) = 0 for a correctly specified model.
● Forecast error variance grows with the horizon h (uncertainty compounds), which is used to construct forecast
intervals (e.g. forecast ± 1.96 × forecast-error standard deviation for a 95% interval).

9.3 Behaviour by model type

● AR forecasts: decay geometrically toward the unconditional mean μ as h → ∞.
● MA(q) forecasts: equal the unconditional mean μ for horizons h > q, since shocks beyond lag q have no further
effect.

Summary
Module 1 in one paragraph
A time series is a single realisation of an underlying stochastic process. Meaningful inference from that single
realisation requires (weak) stationarity — constant mean, constant variance, and autocovariances that depend only on
lag. Wold's theorem shows any stationary process can be written as an infinite MA process, motivating the finite-
order AR and MA (and combined ARMA) approximations. The ACF/PACF correlogram patterns help identify
candidate orders, subject to stationarity conditions (AR) and invertibility conditions (MA). The Box–Jenkins
identify–estimate–diagnose cycle formalises model building, and the fitted model is then used to generate optimal
forecasts with quantifiable uncertainty.