ARIMA Models

Handout

Stationarity

Strictly stationary time series

Properties

E(ys ) = E(yt ) = μfor all s, t

​ ​

Weakly stationary time series

Finite Variance Process

Properties

μt - Constant (does not depend on t)

​

Constant variance (does not depend on t)

γs,t = γs+h,t+h 
​ ​

If they have the same time lag, the covariance should be the same. (i.e.
Cov(y1 , y2 ) = Cov(y2 , y3 ) = Cov(y3 , y4 ). This is pretty useful in
​ ​ ​ ​ ​ ​

checking if the series is stationary

Note that if it is a function of h, it can still be stationary so long as it is not a

function of t

Identification of Stationary Time Series

Time Series Plot

1. Stationary time series

a. Roughly horizontal

b. Has constant variance

c. No predictable patterns in the long term (i.e. no trend)

ARIMA Models 1
 2. Non-stationary time series

a. Has trend or seasonality. They are not stationary since the mean is changing

ACF

1. Stationary series drops to 0 relatively quickly

2. Non-stationary series decreases slowly

3. Non-stationary series for r1 is often large and positive

​

Note that for ACF plots, it does not need to be all within the blue lines since we
are not looking at white noise.

Differencing

yt′ = yt − yt−1
​ ​ ​

💡 Note that we will only have T-1 data since we can’t compute the difference for
the first observation y1′ 
​

Purpose: To get the ACF to be well behaved. But note that we don’t need to get it to
white noise (but this will be ideally)

Unit Root Test

Kwiatkowski-Phillips-Schmidt-Shin (KPSS)

yt = μ + ρt + μt + et
​ ​ ​

H0 - Data are stationary and non seasonal

​

H1 - Data are not stationary

​

ARIMA Models 2
 Idea: We will fit a linear regression line on the data. If the test statistic is large it
means that it is on one side of the line and therefore it could be non stationary.

We can keep applying the KPSS test repeatedly to successive differencing to

determine the number of differencing that should be carried out.

p-value: If the p-value is significant then we can reject the null hypothesis and
conclude that we need to difference the data

Backshift Notation

Byt = yt−1
​ ​

The backshift operator Boperates on the yt and it will shift it back in time by 1
​

d-th order difference:

(1 − B)d yt ​

Seasonal Difference:
First seasonal difference is given by

(1 − Bm )yt ​

Autoregressive Model

yt = c + ϕ1 yt−1 + ϕ2 yt−2 + ⋯ + ϕp yt−p + et

​ ​ ​ ​ ​ ​ ​ ​

💡 Note that cis not the mean of the time series. It is a function of the mean and
the ϕcoefficients. c = μ − ϕ1 μ − ϕ2 μ − ⋯ − ϕp μ
​ ​ ​

ARIMA Models 3
 ϕ1 , ⋯ , ϕp - All constants
​ ​

Changing these values will result in different time series patterns

ϕp = 0
​

et - Variance of the et will only change the scale of the series, it doesn’t change the
​ ​

pattern
Roots determine if it is a casual process or not

Linear Processes

💡 Theorem: A linear process yt is defined to be a linear combination of white

​ ​
​

∞
noise et and is given by yt = μ + ∑j=−∞ ψj et−j where​ ​ ​

∞
∑j=−∞ ​
∣ψj ∣ < ∞
​

Autocovariance Function:
Note that this is infinite-MA representation

∞
γ(h) = σ 2
∑ ψj+h ψj
​ ​

j=−∞

This will show that the autocovariance is a function of hand therefore is

stationary

Note that once we have the ϕweights then we can work out γ(h)

Purpose

This gives a way “in” to work out γ(h)for AR(p)processes. Because once we
can determine that it is a linear process, we can use the expression for γ(h)

Linear Processes are also stationary

AR(1) Process is stationary

Proof is within OneNote

ARIMA Models 4
 Is it possible to have a stationary AR(1) process when ϕ1 ​ > 1

∞
yt = − ∑j=1 ϕ−j et+j 
​ ​ ​

Stationary but it is not casual since it depends on future values

Non casual time series is not really useful for us!

We want to only look at casual processes instead

Casual AR Process
An AR Process is casual if it can be expressed by the previous values and not in
terms of future values .

To check if a process is casual

The roots of the AR(p)operator τ(B) = 1 − ϕ1 B − ϕ2 B2 − ⋯ − ϕp Bp 

​ ​ ​

determine if the AR(p)process is casual, and hence stationary

Need to check if the complex roots lie outside the unit square, it is then it is a
casual process

Example
For AR(1),

τ(z) = 1 − ϕ1 z = 0 ​

1
z=
​ ​

ϕ1 ​

ARIMA Models 5
 MA Process
yt = c + ϵt + θ1 ϵt−1 + θ2 ϵt−2 + ⋯ + θq ϵt−q
​ ​ ​ ​ ​ ​ ​ ​

ϵt - White Noise
​

This is a multiple regression with past errors as predictors.

Note: This is not moving average smoothing!

Writing AR(p)process as MA(∞)

Note that it is possible to write any stationary AR(p)process as an MA(∞)
process
Example: AR(1) Process

yt = ϕ1 yt−1 + ϵt
​ ​ ​ ​

= ϕ1 (ϕ1 yt−2 + ϵt−1 ) + ϵt

​ ​ ​ ​ ​

= ϕ21 yt−2 + ϕ1 ϵt−1 + ϵt

​
​ ​ ​ ​ ​

= ϕ31 yt−3 + ϕ21 et−2 + ϕ1 ϵt−1 + ϵt

​ ​ ​ ​ ​ ​ ​

Provided −1 < ϕ1 < 1

​

Invertibility
General conditions for invertibility

💡 Complex roots of 1 + θ1 z
on the complex plane
​ + θ2 z 2 + ⋯ + θq z q lie outside the unit circle
​ ​

Essentially, solve for the roots of the moving average polynomial and check that the
roots are outside the unit circle

To check for complex roots, z = x + yi

Check the magnitude of ∣z∣ = x2 + y2  ​

ARIMA Models 6
 If ∣z∣ > 1then it lies outside the unit circle

Conditions for different models

For q = 1: −1 < θ1 < 1 ​

For q = 2: −1 < θ2 < 1 ​ θ2 + θ1 > −1​ ​ θ1 − θ2 < 1

​ ​

More complicated conditions hold for q ≥ 3

Properties

1. If the MA model is invertible, any MA(q) process can be written as an AR(∞)

process

Invertibility of an ARIMA model is equivalent to forecastability of an ETS model

ARIMA
Components

AR: Auto Regressive (Lagged Observations as inputs)

I: Integrated (Differencing to make series stationary)

MA: Moving Average (Lagged errors as inputs)

Interpretability: It is rarely interpretable in terms of visible data structures like trend and
seasonality. But it can capture a huge range of time series patterns.

ARMA Model Equation

yt = c + ϕ1 yt−1 + ⋯ + ϕp yt−p
​ ​ ​ ​ ​

+ θ1 et−1 + ⋯ + θq et−q + et
​ ​

​ ​ ​ ​ ​

Note that this is when we have not included the I part which represents the
differencing component of it.

ARIMA Model Equation

yt′ = c + ϕ1 yt−1
​
′
​
′
+ ⋯ + ϕp yt−p
​ + θ1 et−1 + ⋯ + θp et−p + et
​ ​ ​ ​ ​ ​ ​

(1 − ϕ1 B − ⋯ − ϕp B )(1 − B)yt = c + (1 + θ1 B + ⋯ + θq Bq )et

​
p
​ ​ ​ ​ ​

ARIMA Models 7
 Writing in a more succinct manner

τ(B)(1 − B)d yt = c + κ(B)et

​ ​

τ(B)- AR Component
(1 − B)d - Number of Difference
κ(B)- MA Component
c = μ(1 − ϕ1 − ⋯ − ϕp )
​ ​

Telling different processes apart

Sometimes some processes may look like others if we add in some redundant terms.

Note that if we add terms and reorder it, it does not change the ARIMA order

What we can do is to factorize the yt side and et side to see if there are any terms
​ ​

that are similar for both the autoregressive operator and moving average operator

We try to get the equation to the base form and that will be the underlying ARIMA
model. We want the Parsimonious Model (model with the least parameters)

Example
Looking at the following equation, we may think that it is a ARIMA(2, 0, 1)
process

yt − 0.8yt−1 + 0.15yt−2 = et − 0.3et−1

​ ​ ​ ​ ​

However, we can do the following

(1 − 0.8B + 0.15B2 )yt = (1 − 0.3B)et

​ ​

(1 − 0.3B)(1 − 0.5B)yt = (1 − 0.3B)et

​

​
​

(1 − 0.5B)yt = et ​ ​

We can see that since there is a common term between them, we can cancel it
and this is just an ARIMA(1, 0, 0) model

Note that if we fit the 2 models, they will still produce similar forecasts

Predictors: Lagged values of yt and lagged errors

​

Conditions on ARcoefficients ensure stationarity

Conditions on MAcoefficients ensure invertibility

ARIMA Models 8
 Link between ARIMA and ARMA
If yt follows ARIMA(p, d, q) ⇔(1 − B)d yt follows ARMA(p, q)
​ ​

Special Cases of ARIMA Models

Model Name Model Parameters

White Noise ARIMA(0, 0, 0) with no constant

Random Walk ARIMA(0, 1, 0) with no constant

Random Walk with Drift ARIMA(0, 1, 0) with a constant

Autoregression ARIMA(p, 0, 0)

Moving Average ARIMA(0, 0, q)

Relationship of c, dwith long term forecasts

Note that the p, qpart affects the short term look of the data and the c, daffects how
the long term forecasts look like

c d Long Term Forecast Pattern

0 0 Long-term forecasts goes to 0

0 1 Long-term forecasts will go to a non-zero constant

0 2 Long-term forecasts will follow a straight line

=
 0 0 Long-term forecasts will go to the mean of the data

=
 0 1 Long-term forecasts will follow a straight line

=
 0 2 Long-term forecasts will follow a quadratic trend

Cyclic Behaviours

Note that this is something unique to ARIMA models that ETS models can’t do. ETS
models can only handle seasonality and not cyclic patterns

Conditions:

p ≥ 2and some restrictions on coefficients are required

If p = 2, we need ϕ21 + 4ϕ2 < 0. Then average cycle of length will be
​ ​

(2π)/[arccos (−ϕ1 (1 − ϕ2 )/4ϕ2 )]

​ ​

ARIMA Models 9
 Notations
p- Refers to the AR part
d- Refers to the degree of differencing involved
q- Refers to the MA part

Determining p, d, q

d- We can repeatedly use KPSS test to decide on d

p- Finding the last significant lag for PACF plot (Note that this is only for pure
AR models)

q- Finding the last significant lag for ACF plot (Note that this is only for pure MA
models)

Code Output

Partial Autocorrelations
Measures the relationship between yt and yt−k , when the effects of other time lags -
​ ​

1, 2, 3, ⋯ , k − 1are removed

Definition
Given a time series yt , the partial autocorrelation of lag k is denoted, ϕk,k , is the
​ ​

autocorrelation between yt and yt+k with the linear dependence of yt on yt+1 to
​ ​ ​ ​

ARIMA Models 10
 yt+k−1 removed.
​

It is the autocorrelation between yt and yt+k that is not accounted for by lags 1 ​ ​

through k − 1inclusive

ϕ1,1 = ρ(zt+1 , zt ),
​ for k = 1
​ ​

ϕk,k = ρ(zt+k − z^t+k , zt − z^t ), for k ≥ 2

​ ​

​ ​ ​ ​ ​

where z^t+k , z^t are linear combinations of {zt+1 , zt+2 , ⋯ , zt+k−1 }that
​ ​ ​ ​ ​

minimises the mean squared error of zt+k and zt respectively. ​ ​

Note: For stationary processes, the coefficients in z^t+k and z^t are the same but ​ ​

reversed

z^t+k = β1 zt+k−1 + ⋯ + βk−1 zt+1 

​ ​ ​ ​ ​

z^t = β1 zt+1 + ⋯ + βk−1 zt+k−1 

​ ​ ​ ​ ​

Reasoning for this

When we are looking at the correlation between yt and yt−1 , it may induce k  ​ ​

lags of autocorrelation since yt−1 may be autocorrelated with yt−2 and so on ​ ​

αk = kth partial autocorrelation coefficient

​

= estimate of ϕk in the following regression:

​
​

yt = c + ϕ1 yt−1 + ϕ2 yt−2 + ⋯ + ϕk yt−k + et

​ ​ ​ ​ ​ ​ ​ ​

Idea behind this:

Regression on the various lagged values and taking out the coefficient for the k th lag
value since this tells us how much the k th lag explains the current time value

Features:

α1 = ρ1 
​ ​

Same critical values of ±1.96/ T as for ACF

​

Last significant αk indicates the order of an AR Model

​

Forecasting

ARIMA Models 11
 ^t+h∣t is chosen to be g(y1 , ⋯
The point forecast y ​ ​ ​ , yt )s.t. ​

E[yt+h − g(y1 , ⋯ , yt )]2

​ ​ ​ is minimised

In general, the appropriate g(.)is

E[yt+h ∣y1 , ⋯ , yt ] ​ ​ ​

Prediction Error

E[yt+h − y^t+h∣t ]2 ​ ​ ​

Example: AR(p)

y^t+h∣t = ϕ1 y^t+h−1∣t + ϕ2 y^t+h−2∣t + ⋯ + ϕp y^t+h−p∣t

​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Example: ARMA(p, q)

y^t+h∣t = ϕ1 y^t+h−1∣t + ϕ2 y^t+h−2∣t + ⋯ + ϕp y^t+h−p∣t + θ1 e^t+h−1∣t + θq e^t+h−q∣t

​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

y^i∣t = yi for i ≤ t ​ ​ ​

e^i∣t = 0 for i > t ​

e^i∣t = τ(B)y^i∣t − θ1 e^i−1∣t − ⋯ − θq e^i−q∣t

​ ​ ​ ​ ​ ​ ​

Forecast Variance Relationship with d

Higher value of d→ More rapidly the prediction intervals increase in size

For d = 0, the long-term forecast standard deviation will go to the standard deviation
of the historical data

SARIMA

ARIMA(p,d,q)(P, D, Q)m ​

ARIMA Models 12
 Model Form

τ(B)(1 − B12 )D (1 − B)d yt = c + κ(B)et ​ ​

(1 − B12 )- Extra differencing term where we are taking the difference between 12
months apart. Note that this is 12 because this is monthly data
τ(B)- It is a function of 2 polynomials
(1 + ϕ1 B + ⋯ + ϕq Bq )(1 + Φ1 B12 + Φ2 B24 + ⋯ + ΦQ B12Q )
​ ​ ​ ​ ​

Note that there are 2 components where one is the non seasonal part and the
other is the seasonal part

κ(B)- It is a function of 2 polynomials

(1 + θ1 B + ⋯ + θq Bq )(1 + θ1 B12 + θ2 B24 + ⋯ + θQ B12Q )
​ ​ ​ ​ ​

Note that there are 2 components where one is the non seasonal part and the
other is the seasonal part

Features

If it is a pure seasonal model, we can fall back on the same guidelines for
identifying ARIMA model. But note that the spikes will be at the seasonal periods

Pure Seasonal Pure Seasonal

Pure Seasonal ARMA(P, Q)
AR(P) MA(Q)

ACF Tail off at lags k ⋅ s Cut off after Q ⋅ s Tails off

PACF Cut off after lag P ⋅ s Tail off at k ⋅ s Tails off

Example 1
SARIMA(0, 0, 0)(1, 0, 1)12  ​

yt = Φyt−12 + et + Θet−12
​ ​ ​ ​

yt − Φyt−12 = et + Θet−12
​ ​ ​ ​

(1 − ΦB12 )yt = (1 + ΘB12 )et

​ ​

τ(B)yt = κ(B)et ​ ​

ARIMA Models 13
 Note that since we do note have any components that are non multiples of 12 (there
are no non seasonal components)

This is a pure seasonal model, because there are no non-seasonal parameters

Example 2
SARIMA(0, 1, 1)(0, 1, 1)12  ​

yt − yt−1 − yt−12 − yt−13 = et + θet−1 + Θet−12 − θΘet−13

​ ​ ​ ​ ​ ​ ​ ​

(1 − B)yt − (1 − B)(B12 )yt = (1 + θB)et + ΘB12 Bet + θΘB12 et

​ ​ ​ ​ ​

(1 − B)(1 − B12 )yt = (1 + θB)et + ΘB12 (1 + θB)et

​ ​ ​

(1 − B)(1 − B12 )yt = (1 + θB)(1 + ΘB12 )et

​ ​

ETS vs ARIMA

Equivalence relationship between ETS and ARIMA models

ETS Model ARIMA Model Parameters

ETS(A,N,N) ARIMA(0,1,1) θ1 = α − 1 
​

ETS(A,A,N) ARIMA(0,2,2) θ 1 = α + β − 2 θ 2 = 1 − α 
​ ​

ETS(A, Ad , N)
​
ARIMA(1,1,2) ϕ1 = ϕθ1 = α + ϕβ − 1 − ϕθ2 = (1 − α)ϕ
​ ​ ​

ETS(A,N,A) ARIMA(0,1,m)(0, 1, 0)m  ​

ARIMA Models 14
 ETS Model ARIMA Model Parameters

ARIMA(0,1,m+1)
ETS(A,A,A)
(0, 1, 0)m 
​

ARIMA(1,0,m+1)
ETS(A,Ad ,A)
(0, 1, 0)m 
​

Note that once there are multiplicative terms, there will not be any equivalent ARIMA
model (i.e. Non-linear exponential smoothing models do not have equivalent ARIMA
counterparts)

Comparison between Models

Note that since ETS and ARIMA are 2 different class of models, we can only compare
their forecast accuracy. We cannot compare their AIC since they have different
number of parameters and will not give a good enough estimate.

ETS Models

Combination of different components (Error, Trend, Seasonality)

Flexible model since it allows for trend etc to change over time

All non-stationary.

Models with seasonality or non-damped trend (or both) have two unit roots; all
other models have one unit root

Number of roots = Number of differencing required

ARIMA Models

Modelling autocorrelations after differencing

Potentially ∞number of models

All stationary models & many large models

Example: Equivalence of ETS(A, N, N)and ARIMA(0, 1, 1)

yt = lt−1 + et
​ ​ ​

lt = lt−1 + αet
​ ​

​ ​ ​

Conversion

ARIMA Models 15
 lt − lt−1 = αet
​ ​

​
​

From (∗)

(1 − B)yt = (1 − B)lt−1 + (1 − B)et

​ ​ ​

= αet−1 + et − et−1
​
​ ​ ​

(1 − B)yt = et + (α − 1)et−1
​ ​ ​

Invertibility Condition

∣α − 1∣ < 1
⇔0<α<2

Example: Equivalence of ETS(A, N, N)and ARIMA(0, 2, 2)

yt = ℓt−1 + bt−1 + et
​ ​
⋯ (∗)​ ​

ℓt = ℓt−1 + bt−1 + αet ⇒ (1 − B)ℓt = bt−1 + αet

​

​
​ ​ ​ ​ ​ ​

bt = bt−1 + βet ⇒ (1 − B)bt = βet

​ ​ ​ ​ ​

ARIMA Models 16