Modeling Univariate Time Series

T. V. Ramanathan

Plaksha University
[Link]@[Link]

Autoregressive Models

February 10, 2026

T. V. Ramanathan Univariate Time Series Analysis

Autoregressive Model of order one: AR(1)

Suppose Xt is related to the lagged values (Xt−1 , Xt−2 , ...).

Then, an obvious thing to try is a regression of Xt on its lags.

A simple model for this would be AR(1), or the

“autoregressive model of order 1: Xt = ϕXt−1 + Zt ”.

The AR(1) Model with drift (non-zero mean):

Xt = µ + ϕXt−1 + Zt , Zt ∼ WN(0, σ 2 )

(We may assume normality for the white noise Zt , in such a

case uncorrelated and independence are the same - For data,
this assumption needs to be verified using the residuals)

Here Zt is assumed to be independent of Xt−1 , Xt−2 , Xt−3 , ....

T. V. Ramanathan Univariate Time Series Analysis

Autoregressive Model of order one: AR(1) (Contd.)
Thus, the model consists of two parts, the part that depends
on the past and the part that is not predictable from the past.

Notice that in AR(1) model, Xt depends on the past, only

through Xt−1 .

Thus, the knowledge of previous lag values (Xt−2 , Xt−3 , ...) is

not going to help to predict Xt , if we already know the value
of Xt−1 . (Markov Property!)

We will denote the conditional mean µt by

µt = E (Xt |Xt−1 , Xt−2 , ...) = µ + ϕXt−1
or, when we assume Zt ∼ Normal(0, σ 2 ),

f (Xt |Xt−1 = xt−1 , Xt−2 = xt−2 , ...) = f (Xt |Xt−1 = xt−1 )

= Normal(µ + ϕxt−1 , σ 2 )
T. V. Ramanathan Univariate Time Series Analysis
Autoregressive Model of order one: AR(1) (Contd.)
Assuming stationarity, the unconditional mean and variance
of Xt can be proved to be (Prove!)
µ σ2
E (Xt ) = (not µ), Var (Xt ) =
1−ϕ 1 − ϕ2

Consider an AR(1) process with drift µ = 0. Then,

Xt = ϕXt−1 +Zt , t = 0, ±1, ...., Zt ∼ WN(0, σ 2 ), |ϕ| < 1.

Each Zt is uncorrelated with Xs for s ≤ t.

Repeated substitution can lead to the equation
∞
X
Xt = ϕj Zt−j .
j=0

This series exists (converges) only when |ϕ| < 1, the condition
for the stationarity of an AR(1) model.
T. V. Ramanathan Univariate Time Series Analysis
Autoregressive Model of order one: AR(1) (Contd.)
From the above representation, we can show that;
σ2
E (Xt ) = 0, Var (Xt ) = .
1 − ϕ2

Note that when |ϕ| is close to one, the stationary variance will
be very large (see the plot).

The ACVF γx (h) (assuming stationarity) is

γx (h) = Cov (Xt , Xt+h )

= Cov (Xt , ϕXt+h−1 + Zt+h )
= Cov (Xt , ϕXt+h−1 ) + Cov (Xt , Zt+h )
= ϕ γx (h − 1) + 0
= ϕ2 γx (h − 2)
= ϕh γx (0).

T. V. Ramanathan Univariate Time Series Analysis

Autoregressive Model of order one: AR(1) (Contd.)
It may be noted that γx (h) = γx (−h).

For h = 0,

γx (0) = Cov (Xt , Xt ) = Cov (ϕXt−1 +Zt , ϕXt−1 +Zt ) = ϕ2 γx (o)+σ 2 ,

as Cov (Xt−1 , Zt ) = 0.

Therefore, we have

σ2
γx (0) = , ϕ ̸= ±1.
1 − ϕ2

Hence the autocorrelation function of an AR(1) process is

γx (h)
ρx (h) = = ϕ|h| , h = 0, ±1, ...
γx (0)

T. V. Ramanathan Univariate Time Series Analysis

Simulated AR(1) Series Xt = 0.3Xt−1 + Zt

T. V. Ramanathan Univariate Time Series Analysis

ACF/PACF of Simulated AR(1) Series Xt = 0.3Xt−1 + Zt

T. V. Ramanathan Univariate Time Series Analysis

AR(1) with ϕ = 0.4, ϕ = 0.9 and ϕ = 0.99

T. V. Ramanathan Univariate Time Series Analysis

Autoregressive Model of order two: AR(2)
AR(2) : Xt = µ + ϕ1 Xt−1 + ϕ2 Xt−2 + Zt , Zt ∼ WN(0, σ 2 )
(Assumed to be stationary - Conditions in terms of the
parameters will be discussed later)

Note that the stationary mean E (Xt ) = µ/(1 − ϕ1 − ϕ2 ).

The stationary variance γ(0) can be computed from the

following equations:

γ(0) = ϕ1 γ(1) + ϕ2 γ(2) + σ 2

γ(1) = ϕ1 γ(0) + ϕ2 γ(1)
γ(2) = ϕ1 γ(1) + ϕ2 γ(0)

These equations are obtained by multiplying the AR(2)

equation with Xt , Xt−1 and Xt−2 respectively and taking
expectations (for simplification, we assumed µ = 0.).

Dividing the first equation by γ(0) and simplifying, we get

T. V. Ramanathan Univariate Time Series Analysis
Autoregressive Model of order one: AR(2) (Contd.)

σ2
γ(0) =
1 − ϕ1 ρ(1) − ϕ2 ρ(2)

Dividing second equation by γ(0), we get

ρ(1) = ϕ1 /(1 − ϕ2 ).

Dividing third equation by γ(0), we get

ϕ21 + ϕ2 (1 − ϕ2 )
ρ(2) = ϕ1 ρ(1) + ϕ2 =
1 − ϕ2

Substituting for ρ(1) and ρ(2) in the expression of γ(0) and

simplifying (Verify!), we get the expression for variance as:
(1 − ϕ2 ) σ2
γ(0) =
(1 + ϕ2 ) ((1 − ϕ2 )2 − ϕ21 )

T. V. Ramanathan Univariate Time Series Analysis

Variance of a stationary AR(2) : A different Derivation
(May be skipped)
Consider a stationary AR(2) model (assuming µ = 0,)

Xt = ϕ1 Xt−1 + ϕ2 Xt−2 + Zt , Zt ∼ WN(0, σ 2 )

V (Xt ) = ϕ21 V (Xt−1 )+ϕ22 V (Xt−2 )+2ϕ1 ϕ2 Cov (Xt−1 , Xt−2 )+σ 2

This is same as

γ(0) = ϕ21 γ(0) + ϕ22 γ(0) + 2ϕ1 ϕ2 γ(1) + σ 2

But, we know that

ρ(1) = γ(1)/γ(0) = ϕ1 /(1 − ϕ2 ).

This implies that γ(1) = [ϕ1 /(1 − ϕ2 )]γ(0).

Therefore, V (Xt ) is given by

T. V. Ramanathan Univariate Time Series Analysis
Autocorrelation function of AR(2) Models

γ(0) = ϕ21 γ(0) + ϕ22 γ(0) + 2ϕ1 ϕ2 ϕ1 γ(0)/(1 − ϕ2 ) + σ 2

= σ 2 /[1 − ϕ21 − ϕ22 − 2ϕ21 ϕ2 /(1 − ϕ2 )]
= (1 − ϕ2 )σ 2 /{(1 + ϕ2 )[(1 − ϕ2 )2 − ϕ21 ]}

Multiply Xt = ϕ1 Xt−1 + ϕ2 Xt−2 + Zt by Xt−h , h > 0, on

both sides and taking expectation, we get

γ(h) = ϕ1 γ(h − 1) + ϕ2 γ(h − 2), h = 1, 2, ...

Dividing by γ(0) on both sides we get the difference equation

for the ACF of the process {Xt }

ρ(h) = ϕ1 ρ(h − 1) + ϕ2 ρ(h − 2), h = 1, 2, ....

T. V. Ramanathan Univariate Time Series Analysis

Autocorrelation function of AR(2) Models (Contd.)

The initial conditions of the difference equation are ρ(0) = 1

and ρ(1) = ρ(0)ϕ1 /(1 − ϕ2 ), which is obtained from
ρ(1) = ϕ1 ρ(0) + ϕ2 ρ(−1), by noting that ρ(1) = ρ(−1).

Thus, the solution of this difference equation ρ(h) is the ACF

of AR(2).

T. V. Ramanathan Univariate Time Series Analysis

ACF/PACF of AR(1), AR(2), AR(3) Models

T. V. Ramanathan Univariate Time Series Analysis

Sample from AR(1), AR(2) and AR(3) with
ϕ1 = 0.4; ϕ1 = .4, ϕ2 = .5; ϕ1 = .4, ϕ2 = .5, ϕ3 = −.3

T. V. Ramanathan Univariate Time Series Analysis

Sample ACF/PACF AR(p) Models with p = 1,2,3 and
ϕ1 = 0.4; ϕ1 = .4, ϕ2 = .5; ϕ1 = .4, ϕ2 = .5, ϕ3 = −.3

T. V. Ramanathan Univariate Time Series Analysis

Sample from AR(1) with ϕ1 = 0.9 & − 0.9

T. V. Ramanathan Univariate Time Series Analysis

Identification of AR Order

Suppose that the true time series process is an AR(1) model

Xt = µ + ϕXt−1 + Zt

Since ACF dampen exponentially, let us see how the PACF

behaves?

The first PACF α11 should be close to ϕ the true slope

coefficient (or ACF).

The second PACF α22 should be close to zero. (Why?)

T. V. Ramanathan Univariate Time Series Analysis

Identification of AR Order (Contd.)

Because in the AR(1) model, Xt depends directly only on

Xt−1 .

Thus, the PACF an AR series shows exactly the same

behaviour that of ACF of a MA series (same order).

Therefore, one can use PACF to identify the order of an AR

series.

Thus, the PACF of an AR(p) series cuts off after lag p.

T. V. Ramanathan Univariate Time Series Analysis