LINEAR PROCESS
BUDDHANANDA BANERJEE
1. S TATIONARITY
Definition 1. Strictly Stationary: If the joint distribution of(Xt 1, Xt 2, . . . , Xt k )
is the same as (Xt 1 +h, Xt 2 +h, . . . , Xt k +h ) for all time shifts h and for any
choice of time points t 1, t 2, . . . , t k .
Weakly (Second-order) Stationary if the first two moments sat-
isfy:
(1) E[Xt ] = µ, a constant for all t.
(2) Var(Xt ) = 2 , a constant for all t.
(3) Cov(Xt, Xt+h ) = (h), a function of h only.
F Mean: The mean of a time series {Xt } is given by:
µ = E[Xt ]. (1.1)
F Variance: The variance of a time series is defined as:
2
= E[(Xt µ) 2 ]. (1.2)
F Long-run Variance: The long-run variance accounts for depen-
dence in the time series and is given by:
1
X
2
LR = (0) + 2 (h), (1.3)
h=1
where (h) is the autocovariance function.
Date: Last updated March 11, 2025.
1
LINEAR PROCESS 2
F Autocovariance: The autocovariance function at lag h is de-
fined as:
(h) = E[(Xt µ)(Xt+h µ)]. (1.4)
F Autocorrelation: The autocorrelation function (ACF) at lag h
is given by:
(h)
⇢(h) = . (1.5)
(0)
F ⇢(h) = ⇢( h)
F Let 1 i, j n and define a matrix R = ⇢(|i j |) i, j then
aT Ra 0 for all 0 , a 2 Rn . R is positive semidefinite matrix and
() and ⇢() are positive semidefinite functions.
2. L INEAR PROCESS
Definition 2. An class of weakly stationary time series known as linear
process defined by
j=+1
X
Xt = µ + j Wt j
j= 1
j=+1
X
where µ 2 R and < 1 and W j ⇠ W N (0, 2
| j| w ).
j= 1
F E(Xt ) = µ
j=+1
X
2
F X (h) = w j j h <1
j= 1
Theorem. Consider a (weakly) stationary time series {Xt } with mean
zero and define
j=+1
X
Yt = µ + j Xt j
j= 1
LINEAR PROCESS 3
j=+1
X
where µ 2 R and | j| < 1 then
j= 1
E(Yt ) = µ
and
X j=+1
k=+1 X
Y (h) = k j X (h j + k) if exists.
k= 1 j= 1
Definition. A sequence of random variables Y1, Y2, · · · converges in mean
square to Z if for which
lim E(Yn Z )2 = 0
n!1
F Consider AR(1) process Xt = Xt + Wt where, Wt ⇠ W N (0, 2
1 w)
and | | < 1.
F AR(1) is an M A(1) process i.e.
2
* Wt j +/ = 0
Xk
lim E . Xt j
, -
k!1
j=0
F Convergence in mean square =) Convergence in probability
=) Convergence in distribution, BUT NOT THE OTHER WAY
ROUND IN GENERAL.
F ## MA(1) Example
[Link](123);
n<-500; p<-0; d<-0;q<-1;
[Link]<-[Link](list(order=c(p,d,q), ma=0.7), n)
[Link]([Link])
LINEAR PROCESS 4
3
2
1
[Link]
0
−1
−3
0 100 200 300 400 500
Time
acf([Link],type = "correlation",plot = T)
Series [Link]
0.8
ACF
0.4
0.0
0 5 10 15 20 25
Lag
pacf([Link],plot = T)
LINEAR PROCESS 5
Series [Link]
0.3
Partial ACF
0.1
−0.1
−0.3
0 5 10 15 20 25
Lag
## AR(1) Example
[Link](123);
n<-500;
p<-1; d<-0;q<-0;
[Link]<-[Link](list(order=c(p,d,q), ar=0.7), n)
[Link]([Link])
3
2
1
[Link]
−1
−3
0 100 200 300 400 500
Time
LINEAR PROCESS 6
acf([Link],type = "correlation",plot = T)
0.8 Series [Link]
ACF
0.4
0.0
0 5 10 15 20 25
Lag
pacf([Link],plot = T)
Series [Link]
0.6
0.4
Partial ACF
0.2
0.0
0 5 10 15 20 25
Lag
## ARMA(1,1) Example
[Link](123);
n<-500;
p<-1; d<-0;q<-1;
[Link]<-[Link](list(order=c(p,d,q), ar=0.7, ma=0.4), n)
LINEAR PROCESS 7
[Link]([Link])
4
2
[Link]
0
−2
−4
0 100 200 300 400 500
Time
acf([Link],type = "correlation",plot = T)
Series [Link]
0.8
ACF
0.4
0.0
0 5 10 15 20 25
Lag
pacf([Link],plot = T)
LINEAR PROCESS 8
Series [Link]
0.2 0.4 0.6 0.8
Partial ACF
−0.2
0 5 10 15 20 25
Lag
## AR(1) and $MA(\infty)$ Example
[Link](123);
t<-100
d<-0;
arsim<-numeric(0)
masim<-numeric(0)
for(i in 1 : 5000){
p<-1; q<-0;
ar1<-[Link](list(order=c(p,d,q), ar=0.7), t)
p<-0; q<-500;
mainf<-[Link](list(order=c(p,d,q), ma=(0.7)^(seq(1:q))), t)
arsim[i]<-ar1[t] ;masim[i]<-mainf[t]}
plot(density(arsim), main="density");
lines(density(masim),col="red")
LINEAR PROCESS 9
density
0.20
Density
0.10
0.00
−4 −2 0 2 4 6
N = 5000 Bandwidth = 0.2316
plot(ecdf(arsim), main="ecdf");
lines(ecdf(masim),col="red")
ecdf
1.0
0.8
0.6
Fn(x)
0.4
0.2
0.0
−4 −2 0 2 4 6
F Causality: A linear process Xt is causal function of Wt if
X1
*
Xt = 1 + i+
i B Wt
, i=1 -
P1
where i=1 | i | < 1
LINEAR PROCESS 10
F AR(1) i.e. Xt = Xt 1 + Wt is casual iff | | < 1 i.e. 1 z has a
solution out of the unit circle on complex plane C.
F Invertibility: A linear process Xt is invertible function ofWt if
there
X 1
*
Wt = 1 + ✓ i B i + Xt
, i=1 -
P1
where i=1 |✓i | < 1
F MA(1) i.e. Xt = Wt + ✓Wt 1 is invertible iff |✓| < 1 i.e. 1 + ✓ z has a
solution out of the unit circle on complex plane C.
F In general ARMA (p, q) process p (B)Xt = ⇥q (B)Wt causal if
p (z) = 0 has roots out of the unit circle and is invertible if of is
invertible ⇥q (z) = 0 has roots out of the unit circle in complex
plane C. Unique stationary solution exists iff p (z) = 0 has no
solution on |z| = 1.
D EPARTMENT OF M ATHEMATICS, IIT K GARAGPUR
URL: [Link]
E-mail address: bbanerjee@[Link]