Chapter 4

Random Process (or Stochastic Process)

Deterministic signals are those signals whose occurrence is deterministic in nature.

Thus every time the signal is produced, the same signal is generated. On the contrary,

the random signals will have different form after each occurrence. As an example,

suppose we measure the temperature of a room every day from 9 AM to 10Am. Each

time, the temperature is recorded, the signal will have different shape. Similarly, if we

generate the signal from an oscillator with random initial phase, then, every time the

signal is generated, the signal will be different because of random initial phase value.

Thus deterministic signal after each repetition remains same whereas the random signal

has different waveform after each repetition. Consider the collection of signals recorded

in different trial and shown in Figure 1.3. Here, a collection of five different time signals

are shown. Each of them is called sample function. The collection of sample functions

is called ensemble. Each sample function shows the recoding of signal in different trial.

We note that we have obtained different functions. Had it been deterministic signal, we

would have obtained the same function in each trial. This process of having different

signals in every trial is called Random Process. If we observe the random process at

certain instant of time, we obtain different possible values. Thus, we obtain random

variable by observing random process at certain instant of time.

Examples of Random process

(1) Random fluctuations in air temperature and air pressure.

(2) Thermal Noise voltage generated in the resistor of electronic device.

(3) Speech signal.

(4) Signal generator’s output.

(5) An audio signal that is transmitted over telephone line.

(6) In digital communication, signal generated by information source.

(7) Output of channel.

(8) Receiver Noise.

(9) Throwing of dice N number of times and repeating the experiment (discrete

random process)

(10)Call arrival at the phone exchange. Note that if we record the call arrival at the

exchange since morning till night, then, this will correspond to one sample function.

If we record the same for seven days, we will get seven sample function and each

will be different from others.

(11) Traffic arrival at the petrol pump.

 Figure 1.3

For instance, speech signal is a random process, If we try to know the value of the

signal at certain moment, it may have a continuous range of − ∞ to + ∞ . This is

continuous random variable. Similarly, if sample functions are discrete in nature, then

the random variable obtained after observing the process at certain time instant will be

discrete random variable which will have finite set of numbers. For instance, Figure 1.4

shows one example of discrete random variable.

 Figure 1.4 Random Waveform (Digital)

One example of random process is the signal Y = Cos(ωt + θ ) generated by signal

oscillator with initial phase θ . Note that every time the oscillator is switched on, the initial

phase of the signal may be different. The ensemble of signals generated with randomly

varying initial phase is shown in Figure 1.5.

Figure 1.5

Auto correlation of Random Process

Consider the random process as shown in the Figure 1.6 (a). After observing the

random process at time instant t = t1 , we obtain random variable X (t1 ) . Similarly at

other time instant t = t2 , the random variable obtained is X (t 2 ) . It is interesting to know

how these two random variables are correlated to each other. An analogy is the

comparison between a photograph of a person with the same photograph taken at

different instant of time. Of course two photographs taken at at the same time will have
maximum correlation whereas if other photograph is delayed in time, say, 1 year, 2

years etc, then, the two photograph will be lesser and lesser correlated.

From Equation (8) and (9), it is very much obvious that the term E[XY] is used to

measure the correlation between two random variable X and Y. From Equation (8), the

minimum value of E[XY] should be equal to m X mY . In this case, Co-variance will be zero

and X and Y will be uncorrelated. The maximum value of E[XY] will be obtained when

X=Y. In this case, both the random variables are exactly same. This is equal to E[X2]

which is variance for zero mean random variable.

m X mY ≤ E[ XY ] ≤ E[ X 2 ]

The term E[XY] is termed as autocorrelation of two random variables X and Y. If in the

random process X(t), X = X (t1 ) and Y = X (t 2 ) , two random variables observed at two

time instant t1 and t 2 , then, autocorrelation of X (t1 ) and X (t 2 ) is given as

Figure 1.6: (a) Random Process (b) Autocorrelation

RX (t1 , t2 ) = RX (τ ) = E[ X (t1 ) × X (t2 )] (13)

Here τ = t1 − t 2 is the time window. It can be shown that autocorrelation between two

random variables observed at two time instants t1 and t 2 for stationary random process

is dependent on only the difference between these two time instants (called time-

window) and not on the time instants.

∞
RX (t1 , t2 ) = E[ X (t1 ) X (t 2 )] = ∫x x
−∞
1 2 × f X ( t1 ) X ( t2 ) ( x1 x2 )dx1dx2 (13.1)

In order that a process be stationary, it’s distribution function should remain invariant to

the time (this is discussed below). Thus, distribution function in (13.1) should be

independent of the time. It implies that the autocorrelation R X (t1 , t 2 ) does not depend on

the time instants t1 , t 2 . Now for given random variables X (t1 ) and X (t 2 ) observed at time

instants t1 , t 2 , the joint distribution function is fixed and hence it is concluded that for

stationary process, the autocorrelation function depends only on the time difference

τ = t1 − t2 . This is the reason why RX (t1, t2 ) is written as R X (τ ) . Now, if the correlation is

rapidly decreasing with increase in this window, then, the random process is said to be

rapidly changing. On the contrary, if autocorrelation is slowly varying with τ , then the

random process is slowly varying random process. This is shown in the Figure 1.6 (b).

Properties of Autocorrelation

(1) Autocorrelation function R X (τ ) is an even function.

RX (τ ) = RX (−τ )

This can easily be verified. Let τ = t1 − t2 ⇒ t2 = t1 − τ

RX (τ ) = RX (t1 , t2 ) = E[ X (t1 ) × X (t1 − τ )]

Let t1 − τ = t '

This implies that R X (τ ) = R X (t1 , t 2 ) = E[ X (t '+τ ) × X (t ' )] = E[ X (t ' ) × X (t '+τ )] = R X ( −τ )

(2) RX (τ ) ≤ RX (0)

(3) R X (0) = Variance of X if the mean of X is zero.

Example 1: Consider the following Random Process

X (t ) = ACos(2πf c t + θ )

Where, A and f c are constants and θ is uniformly distributed random variable. Find the

autocorrelation.

Solution: Since the random variable is uniformly distributed and hence

Figure 1.7
 π
∫ fϑ (θ )dθ = 1
−π

fθ (θ ) =
1
−π ≤θ ≤ π
2π

=0
+π
RX (t , t + τ ) = E[ X (t ) X (t + τ )] = A2 ∫ cos(2πf c t + θ ) × cos(2πf c (t + τ ) + θ ) × fθ (θ )dθ
−π elsew
A2
+π here
=
2π ∫ cos(2πf t + θ ) × cos(2πf (t + τ ) + θ )dθ
c c
−π
X (t ) = ACos(2πf c t + θ )
2 +π
A 1
2π 2 −∫π
= × cos(4πf ct + 2πf cτ + 2θ ) + cos(2πf cτ )dθ

A2
=cos(2πf cτ )
2
Regarding the first part of the integration to be zero, it can easily be observed that the

cosine function in the interval -π to π, has the equal area of positive cycle and negative

cycle. So total area will be zero.

Considering random process as X (t ) = A sin(2πfct + θ ) , we obtain the

+π
RX (t , t + τ ) = E[ X (t ) X (t + τ )] = A2 ∫ sin( 2πf ct + θ ) × sin(2πf c (t + τ ) + θ ) × fθ (θ )dθ
−π
2 +π
A 1
[cos(2πf cτ ) − cos(4πf ct + 2πf cτ + 2θ )]dθ
2π 2 −∫π
= ×

A2
= cos(2πf cτ )
2

So autocorrelation is even function. Therefore, the autocorrelation of random process

with any linear combination of sin and cos will also be an even function. This is shown in

the following example.

 Example: Check if the autocorrelation of the following random process is even function.

X (t ) = Asin(2πf ct + θ ) + A cos(2πf ct + θ )

Solution: Using the trigonometric property, and following the above method, it can

easily be shown that the autocorrelation of X(t) will be even function.

+π
E[ X (t ) X (t + τ )] = A2 ∫ [sin(2πf c t + θ ) + cos(2πf ct + θ )]× [sin( 2πf c (t + τ ) + θ ) + cos(2πf c (t + τ ) + θ )]× fθ (θ )dθ
−π
2 +π
A 1
[sin(2πf ct + θ ) sin(2πf c (t + τ ) + θ )] + [sin(2πf ct + θ ) cos(2πf c (t + τ ) + θ )]
2π 2 −∫π
= ×

+ [cos(2πf ct + θ ) sin( 2πf c (t + τ ) + θ )] + [cos(2πf ct + θ ) cos(2πf c (t + τ ) + θ )]dθ

= A cos(2πf cτ )
2

Thus it can be concluded that the autocorrelation for any periodic function with arbitrary

intial phase will be even function.

Now any signal x(t) is sum of even function xev(t) and xod(t). That is,

x(t ) = xev (t ) + xodd (t )

It can be shown from the Fourier analysis that xev (t ) is nothing but the set of cosine

function and xodd (t ) is a set of sine functions. Thus, it can easily be verified following

above examples that autocorrelation of X(t) will have even function.

Note that if the PDF of cos(2πf ct + θ ) and cos(2πfc (t + τ )) is fθ (θ ) and PDF of

cos(2πf ct + θ ) × cos(2πf c (t + τ ) + θ ) is also fθ (θ ) . Thus, the autocorrelation depends only on

the time window. In other word, keeping the time-window fixed, if it shifted at any

position, the autocorrelation obtained will remain same.

Wide-sense and strict-sense stationary process

Random process can be stationary or non-stationary. In a crude sense, the process is

said to be stationary if some of its property remains constant irrespective of time when it

is observed. This property simplifies the design communication system. If the property is

variant with respect to time, then, the process is termed as non-stationary. We know

that statistical property of random variable is characterized by its distribution function.

Let us have n random variables X (t1 ), X (t 2 ),...., X (t n ) obtained by cutting the random

process at time instant t1 , t2 ,..., tn respectively. Now these n random variables are

characterized by its joint distribution function f X ( t1 ),...., X ( tn ) ( x1 , x2 ,.., xn ) . In other words,

these random variables are known only by their joint PDF. The properties such as

correlatedness, orthogonality etc between these random variables can only be obtained

by their joint PDF. Now let us obtain the joint PDF of another set of random variables

X (t1 + τ ), X (t2 + τ ),...., X (tn + τ ) . These PDFs may or may not be same. If these two PDFs

are same, that is

f X ( t1 ),...., X ( tn ) ( x1 , x2 ,.., xn ) = f X ( t1+τ ),...., X ( tn +τ ) ( x1 , x2 ,.., xn ) (14)

Then these samples obtained at different time instant will jointly show the same

statistical property. Thus for stationary process, we can assume that if samples are

taken at different time instant t1 , t2 ,..., tn , then, joint statistical property of these random

variable will remain constant. That is, the statistics of stationary process are invariant to

any translation of time-axis. Such random process is said to be stationary process in

strict sense. These time-invariant statistical properties of these random variables greatly

simplifies the system design.. For iid random process, the condition in (14) is simplified

as

n
f X ( t1 ),...., X ( tn ) ( x1 , x2 ,.., xn ) = f X ( t1+τ ),...., X ( tn +τ ) ( x1 , x2 ,.., xn ) = ∏ f X ( ti +τ ) ( xi )
i =1
(14.1)

Hence, in order to show that a process is SSS, we have to find a generalized

distribution function of nth order and if it does not depend upon time, then, we can

conclude that the process is stationary in strict sense. For iid random process, this

becomes even easier to verify the same by simply finding first order distribution function

at any time instant and showing it to be independent of time.

Some examples of stationary process

Suppose we record the temperature of the city for entire day from morning till night. This

is one sample function. If this experiment is repeated for seven days, then, we have

seven sample functions. We know that mean of the temperature of a city varies from

morning till night. In morning it is medium, during day it is maximum and again it starts

decreasing and in night, it is minimum. So we note that statistical parameter “mean”

does not remain constant from time to time. Thus, this random process is non-

stationary.

Thermal noise of the resistor is stationary process

Wide-sense stationary (WSS)

Some time a non-stationary random process exhibits its statistical parameters such as
mean and autocorrelation to be independent of time. That is, these parameters are
invariant to the time-axis. Such process is called wide-sense stationary random
process. Thus, the condition for WSS is less stringent than that of strict-sense
stationary.

Let us find out three important tools of any random Process X (t ) .

(i) Mean
∞
m X (t k ) = E[X (t k )] = ∫ xf X (t k ) (x )dx
−∞
(ii) Auto-Correlation

∞ ∞
R X (t k , ti ) = E[X (t k )X (ti )] = ∫ ∫ xyf X (t k ) , X (ti ) (x, y )dxdy
−∞ −∞

(iii) Auto-Covariance
K x (t k , ti ) = E[{X (t k ) − m X (t k )}{X (ti ) − m X (ti )}]
 = R X (t k , t i ) − m X (t k )m X (t i )

The requisite condition for a random process to be wide-sense stationary is that

these parameters should not depend upon time instant t k , t i but should depend on only

the time-window t k − t i . Hence for WSS,

m X (t k ) = m X
R X (t k , ti ) = R X (τ ), τ = t k − ti (18)
K x (t k , ti ) = K x (τ ), τ = t k − ti

Note that if random process is strictly stationary, then all the above conditions are

definitely held but reverse may not be true. That is, even if all the above conditions are

satisfied, even then, the random process may not be stationary in strict sense.

For the process to be white, it is important that the auto co-variance of random variables

observed at two time instants should be zero. This implies that these random variables

are uncorrelated. This case is generally satisfied by thermal noise, receiver noise. This

gives the idea of white noise at the input of the receiver (white noise also means the

random variables observed at the two times instants are uncorrelated).

Hence, a wide-sense stationary represents a weak kind of stationary in the sense

that all the strictly stationary process are also wide-sense stationary but the

converse is not necessarily true. If process is not WSS, then, the process is said to

be non-stationary Hence

Steps to check if a random process is wide-sense stationary is

(i) To find the mean, auto-correlation and auto-covariance

 (ii) Check if these tools only depend on the time-window (and not on time

instants at which random variables are observed).

An example of WSS process is

X (t ) = ACos(2πf c t + θ )

A2
We note from the previous example that mean is 0 and autocorrelation is cos(τ )
2
which depends on the window of width τ and not on the time instant.

Example: Consider a sinusoidal signal with binary random phase given as

X (t ) = Cos(t + φ )

Where φ is random variable determined by

φ =0 with P(φ = 0) = 1
2
=π ,, P(φ = π ) = 1
2

We note that X (t ) is random since its value can not be determined

beforehand. Further X (t ) is not a RV since a time function rather than a
number is assigned to it.

Determine:

(a) Is X (t ) discrete or continuous.

(b) SSS or WSS
(c) While or not
(d) Is it ergodic process

Solution:

(a) For SSS, PDF should since it has two possible values at given
time instant to the time
 X (t ) t =0 = 1 with P = 1
At t = 0, 2
= −1 with P= 1
2

f X (t = 0 ) ( x ) = δ ( x − 1) + δ (x + 1)
1 1
⇒
2 2

X (t ) t = π = 1 wit h P= 1
π 2
At t = , 2
2 =0 with P= 1
2

⇒ fX (t =π 2) (x ) = 1 δ (x − 1) + 1 δ (x )
2 2

Hence, PDF is varying with time. So, it is not strict- sense stationary
process.

For WSS, its mean and auto correlation should be time- invariant.

Mean

E [ X (t )] = E [Cos (t + φ )]

× Cos (t ) + × Cos (t + π )
1 1
=
2 2
1 1
= Cost − Cost = 0
2 2

Auto-correlation:

E [X (t )X (t + τ )] = E [X (t1 )X (t 2 )]
= E [Cos (t1 + φ ) × Cos (t 2 + φ )]

E [Cos (t1 + t 2 + 2φ ) + Cos (t1 − t 2 )]

1
=
2
1 ⎧1 ⎫
= Cos (t1 − t 2 ) + ⎨ Cos (t1 + t 2 ) + Cos (t1 + t 2 + 2π )⎬
1 1
2 2 ⎩2 2 ⎭
= Cos (t1 − t 2 ) + Cos (t1 + t 2 )
1 1
2 2

Auto Co-variance
 E [{X (t1 ) − m1 }{X (t 2 ) − m2 }] = E [ X (t1 )X (t 2 )] − m1 m2

Hence, we note that the process is not a WSS process. Also it is not
white since its auto co-variance is not zero. Note that for random process to
be white, its auto co-variance should be zero, that is, the two random
variables observed at any two time instants t1 ,t 2 should be uncorrelated.

For ergodic in mean,

∞ T
E [X (t )] = ∫ x(t )dt = x(t )
1
∫ x × f X ( x)dx = Lim
−∞
T →∞ 2T −T

That is, statistical mean is time-mean. Let us take time-mean first.

∫ Cos (t + φ )dt = 2T [Sin(t + φ )]

1 1
< x (t ) >= Lim
T
−T
T →∞ 2T −T

=
1
[Sin(T + φ ) − Sin(− T + φ )]
2T
=
1
[Sin(T + φ ) + Sin(T − φ )]
2T

=
1 ⎡ (T + φ + T − φ ) Cos⎛ 2φ ⎞⎤
⎢ 2Sin ⎜ ⎟⎥
2T ⎣ 2 ⎝ 2 ⎠⎦

=
1
[2SinT × Cosφ ]
2T
2π
We note that for Cos (t + φ ), ω = = 1 ⇒ T = 2π
T
⇒ x(t ) = 0 ⇒ E[X (t )] = x(t ) = 0
So, the process is ergodic in mean. For ergodic in auto-correlation, it is quite
obvious that the process can not be ergodic in autocorrelation because
statistical average of autocorrelation E[ X (t1 )X (t 2 )], itself is depending on time
instant and hence it is not constant. For X (t ) being ergodic in
autocorrelation, its ensemble average should be equal to time-average
autocorrelation of any sample function.

Example: Consider the following random process

X (t ) = ASin(ω 0 t + φ )
where A is constant, ω 0 > 0 and PDF φ is

f (φ ) =
1
−π ≤ φ ≤ π
2π
=0 else
Check the above random process for the following
(i) Strict-sense stationarity
(ii) Wide-sense stationarity
(iii) White process
(iv) ergodicity

Solution:

For SSS, the distribution of X (t ) should be time invariant.

⎛ω π ⎞
At t = π , X (t ) t = π = ASin⎜ 0 + φ ⎟
4 4 ⎝ 4 ⎠

⎡ ⎛ω π ⎞ ⎤ ⎡ ⎧ ⎛x⎞ ⎫⎤
Pr [ X (π 4) ≤ x ] = Pr ⎢ ASin⎜ 0 + φ ⎟ ≤ x ⎥ = Pr ⎢φ ≤ ⎨Sin −1 ⎜ ⎟ − k1 ⎬⎥
⎣ ⎝ 4 ⎠ ⎦ ⎣ ⎩ ⎝ A⎠ ⎭⎦

ω0π
Where, k1 =
4
⎛x⎞ ⎛x⎞
Sin −1 ⎜ ⎟−k1 Sin −1 ⎜ ⎟ −k1
⎝ A⎠ ⎝ A⎠
Pr [ X (π 4) ≤ x] = ∫ f (φ )dφ = ∫π f (φ )dφ
−∞ −

PDF is obtained by differentiating CDF. Hence, PDF is given by

φ φ
∂ 0 ∂ 0 ∂φ
f X (x ) = ∫ f (φ )d φ = ∫ f (φ )dφ ×
∂x −π ∂φ −π ∂x
∂ ⎡ −1 ⎛ x ⎞ ⎤
= f (φ0 )×
1
⎢ Sin ⎜ ⎟ − k1 ⎥ =
∂x ⎣ ⎝ A⎠ ⎦ 2π A2 − x 2
Similarly, for any other time instant, the same distribution function will be
obtained. Therefore, PDF does not depend upon the time. So it appears that
the process might be stationary in strict sense but it cannot be confirmed
until and unless we obtain a generalized nth order distribution function.

Other parts have already been solved in other example regarding “white”
condition. If the process is white, then, auto co-variance should be zero.

C{X (t1 ), X (t 2 )} = E [{X (t1 ) − m1 }{X (t 2 ) − m2 }]

= E [ X (t1 )X (t 2 )] − m1 m2
= E [ X (t1 )X (t 2 )]
A2
= Cos (2πf cτ ) ≠ 0
2

So, the random variables at different time instants are correlated to each
other therefore, it is not white random process.

Example: Consider the following random process

X (t ) = aCosωt + bSinωt

where a & b are random variables. Determine necessary condition for a and
b for WSS.

Solution:
(i)

E[ X (t )] = E[aCosωt + bSinωt ]

= E[a]Cosωt + E[b]Sinωt (18.1)

For the process to be stationary (i.e. time-invariant), E[a] and E[b] must be
zero otherwise mean in (18.1) will depend upon time. This is necessary
condition but not a sufficient condition.
E [ X (t )X (t + τ )] = E [{aCosωt + bSinωt}× {aCosω (t + τ ) + bSinω (t + τ )}]
= a 2CosωtCosω (t + τ ) + (ab )CosωtSinω (t + τ ) + abSinωtCosω (t + τ ) + b 2 SinωtSinωt (t + τ )
a2
=[Cos(2ωt + ωτ ) + Cosωτ ] + (ab )CosωtSinω (t + τ ) + abSinωtCosω (t + τ )
2
b2
+ [Cosωτ − Cos (2ωt + ωτ )]
2
Hence, if this was to be independent of time, then,

[ ] [ ]
E a 2 = E b 2 = K (Let )
E[ab] = 0 (18.2)

⇒ RX (τ ) = K × Cosωτ

Hence (18.2) is requisite condition for the process to be WSS.

Time averages and Ergodicity

Given random process X (t ) , the ensemble average and autocorrelation are given by

∞
m X = E [X (t )] = ∫ xf ( ) (x )dx
X t
−∞ (19)

∞
R X (τ ) = E [ X (t )X (t + τ )] = ∫ xyf X (t ), X (t +τ ) (x, y )dxdy
−∞ (20)

Here, we are averaging all the sample function of random process X (t ) . In other word,

we have to have the knowledge of all the sample functions of X (t ) .

In some situation, we don’t have knowledge about all sample functions but one or

at the best a few sampling functions are recorded. For example, the output of oscillator

is assumed to be Cos (ωt + θ ) , θ = 0o. This is one sample function. In practice, the phase
θ is random variable and may lie in the range 0 ≤ θ ≤ 2π (initial phase of oscillator could

be anything between 0 & 2π ).

Hence, in such a situation, where only one sample function is available, we

cannot find ensemble average. Hence, we have to find time average of individual

sample function.

Time-averaged mean and autocorrelation are given as

+T
< x (t ) >= Lim ∫ x(t )dt
1
T → ∞ 2T
−T (21)

+T
< x (t + τ )x (t ) >= Lim ∫ x(t + τ )x(t )dt
1
T →∞ 2T −T

+T

∫ x(t − τ )x(t )dt

1
= Lim
T →∞ 2T −T (22)

Here, x(t ) is one sample function of random process X (t ) . If time-averaged “mean” and

“autocorrelation” are equal to ensemble-averaged mean and autocorrelation, then, such

random-process is called “Ergodic process”

Note:

(i) Time averages x(t ) and x(t + τ )x(t ) are random variables as their values

depend on which sample function of random process is chosen.

(ii) In general, time-average is not equal to ensemble average, except for special

class of random process known as ergodic process. Hence, if X (t ) happens to

 be ergodic process, then, to find out ensemble average, we don’t need to have

knowledge of all sample functions. We just need to know one sample function

and find time-average of this sample function and that will be equal to ensemble

average.

(iii) If a process is ergodic then it implies that process is stationary in strict-sense but

opposite may not necessarily be true, that is, strict-sense stationary process

need not be ergodic process as well.

Reason: If a process is ergodic, then, time-average will be equal to ensemble average

of statistical parameters. This is possible only when statistical parameters are invariant

to the time. This, in turn, implies that the joint PDF is constant and does not depend on

the time instant at which random variables are observed.

But, on the other hand, if random process is stationary in strict-sense, then, it

need not necessarily mean that its ensemble average will be equal to time-average.

Ergodic in mean

A statistical mean of random process X (t ) is given by

ergodic in the mean if

+T
< x (t ) >= Lim ∫ x(t )dt = m
1
X (23)
T →∞ 2T
−T

where m X is the ensemble average of X (t ) and x(t ) is one sample function.

Taking ensemble average of LHS of (23), we have

⎡ 1 T
⎤ 1 T
E⎢
⎣ 2T
∫
−T
x (t )dt ⎥=
⎦ 2T
∫ E[x(t )]dt
−T

If X (t ) is stationary, then, E [x(t )] = mX indicating that statistical mean is invariant to the

time. Hence,

⎡1 T
⎤ 1 T
E⎢
⎣ 2T
∫−T x(t )dt ⎥⎦ = 2T ∫m
−T
X dt = m X (24)

From (23) and (24), we can draw a very important conclusion that for T → ∞ , if the term

∫ x(t )dt , which is random variable, has

1
constant value (i.e. it does not depend on
2T −T

time), then, the process will definitely be ergodic in mean. As a result

⎡ T
⎤ T

∫−T x(t )dt ⎥⎦ = LimT →∞ 2T ∫ x(t )dt

1 1
E ⎢ LimT →∞ and it has been shown that for stationary
⎣ 2T −T

⎡ 1 T
⎤
process that E ⎢
⎣ 2T
∫ x(t )dt ⎥⎦ = m
−T
X .

Ergodicity in autocorrelation

∫ x(t + τ )x(t )dt = R (τ ) = E[X (t + τ )X (t )]

1
Lim X
T → ∞ 2T
−T

That is, time-domain auto-correlation is equal to its statistical average and it is given as

< x (t ) x (t + τ ) >= R X (τ ) (25)

If a process is stationary in wide-sense, then

⎡ 1 T
⎤ 1 T
E⎢
⎣ 2T
∫
−T
x (t + τ )x (t )dt ⎥=
⎦ 2T
∫ E[x(t + τ )x(t )]dt
−T

= R X (τ )

Hence, as above explained, here also, we can conclude that” The necessary and

∫ x(t + τ )x(t )dt

1
sufficient condition for ergodicity in autocorrelation is that the term
2T −T

should have constant value.

If the process is ergodic in all the parameters such as mean, autocorrelation,

auto covariance etc, then, it will automatically imply that the process is strict sense

stationary. It is simply because of the fact that if the process is ergodic in all the

parameters, then, it is possible only when all its statistical parameters are time-invariant.

This, in turn, is possible only when the PDF is time-invariant. Hence,

Ergodic process ⇒ Strict-sense stationary process

But, reverse may not be true. For example, consider the following random process

X (t ) = A
where A is a random variable. This random process has joint distribution function of all

random variable X (ti ), i = 1,2,3,..., n and its delayed version X (ti + τ ), i = 1,2,3,..., n exactly

same since X (ti ) = X (ti + τ ) . Thus, (14) is fully satisfied. Thus, this process is stationary

in strict sense. But this is not ergodic in any sense.

Example: Consider the following random process

X (t ) = ACos(2πf c t + θ )

Where, A and f c are constants and θ is uniformly distributed random variable. Show

that this process is “Ergodic process”

f θ (θ ) =
1
0 ≤ θ ≤ 2π
2π

=0 elsewhere

The mean of random process is

∞
mX = ∫ ACos (2πf t
−∞
c 1 + θ ). f θ (θ )dθ
 2π
= A ∫ Cos (2πf c t1 + θ ).
1
dv = 0
0
2π

Autocorrelation function of this process is

A2
R X (τ ) = .Cos (2πf cτ )
2

Let us take one sample function of the process i.e.

x(t ) = ACos(2πf c t + θ )

Here θ is constant

+T
x (t ) = LimT →∞ . ACos (2πf c t + θ )dt = 0
1
2T −∫T
+T
x (t )x (t + τ ) = LimT →∞ . {ACos (2πf c t + θ )}{ACos (2πf c (t + τ )) + (θ )dt }
1
2T −∫T

+T
A2
. [Cos(4πf c t + 2πf cτ + 2θ ) + Cos(2πf cτ )]dt
4T −∫T
= Lim
T →∞

A2
= .Cos(2πf cτ )
2

Hence, we see that time-averaged mean and time-averaged autocorrelation are exactly

same as ensemble average mean and ensemble average autocorrelation. Hence, the

process is “Ergodic”.

Examples of ergodic random process

Tchebycheft inequality:

(2) There are many application where we need to calculate the probability that a r.v X

is lying in certain range For this, we need pdf . But, we often, face two difficulties.

(i) Either pdf is not clearly known and only mean and variance of r.v X is

available .

(ii) or if pdf is known, then, its integration is very complex and that can not be

computed in a very straight forward manner.

Hence, under this circumstances, we need to resort to some upper bound of the

probability and we are not, much, concerned about the exact calculation of probability.

As an example, erfc function does not have the closed form expression. The probability

of error in digital communication is given as

1 ⎡ A ⎤
Pe = erfc ⎢ ⎥
2 ⎢⎣ N 0 Tb ⎥⎦

This does not have a closed form expression. It can be computed numerically only. The

other way is to have upper bound approximation which is given as follows.

We know that
 ∞

∫ exp(− z )dz
2
erfc(u ) = 2

π u

For a large positive value of u, we have upper bound on the complementary error

erfc (u ) <
(
exp − u 2 )
function u π

Thus, probability of error can be approximated as

Pe ≈
exp − u 2 ( )
2u π

A
where u =
N 0 Tb

Tchebycheff’s inequality is one such procedure that gives upper bound of certain

probability. It is given as

P( X >∈) ≤
[ ]
E X2
(26)
∈2

Here, ∈ is some positive constant value.

It says that “The probability that random variable X is greater than or equal to ∈ is less

[ ]
than or equal to X 2 ∈2 . Hence, even if pdf is not known and only variance is

available, we can calculate the upper bound this probability.

Proof: Let us define Ye a random variable such that

Ye = 1 if X ≥∈
 =0 if X <∈

where, ∈= some positive constant value. From definition, we can write that

X 2 ≥ Ye X 2 ≥∈2 Ye (27)

Hence,

[ ] [ ]
E X 2 ≥ E X 2Ye ≥∈2 E [Ye ] (28)

However,

E[Ye ] = 1× Its probability of occurrence + 0 × its prob of occurencs

= 1 ∗ P( X ≥∈) + 0 ∗ P( X <∈)
= P( X ≥∈)

Hence, putting this result, in (28), we have

[ ]
∈2 P( X ≥∈) ≤ E X 2

Or

P( X ≥∈) ≤
[ ]
E X2
∈2

Note:

• In many applications, a tighter upper bound can also serve our purpose and we

are not concerned about exact computation of probability.

 • This inequality does not require the complete distribution of X , i.e. it is

“distribution free”.

Chern-off bound

Tchebycheff inequality gives the “loose” upper bound on probability whereas chern-off

bound yields tighter bound. It is derived as follows:

Let us define

Ye = 1 If X ≥∈

=0 If X <∈

Then, we have, for all t ≥ 0

e tX ≥ e t∈ .Ye (29)

Taking ensemble average on both the sides,

[ ]
E e tX ≥ e t∈ .E [Ye ] (30)

Knowing that E[Ye ] = 1× P( X ≥∈) + 0 × P( X <∈) = P( X ≥∈)

And putting this result in (30), we have

[ ]
E e tX ≥ e t∈.P( X ≥∈)

⇒ P( X ≥∈) ≤ e −t∈ E e tX [ ]
Since, t and ∈ are independent parameters, Hence, we can choose t and ∈ such that

RHS be minimum. It will give tighter bound. Hence,

{ [ ]} { [ { }]}
P( X ≥∈) ≤ min exp(− t ∈) × E etX = min exp − t ∈ + loge E etX
t ≥0 t ≥0
(31)

Note:

• We have not imposed any condition on ∈ , i.e. whether it is positive or negative.

• It gives tighter bound on probability as compared to Tchebychef. This is an

advantage. But, disadvantage is that it requires computation of E e tX [ ] which

requires knowledge of pdf of X .

Union bound:

P( AUB) = P( A) + P(B ) − P( AB) ≤ P( A)P(B )

Since P( AB ) ≥ 0

We can generalize this idea as

⎛ ⎞
P⎜UAi ⎟ ≤ ∑ P( Ai )
⎝ i ⎠ i

This upper bound is very useful when we find the upper bound of Pe when it is very

difficult to express the result in closed form.

Example: X 1 and X2 are two independent Gaussian random variables with

μ X = μ X = 0 and σ X21 = 1 and σ X2 2 = 4 .

1 2

(a) Find the Tchebycheff and Chern-off bound on P( X 1 ≥ 3) and compare it with

exact value of P( X 1 ≥ 3) .

(b) Find the union bound on P( X 1 ≥ 3) or P( X 2 ≥ 4) and compare with exact value.

Solution: We know that

(a) P( X 1 ≥ 3) ≤ P( X 1 ≥ 3)

Because R.H.S. says X 1 can take value from +3 and above or -3 and below it

whereas L.H.S. says that X 1 can take value above +3 only.

Now, as per Tchebycheff’ inequality, P( X 1 ≥ 3) ≤

[ ]
E X2
∈2

P ( X 1 ≥ 3) ≤
1
⇒ ≅ 0.111 (i)
9

For Chernoff bound, we have

{ }
P ( X 1 ≥ 3) ≤ min e − t∈.E e tX
t ≥0

tX
Hence, let us, first, find E e . { }
∞
Ee[ ]= ∫ e
tX 1 tX 1
. f X 1 ( x1 )dx1
−∞
 ∞
⎡ 1 ⎧ x12 ⎫⎤
= ∫−∞ ⎢⎢ 2π .1 ⎨⎩− 2 ⎬⎭⎥⎥ dx1
tX 1
e . exp
⎣ ⎦
∞
=
1
(
. ∫ e x1 . exp − x12 2 dx1 )
2π −∞
∞
=
1
[
. ∫ exp tx1 − x12 2 dx1 ]
2π −∞

Now,

tx1 −
x12
2
1
[
= − x12 − 2tx1 + t 2 − t 2
2
]

=−
1
2
[
(x1 − t )2 − t 2 ]
Hence,

∞
[ ]=
2
et ⎡ 1 2⎤
2
Ee tX 1
. ∫ exp ⎢ − ( x1 − t ) ⎥dx1
2π − ∞ ⎣ 2 ⎦

Let x1 − t = z ⇒ dx1 = dz

[ ] { }
∞
⇒ Ee tX1
={
e .
1
. ∫ exp − z 2 2 dz
t2 2
( )
2π −∞
I 144 424443
II

Here, note that

1
( )
. exp − z 2 2 is pdf function of random variable z with mean 0 and
2π

variance 1. Hence its total area should be 1. So, II term results in 1. Hence,

[ ]
E e tX1 = e t
2
2

⇒ P ( X 1 ≥∈) ≤ min exp − t × 3 + t 2 2

t ≥0
( )
= min exp − 3t − t 2 2
t ≥0
[ ( )]
For R.H.S. to be minimum, we have to find t such that the term 3t − t 2 be maximum.
2

dY
Let Y = 3t − t 2 ⇒
2
= 3 − 2t 2 = 0 ⇒ t = 3 . Hence, for t = 3 , The term 3t − t 2 2 will be
dt

maximum, thus, making RHS minimum. Hence,

P( X 1 ≥ 3) ≤ exp[− 9 + 9 2] = exp(− 9 2) ≅ 0.0111

Hence, we note that Chern-off bound gives tighter bound than Tchebycheft inequality.

Now, exact value of P( X1 ≥ 3) is calculated as

∞
P ( X 1 ≥ 3) =
1
( )
.∫ exp − z 2 2 dz = Q (3) = 0.0013 (From Table of Q ).
2π 3

Hence,

From comparison of exact value with Chernoff and Tchebycheft, is concluded

that Chernoft gives tighter bound.

P ( X 1 ≥ 3 Or X 2 ≥ 4 ) ≤ P( X 1 ≥ 3) + P( X 2 ≥ 4)

P( X 1 ≥ 3) = Q(3) = 0.00013
P( X 1 ≥ 4) = Q(4) = 0.228

Hence,

P ( X 1 ≥ 3 Or X 2 ≥ 4 ) ≤ 0.0013 + 0.0228 = 0.0241

Exact value of P ( X 1 ≥ 3 or X 2 ≥ 4 ) = 0.0013 + 0.0228 − 0.0013 ∗ 0.0228 = 0.02407

Auto-Correlation and Cross–Correlation

The process of autocorrelation provides a measure of the similarly or coherence

between a given signal and a replica of signal delayed by a variable amount.

Here, our goal is to define auto-correlation for energy signal and periodic signal, then, it

is followed by random signal.

If g(t ) is energy signal that may be complex-valued ( For example QPSK signal

is complex-valued signal because real part is Cosω0t and imaginary part is Sinω0t which

can be written as jω0Cosω0t ), then, autocorrelation of this complex-valued signal is

give as

∞
Rg (τ ) = ∫ g (t )× g (t −τ )dt
∗

−∞ (19)

where g (t ) is complex conjugate of g(t ) . Here, complex conjugated version is shifted

∗

in R.H.S. by τ . If g(t ) is shifted in negative direction, then

∞
Rg (τ ) = ∫ g(t + τ )g (t )dt
∗

−∞ (20)

If g(t ) is complex-valued function, then, its auto-correlation will be also complex-valued

function.

Some Examples of complex signals

Figure 1.8 shows an example of complex signal of the form A + jB . Here A and jB are

orthogonal to each other. Similarly signals g1 (t ) and g 2 (t ) are orthogonal to each other.

These signals are shown in Figure 1.8 (a) and (b). Hence g (t ) = g1 (t ) + g 2 (t ) is a

complex signal. This is shown in Figure 1.8(c). The complex of this signal is

g * (t ) = g1 (t ) − g 2 (t ) that is shown in Figure 1.8 (d).

Figure 1.8

Example: To discuss a problem of finding autocorrelation of a complex signal in two

ways: first by using (19) and second by using (20) and to check if they are same.
Property 1- Autocorrelation function exhibits conjugate symmetry

Rg (τ ) = Rg∗ (− τ )

Rg (τ )
That is, real part of is an even function of τ , whereas the imaginary part is an odd

function of τ .This can easily be shown using (1) & (2). If we replace τ → −τ and, then,

take complex value in (1), we can get (2).

∞
Rg∗ (− τ ) = ∫ g (t )g (t + τ )dt = Rg (τ )
∗

−∞

The meaning of conjugate symmetry is as follows. Suppose R(τ ) = f1 (τ ) + j × f 2 (τ )

where functions f1 (τ ) and f 2 (τ ) are real functions. If we take complex conjugate then

R ∗ (τ ) = f1 (τ ) − j × f 2 (τ ) . If we do the replacement τ → −τ , then,

R ∗ (−τ ) = f1 (−τ ) − j × f 2 (−τ ) . If this happens to be same as R(τ ) , that is,

f1 (τ ) + j × f 2 (τ ) = f1 (−τ ) − j × f 2 (−τ ) , then, this is true only when real and imaginary

parts are separately equal. This implies that real part has to be an even function and

imaginary part has to be odd function.

Property 2- The value of autocorrelation function at origin is equal to the energy of the
signal.

∞ 2

Rg (0) = ∫ g (t ) dt
−∞ (21)
This is obtained by putting τ = 0 in (19).

Property3-

The maximum value of autocorrelation function Rg (τ ) occurs at the origin. That is,

Rg (τ ) ≤ Rg (0) for all τ

For a signal g(t ) of finite energy, Rg (0) is finite, ⇒ Rg (τ ) is also finite.

The auto-correlation function and energy spectral density are Fourier-Transform pair.

Rg (τ ) ↔ ϕg ( f )
(22)

Proof

∞
In (20), t → −δ ⇒ Rg (τ ) = − ∫ g (τ − δ )g ∗ (− δ )dδ
−∞

∞
Rg (τ ) = − ∫ g ∗ (− δ ).g (τ − δ )dδ
−∞

Hence Rg (τ ) can be rewritten as convolution of function g (− τ ) and g (τ ) as follows

∗

R g (τ ) = − g ∗ (− τ ) ∗ g (τ )

Now g (− τ ) ∗ g (τ ) ↔ G ( f ) × G( f ) = G( f )
∗ ∗ 2

Hence,

Rg (τ ) ↔ G( f )
2
NOTE: Proof of g ∗ (− t ) ↔ G ∗ ( f )

∞
g (t ) = ∫ G( f )exp(+ j 2πf )df
−∞
∞ ∞
g (t ) = ∫ G ( f )[exp(+ j 2πft )] df = ∫ G∗ ( f )exp(− j 2πft )df
* ∗ ∗

−∞ −∞

t → −t

∞
g (− t ) = ∫ G* ( f )exp(+ j 2πft )df ⇒ g ∗ (− t ) ↔ G∗ ( f )
∗

−∞

Auto Correlation of Periodic Signal

Autocorrelation of non-periodic power signal is given as

T
Rg (τ ) = LimT →∞ ∫ g(t )× g (t −τ )dt
1 ∗
2T (23)
−T

When, g(t ) is periodic, the integrand is also periodic and hence we can have time
duration of integration over one period. Thus for a periodic signal, the above result
reduces to

T0 2
Rgp (τ ) = ∫ g p (t )× g p (t −τ )dt
1 ∗
T0 (24)
−T0 2

Note: The autocorrelation function Rgp (τ ) of a periodic signal has properties similar to
those of an energy signal.
∗
(1) Rgp (τ ) = Rgp (− τ ) (Conjugate Symmetry)
(2) Rgp (τ ) ≤ Rgp (0) for all τ
(3) Rgp (τ ) ↔ S gp ( f ) Power spectral Density
 (4) The autocorrelation function is periodic with the same period as the periodic
signal itself

Rgp (τ ) = Rgp (τ ± nT0 ), n = 1,2,..... ,

For the meaning of conjugate symmetry, refer to property 1 of autocorrelation.

Cross-Correlation Function

The auto correlation provides similarity between a signal and it s time-delayed signal
whereas, cross-correlation is a measure of the similarity between a signal and the time
delayed version of second signal.

Let g1 (t ) and g2 (t ) are two complex-valued finite energy signals. The cross-correlation is

∞
R12 (τ ) = ∫ g1(t)g2 (t −τ )dt
∗
(25)
−∞

The energy signals g1 (t ) and g2 (t ) are said to be orthogonal over the entire time interval
if R12 (0) = 0

∞ ∗

∫ g1 (t ). g 2 (t )dt = 0
−∞
(26)

∞ ∗

Now, R21 (τ ) = ∫ g 2 (t ) g 1 (t − τ )dt (27)

−∞

Hence, from (25) and (27), we have

∗
R12 (τ ) = R21 (− τ ) (28)

Unlike the convolution, correlation is not in general commutative, that is, R12 (τ ) ≠ R21 (τ )
whereas g1 (t ) ∗ g2 (t ) = g 2 (t ) ∗ g1 (t ) .
 [
One-to-one mapping ⇒ pdf of E X (t k )X (t i ) t d = pdf of t d . ]

Hence,

⎡ t k − ti ⎤
E [ X (t k )X (t i )] = A 2 ⎢1 − ⎥, t k − ti < T
⎣ T ⎦

If t k − t i = τ

⎛ τ ⎞
RX (τ ) = A 2 ⎜⎜1 − ⎟
⎟ τ <T
⎝ T ⎠

=0 τ ≥T

FIGURE

Cyclo-stationary Process

When a stochastic processes have statistical averages and

autocorrelation that are periodic (instead of constant as we have seen so far), then, the

process is called cyclo-stationary process. Consider following random process

∞
X (t ) = ∑ a g (t − nT ) = ... + a g (t + 2T ) + a g (t + T ) + a g (t ) + a g (t − T ) + a g (t − 2T ) + ...
n −2 −1 0 1 2
n = −∞
Note that an is a random variable and g (t ) is deterministic signal hence, the ensemble

average of process X (t ) is obtained as

⎡ ∞ ⎤
E[X (t )] = E ⎢ ∑ an g (t − nT )⎥
⎢⎣n = −∞ ⎥⎦

∞
= ∑ E[a ]g (t − nT )
n = −∞
n

∞
= m0 ∑ g (t − nT )
n = −∞
(i)

Hence, we see that mean is not constant but is varying with time. Infact, It is periodic

with period T

We see that mean is periodic with period T .The plot of E[ X (t )] is shown in Figure 1.9

(b) for the pulse g(t) shown in Figure 1.9 (a). We note that mean is not constant but is

varying with time periodically.

 Figure 1.9

The Autocorrelation function of X (t ) is obtained a

E[ X (t )X (t + τ )] = R X (t + τ , t )

⎡ ∞ ∞
⎤
= E ⎢ ∑ a n g (t − nT ) ∑ a m g (t + τ − mT )⎥
⎣ n = −∞ m = −∞ ⎦
⎡ ∞ ∞ ⎤
= E ⎢ ∑ ∑ a n a m g (t − nT )g (t + τ − mT )⎥
⎣ n = −∞ m = −∞ ⎦
= ∑∑ E [a n a m ]g (t − nT )g (t + τ − mT )]
∞ ∞
= Ra (m − n ) ∑ ∑ g (t − nT )g (t + τ − mT )
n = −∞ m = −∞

Hence, we see that Auto-correlation function is also a function of t and is periodic with

period T .

Hence, such processes whose stochastic averages such as mean and auto-

correlation are also periodic are called cyclo-stationary process or periodically

stationary. Since, auto-correlation function depends on both t & τ , its frequency domain

representation requires the use of a two- dimensional Fourier Transform.

Since, it is highly desirable to characterize such process by its PSD. Hence, first

we estimate the dependence of Autocorrelation on two time-parameter t ,τ by averaging

it over a single period

+T 2

R X (τ ) = R X (t + τ , t )dt
1
T −T∫ 2
Now, taking the Fourier transform of RX (τ ) yields the average power spectrum density.

Hence,

∞
S( f ) = ∫ R (τ )exp (− j 2πfτ )dτ
X
−∞

* →The reason why this is periodic with period T is that when the m ≠ n summation

term will contribute to zero. When m = n , then, product term is the product of two pulse

shifted by τ when τ < T . Note that even for τ > T , product term will result in zero.

Hence, for τ < T ,the multiplication term will yield some kind of another definite pulse,

then situation will just like (i) for periodic mean.