Tutorial Sheet-6

1. Classify the following stochastic process based on the state space and index set. The number
of customers in queue in front of an ATM for at the end of each hour of a day.

(i) Discrete time discrete state stochastic process.

(ii) Discrete time continuous state stochastic process.
(iii) Continuous time discrete state stochastic process.
(iv) Continuous time continuous state stochastic process.

2. Classify the following stochastic process based on the state space and index set. Number of
vehicles in parking of a shopping mall at any time during the day.

(i) Discrete time discrete state stochastic process.

(ii) Discrete time continuous state stochastic process.
(iii) Continuous time discrete state stochastic process.
(iv) Continuous time continuous state stochastic process.

3. On any given day Nikita is cheerful (C), normal (N) or depressed (D). If she is cheerful today,
then she will be C, N or D tomorrow with probabilities 0.5, 0.4, 0.1, respectively. If she is
feeling so-so today, then she will be C, N or D tomorrow with probabilities 0.3, 0.4, 0.3. If she
is glum today, then she will be C, N, or D tomorrow with probabilities 0.2, 0.3, 0.5. Let Xn
denote Nikita’s mood on the nth day.

(a) Verify whether {Xn : n ≥ 1} is a Markov chain or not.

(b) Write the one-step transition probability matrix and draw the state-transition diagram.
(c) Does the stationary distribution exist? If yes, find it.
(d) Does the limiting distribution exist? If yes, find it.
(e) If Nikita is depressed today, then − what percentage chance is there that she will be
cheerful day after tomorrow?

4. Answer the following questions for the set of problems given below.

(a) Draw the state-transition diagram.

(b) Is it an irreducible Markov chain?
(c) Classify the states as recurrent/transient.
(d) Is there any positive recurrent/null recurrent state(s)?
(e) Determine the periodicity of all states.

1
 (f) Is it an aperiodic Markov chain?
(g) Is there any ergodic state(s)?
(h) Is it an ergodic Markov chain?
(i) Does there exist stationary distribution(s)? If yes, determine it.
(j) Find the two-step transition probability matrix.
(k) Find P (X2 = 1).

(i) Let {Xn ; n ≥ 0} be a Markov chain with state space {0, 1, 2} and one-step transition proba-
bility matrix
 
0.75 0.25 0
P (1) = 0.25 0.5 0.25 ,
 

0 0.75 0.25

where the initial distribution is given by P (X0 = i) = 1/3, for all i = 0, 1, 2.

(ii) Let {Xn ; n ≥ 0} be a Markov chain with state space {1, 3, 5, 6, 8} and one-step transition
probability matrix
 
1 0 0 0 0
 0 1/4 3/4 0 0 
 
(1)
 
P =   0 1/2 1/2 0 0 ,
1/4 1/4 0 1/4 1/4
 

0 0 0 1/2 1/2

where the initial distribution is given by P (X0 = 1) = 1/5, P (X0 = 3) = 0, P (X0 = 5) = 2/5,
P (X0 = 6) = 2/5 and P (X0 = 8) = 0.

(iii) Let {Xn ; n ≥ 0} be a Markov chain with state space {1, 2, 3} and one-step transition proba-
bility matrix
 
0 1 0
(1)
P = 1/2 0 1/2 ,
 

0 1 0

where the initial distribution is given by P (X0 = 1) = 1/10, P (X0 = 2) = 6/10 and P (X0 =
3) = 3/10.

(iv) Let {Xn : n ≥ 0} be a Markov chain with state space {1, 2, 3, 4, 5} and one-step transition

2
 probability matrix
 
0.5 0 0 0.5 0
 0 0.6 0 0 0.4
 

P (1)
 
= 
0.3 0 0.7 0 0,
0 0 1 0 0
 

0 1 0 0 0

where the initial distribution is given by P (X0 = 1) = 1/12, P (X0 = 2) = 2/12, P (X0 =
3) = 3/12, P (X0 = 4) = 4/12 and P (X0 = 5) = 2/12.

(v) Let {Xn : n ≥ 0} be a Markov chain with state space {0, 1, 2, 3} and one-step transition
probability matrix matrix
 
0.8 0 0.2 0
 
(1)
0 0 1 0
P = 
 ,
1 0 0 0
0.3 0.4 0 0.3

where the initial distribution is given by P (X0 = 0) = 1/7, P (X0 = 1) = 2/7, P (X0 = 2) = 0,
P (X0 = 3) = 3/7 and P (X0 = 4) = 1/7.

5. Suppose that the chance of rain tomorrow depends on previous weather conditions only
through whether or not it is raining today and not on past weather conditions. Suppose also
that if it rains today, then it will rain tomorrow with probability α; and if it does not rain
today, then it will rain tomorrow with probability β. Let Xn be the weather on the n-th day,
where 1, 2, 3, . . .

(a) Verify whether {Xn : n ≥ 1} is a Markov chain or not.

(b) Write the one-step transition probability matrix and draw the state-transition diagram.
(c) Does the stationary distribution exist? If yes, find it.
(d) Does the limiting distribution exist? If yes, find it.
(e) Suppose that there is rain on 98-th day. Find out the probability that there will on
100-th day.

6. Consider a communications system that transmits the digits 0 and 1. Each digit transmitted
must pass through several stages, at each of which there is a probability p that the digit
entered will be unchanged when it leaves. Let Xn denote the digit entering the n-th stage,
n = 0, 1, 2, . . .

(a) Verify whether {Xn : n ≥ 0} is a Markov chain or not.

3
 (b) Write the one-step transition probability matrix and draw the state-transition diagram.
(c) Find out the n-th step transition probability matrix, and then calculate the limiting
distribution.
(d) Does the stationary distribution exist? If yes, find it.
(e) Does the limiting distribution and the stationary distribution equal?
(f ) What is the probability that 29-th stage entering digit is 1?
(g) It is known that the digit entered in 10-th stage was 0. Then find out the probability
that 123-th stage entering digit is 1.

7. Assume that a man’s profession can be classified as professional, skilled labourer, or unskilled
labourer. Assume that, of the sons of professional men, 80 percent are professional, 10 percent
are skilled labourers, and 10 percent are unskilled labourers. In the case of sons of skilled
labourers, 60 percent are skilled labourers, 20 percent are professional, and 20 percent are
unskilled. Finally, in the case of unskilled labourers, 50 percent of the sons are unskilled
labourers, and 25 percent each are in the other two categories. Assume that every man has
at least one son, and form a Markov chain by following the profession of a randomly chosen
son of a given family through several generations.

(a) Write one-step transition probability matrix.

(b) Find the probability that a randomly chosen grandson of an unskilled labourer is a
professional man.
(c) What percentage chance is there that a successor of a randomly chosen labourer, ater
n-th (where n is sufficiently large) generation, will be a skilled labourer?

8. An auto insurance company classifies its customers in three categories: poor, satisfactory and
preferred. At the end of one year, 20% poor moves to preferred and 10% preferred moves
to poor; 30% of the customers in the poor category become satisfactory; 30% of those in
the satisfactory category moves to preferred, while 10% become poor; 20% of those in the
preferred category are downgraded to satisfactory.

(a) Model this problem as a Markov chain.

(b) Write the one-step transition probability matrix for the model.
(c) What is the limiting fraction of customers in each of these categories?
(d) Company’s record shows that there were initially 20% poor customers, 30% satisfactory
customers and 50% preferred customers. What percentage customers will be in the
preferred category after the end of second year?
(e) Suppose that a customer is in satisfactory category at the end of fifteen years. Then
find the probability that he/she will be in poor category after just two years.

4
 9. Suppose that stocks going up tomorrow is dependent on whether it increased today and
yesterday. In particular, if the stock has increased in past two days, it will increase tomorrow
with a probability of 0.9. If the stock increased today but decreased yesterday, then it will
increase tomorrow with probability 0.6. If the stock decreased today but increased yesterday,
then it will increase tomorrow with probability 0.5. Finally, if the stock decreased for the
past two days, then it will increase tomorrow with probability 0.3.

(a) Model this problem as a Markov Chain.

(b) Find one-step transition probability matrix.
(c) Is it an ergodic Markov Chain?
(d) Does the stationary distribution exist? If yes, find it.

10. Consider an experiment of mating rabbits. We watch the evolution of a particular gene that
appears in two types, G or g. A rabbit has a pair of genes, either GG (dominant), Gg (hybrid,
the order is irrelevant; so gG is the same as Gg) or gg (recessive). In mating two rabbits, the
offspring inherits a gene from each of its parents with equal probability. Thus, if we mate a
dominant (GG) with a hybrid (Gg), the offspring is dominant with probability 1/2 or hybrid
with probability 1/2. Start with a rabbit of given character (GG, Gg, or gg) and mate it with
a hybrid. The offspring produced is again mated with a hybrid, and the process is repeated
through a number of generations, always mating with a hybrid.

(a) Model this process as a Markov Chain.

(b) Find one-step transition probability matrix.
(c) Show that the n-th step transition probability matrix given by
3 1


+ 2n−2 − 1 2n−1 + 2n−2 − 1


2 2
P (n) = 2−n  2n−2 2n−1 2n−2 .
 
1 n−2 − 1 3
2n−1 + 2n−2 − 1



2 + 2 2

(d) Does the stationary distribution exist? If yes, find it.

(e) Does the limiting distribution exist? If yes, find it.
(f) What fraction of rabbit population will have hybrid character after m-th generation,where
m is sufficiently large?
(g) If the offspring in the 100-th generation has dominant character, then − what is the
chance that the offspring in the 143-th generation will have recessive character?