STAT 333 - Problem Set 8
Exercises marked with an asterisk ∗ are considered to be recommended exercises. The non-
asterisk exercises are intended to serve as extra practice exercises.
Section 3.5
Exercise 3.5.1. Suppose that the TPM for a DTMC can be expressed as
Q R
P = .
0 I
Use mathematical induction to show that
n Pn−1 i
(n) Q i=0 Q R
P = , n ∈ Z+ .
0 I
Exercise 3.5.2.∗ Consider a DTMC {Xn , n ∈ N} with TPM
0 1 2 3 4
0 0 0.2 0 0.8 0
1
0.8 0 0.2 0 0
P = 2 0 0.8 0
0 0.2
.
3 0
0 0 1 0
4 0 0 0 0 1
Suppose that X0 = 2 with probability 1.
(a) What is the probability that the DTMC ultimately ends up in state 3?
(b) Reorder the states of this DTMC and apply the result of the Gambler’s Ruin Problem
to answer part (a).
(c) How many transitions, on average, does it take to end up in one of the absorbing
states?
(d) Determine the mean number of visits to state 0 prior to absorption.
(e) Determine the probability that the DTMC ever makes a visit to state 0.
Exercise 3.5.3.∗ Consider a DTMC on the state space S = {1, 2, 3, 4} with TPM of the
form
1 2 3 4
1 0.4 0.3 0.2 0.1
2 0.2 0.2 0.2 0.4
P = .
3 0.25 0.25 0.5 0
4 0.2 0.1 0.4 0.3
Suppose that the DTMC begins in state 1 with probability 1.
1
(a) Calculate the probability that state 3 is encountered before state 4.
(b) Calculate the mean number of transitions until either state 3 or state 4 is entered.
Exercise 3.5.4.∗ Consider a DTMC with TPM
0 1 2 3 4
0 0.3 0.4 0.2 0 0.1
1
0.4 0.1 0.1 0.1 0.3
P = 2
0 0 0.8 0.2 0 .
3 0
0 0.5 0.5 0
4 0 0 0 0 1
Suppose that the DTMC is equally likely to begin in state 0 or state 1. What is the
probability that the DTMC ultimately ends up in state i, i = 2, 3, 4?
Exercise 3.5.5.∗ Consider a DTMC {Xn , n ∈ N} with TPM
0 1 2 3
0 0 p 0 1−p
1 q 1−q 0 0
P = ,
2 1−p
0 p 0
3 0 p 1−p 0
where 0 < p < 1 and 0 ≤ q ≤ 1. Suppose that the DTMC is equally likely to begin at time
0 in either state 2 or state 3.
(a) Assuming that q ̸= 0, prove that a limiting distribution exists for this DTMC.
(b) Assuming that q = p, determine the limiting probabilities (in terms of p) of this DTMC.
(c) Assuming that q = 0, determine an explicit expression (in terms of p) for the mean
number of times state 0 is visited prior to reaching state 1.
(d) Let Z represent the number of transitions needed to reach either state 0 or state 1,
whichever happens first. Determine E[Z].
(e) Find the pmf of Z from part (d) and then use it to determine E[Z].
2
Exercise 3.5.6. Consider a gambler who, at each play of a game, rolls a fair six-sided die.
Assume that successive plays of the game are independent of each other. If the gambler
rolls a 5 or a 6, one unit is won. If the gambler rolls a 2, 3, or 4, no units are won or lost.
However, if a 1 is rolled, then the gambler loses all of their units and goes bankrupt. The
gambler will continue playing the game until their fortune either reaches 0 units or N units
(N ∈ Z+ ), where N represents the jackpot. For n ∈ N, define Xn as the gambler’s fortune
after the nth play of the game. Suppose that the gambler starts with one unit (i.e., X0 = 1
with probability 1).
(a) Specify the form of the TPM for the DTMC {Xn , n ∈ N}.
(b) What is the probability that the gambler wins the jackpot before going bankrupt?
(c) Assume now that the jackpot is N = 3 units. How many times should the gam-
bler expect to have their fortune be at 2 units before they stop playing? How many
games should the gambler expect to play in total before going bankrupt or winning
the jackpot?
(d) Assuming again that N = 3, what is the probability that the gambler’s fortune never
increases higher than 1 unit?