Download Probability

2.5 Hitting Times
2.5.1 Hitting Times
Sometimes we want to know things like
What is the probability that the system will have been in a certain state by a certain time?
or
What is the probability that the system will be in a certain state for the first time at a certain time?
Here is an example of the first type of question.
Example 1. (see Example 1 of section 2.1) At the start of each day an office copier is checked and its
condition is classified as either good (1), poor (2) or broken (3). Suppose the transition matrix from one
state to another in the course of day is
(1)
 0.8
P =  0
 0.32
0.06
0.4
0.08
0.14 
0.6 
0.6 
We might want to know the following
Suppose the copier was in good condition at the start of the day today. What is the probability that
the copier will be broken at the start of the day in at least one of the next five days?
There are two ways one can represent this probability symbolically. One way is with the random variables
Xn that define the process. Then the probability that the copier will be broken at the start of the day in at
least one of the next five days given that it is in good condition today is
Pr{X1 = 3 or X2 = 3 or X3 = 3 or X4 = 3 or X5 = 3 | X0 = 1}
Another way to represent this probability symbolically is with what is called the hitting time (or first
passage time) for a state. If j is a state then
(2)
T(j) = first time (greater than or equal to one) that the system is in state j
= hitting time for state j
= time to reach j (or return to j if you start there)
= first passage time for state j
Note that T(j) is a random variable. It depends on the values of X0, X1, … If the system never returns to
state j then we define T(j) = . In terms of hitting times the probability that the copier will be broken at the
start of the day in at least one of the next five days given that it is in good condition today is
2.5.1 - 1
Pr{T(3)  5 | X0 = 1}
Questions regarding hitting times occur frequently, so notation has been developed for their probabilities.
We let
(3)
Fij(n) = Pr{T(j)  n | X0 = i}
If we fix i and j and let n vary then Fij(n) is the cumulative distribution of T(j) given that we start in state i. In
terms of Fij(n) the probability that the copier will be broken at the start of the day in at least one of the next
five days given that it is in good condition today is F13(5). There are various ways one can compute the Fij(n).
One way is to create another Markov chain which is the same as the original one except that once we reach
state j we stay there. In Example 1 if we
0.4
want to compute the probability that the
0.8
copier will be broken at the start of the
day in at least one of the next five days
Good condition now
and never broken
then we would create another Markov
0.06
Poor condition now and
never broken
chain that is the same as the original one
except once the copier is broken it stays
broken. It doesn't mean the copier is not
0.6
0.14
fixed in reality. The modified state means
the copier is or has been broken some
Broken now or at some
prior time
time prior to the present. The transition
1
diagram for this modified Markov chain is
at the right and the transition matrix is
(4)
Q =
 0.8
 0
 0
0.06
0.4
0
0.14 
0.6 
1 
If we call this new Markov chain Yn, then Pr{T(3)  5 | X0 = 1} = Pr{Y5 = 3 |Y0 = 1} = (Q5)13. In section
2.5.2 we computed Q5 which is
Q5 =
 0.328
 0
 0
0.048
0.010
0
0.625 
0.990 

1
So the probability that the copier will be broken at the start of the day in at least one of the next five days
given that it is in good condition today is (Q5)13 = 0.625.
Using the eigenvalue technique discussed in section 2.2.1, one can find a formula for Qn. This is done in
section 2.5.2. Using that, one obtains Pr{T(3)  n | X0 = 1} = F13(n) = (Qn)13 = 1 – 0.15(0.4n) – 1.15(0.8n).
As expected, Pr{T(3)  n | X0 = 1} = F13(n)  1 as n  .
There is one case where we have to make a slight modification in the above procedure. This is when the
starting state and ending state are the same.
2.5.1 - 2
Example 2. Consider the office copier in Example 1. Suppose the copier was broken at the start of the day
today. What is the probability that the copier will be broken again at the start of the day in at least one of
the next five days? In terms of the hitting time T(3) we want to know Pr{T(3)  5 | X0 = 3}.
It is not (Q5)33 with Q as in (4). The reason is that (Q5)33 = 1 since one is stuck in state 3 once one gets
there. There are several ways to determine Pr{T(3)  5 | X0 = 3}.
One way is use the conditional additivity formula (Proposition 1 in section 1.3.6) to divide
Pr{T(3)  5 | X0 = 3} into three pieces depending on the first transition, i.e.
Pr{T(3)  5 | X0 = 3} = Pr{T(3)  5, X1 = 1 | X0 = 3} + Pr{T(3)  5, X1 = 2 | X0 = 3}
+ Pr{T(3)  5, X1 = 3 | X0 = 3}
Consider the first term on the right of the equal sign. Using the conditional intersection formula
(Proposition 2 in section 1.3.6) one has
Pr{T(3)  5, X1 = 1 | X0 = 3} = Pr{T(3)  5 | X1 = 1, X0 = 3} Pr{ X1 = 1 | X0 = 3}
= Pr{T(3)  5 | X1 = 1, X0 = 3} p31
Note that {T(3)  5} =

(i1,... i5)  E
{ X1 = i1, ..., X5 = i5} where E = {(i1, ..., i5): at least one of the ij is 3}. In the
same fashion {T(3)  5, X1 = 1, X0 = 3} =

(i2,... i5)  F
{ X2 = i2, ..., X5 = i5, X1 = 1, X0 = 3} where
F = {(i2, ..., i5): at least one of the ij is 3}. So
Pr{T(3)  5, | X1 = 1, X0 = 3} = Pr{

(i2,... i5)  F
{ X2 = i2, ..., X5 = i5 | X1 = 1, X0 = 3}
Using the extended Markov property ((12) in Proposition 1 in section 2.1.3), this can be written as
Pr{T(3)  5, | X1 = 1, X0 = 3} = Pr{
= Pr{

(i2,... i5)  F

(i1,... i4)  F
{ X1 = i2, ..., X4 = i5 | X0 = 1}
{ X1 = i1, ..., X4 = i4 | X0 = 1}
However, by the same argument one has
Pr{T(3)  4 | X0 = 1} = Pr{

(i1,... i4)  F
{ X1 = i1, ..., X4 = i4 | X0 = 1}
So
Pr{T(3)  5, | X1 = 1, X0 = 3} = Pr{T(3)  4 | X0 = 1} = (Q4)13
and
2.5.1 - 3
Pr{T(3)  5, X1 = 1 | X0 = 3} = p31(Q4)13
By the same argument
Pr{T(3)  5, X1 = 2 | X0 = 3} = p32(Q4)23
For Pr{T(3)  5, X1 = 3 | X0= 3} one has
Pr{T(3)  5, X1 = 3 | X0 = 3} = Pr{T(3)  5 | X1 = 3, X0 = 3} Pr{ X1 = 3 | X0 = 3}
= Pr{T(3)  5 | X1 = 3, X0 = 3} p33 = p33 = p33(Q4)33
This is because Pr{T(3)  5 | X1 = 3, X0 = 3} = 1 since T(3)  5 if X1 = 3. Putting this altogether gives
Pr{T(3)  5 | X0= 3} = p31(Q4)13 + p32(Q4)23 + p33(Q4)33 = (PQ4)33
When one computes PQ4 one obtains
PQ4 =
 0.328
 0
 0.131
0.625 
0.990 
0.848 
0.048
0.010
0.020
So the probability that the copier will be broken at the start of the day in at least one of the next five days
given that it is broken today is (PQ4)33 = 0.848.
Another way to look at this
0.4
computation is as follows. We make
0.8
an additional modification to the
Markov chain above by adding a
Good condition now
and never broken = 1
special state (let's number it state 4)
0.06
Poor condition now and
never broken = 2
corresponding to being broken to
begin with. The transition diagram
0.08
for this modified Markov chain is at
0.32
0.14
the right and the transition matrix is
 0.8
0
U =  0

 0.32
0.06
0.4
0
0.08
0.14
0.6
1
0.6
0
0
0
0
Start broken = 4
0.6
Broken now or at some
prior time = 3
0.6
In terms of U the probability that the copier will be broken at the start of the day in at least one of the next
five days given that it is broken today is (U5)43. If one computes U5 one obtains
 0.328
0
U5 = 
 0
 0.131
0.048
0.010
0
0.020
0.625
0.990
1
0.848
0
0
0
0
2.5.1 - 4
1
So the probability that the copier will be broken at the start of the day in at least one of the next five days
given that it is broken today is (U5)43 = 0.848.
Here is an example of finding the probability that the system will be in a certain state for the first time at a
certain time.
Example 3. Consider the office copier in Example 1. What is the probability that the copier will have
broken for the first time five days from now given that it is in good condition today.
In symbols this is
Pr{X1  3, X2  3, X3  3, X4  3, X5 = 3 | X0 = 1} = Pr{T(3) = 5 | X0 = 1} = f13(5)
where
fij(n) = Pr{T(j) = n | X0 = i}
(5)
If we fix i and j and let n vary then fij(n) is the pmf of T(j) given that we start in state i. There are various
ways one can compute the fij(n). One way is simply
 Fij(n) - Fij(n-1)
 Fij(1)
fij(n) = 
if n  2
if n = 1
which is the general relation between the pmf and the cumulative distribution function for a random
variable.
So, in Example 3, one has
0.4
Pr{T(3) = 5 | X0 = 1} = f13(5) = F13(5) - F13(4)
0.8
= (Q5)13 - (Q4)13 = 0.0919
Another way to answer this is to make
Good condition now
and never broken = 1
another modification of the Markov chain
0.06
Poor condition now and
never broken = 2
in Example 1. We add another state which
0.08
we label "Broken on the first time on some
previous day". Let's call this state 4. We
go to this state after we hit the Broken
0.6
0.14
1
state, state 3. The transition diagram for
this modified Markov chain is at the right
and the transition matrix is
 0.8
0
S =  0

 0
0.06
0.4
0
0
0.14
0.6
0
0
Broken on the first time
on some previous day
=4
Broken for first time
now = 3
1
0
0
1
1
If we call this new Markov chain Zn, then Pr{T(3) = 5 | X0 = 1} = Pr{Z5 = 3 |Z0 = 1} = (S5)13 = 0.0919. In
fact, one has
2.5.1 - 5
 0.328
0
S =  0

 0
0.0476
0.0102
0
0
0.0919
0.0154
0
0
0.533
0.974
1 
1 
2.5.1 - 6