Download Basic Principles

4
Continuous Time Markov Processes
4.1 Basic Principles.
Many systems in the real world are more naturally modeled in continuous time as opposed to discrete time.
For example, if we are modeling the number of customers waiting for service at a bank, then it might be
more appropriate for the model to describe how the number of customers changes with each instant in time
rather than every five minutes. A continuous time stochastic process is a mathematical model of a system
that can be in one of a variety of states at any particular time t, where t can be any non-negative real
number. This is in contrast to a Markov chain which models a system evolving in discrete time, so the time
variable n only assumes integer values. We shall restrict our attention to the case where the states are still
discrete and usually finite. Often the states will be numbered i = 1, 2, ..., N where N is the number of states.
If we let X(t) denote the state at time t, then a continuous-time stochastic process can be described by giving
the random variable X(t) for each t in some interval. Problem 1 in Section 2.1.1 considered a Markov chain
model of the condition of an office copier. In that case it was modeled in discrete time by only considering
its condition at the start of each day. Let's look at a continuous time model of the same situation.
Example 1. The condition of an office copier at any time is one of the following.
1.
good - 1
2.
poor - 2
3.
broken - 3
state
3.5
Let t be time in days measured from
broken 3.0
the start of the work day on March 1.
2.5
Assume a ten hour work day running
from 8 a.m. to 6 p.m. For example
poor 2.0
t = 0.6 would be 2 p.m. on March 1
1.5
while t = 1.3 would be 11 a.m. on
March 2. Let X(t) be the state at time
good 1.0
t. At the right is a graph showing how
0.5
the state of the copier might vary over
the first eight days.
t days
2
0
2
4
6
8
In order to describe this mathematically, one needs to define the random variables X(t) for each t. One
possibility would be to give the joint probability mass functions of the X(t) for all choices of finitely many t.
For example, that would allow one to find directly probabilities like Pr{X(0) = 1 & X(1.4) = 2} (the
probability that the copier is good condition at the start of the day on March 1 and in poor condition at 12
noon on March 2).
4.1 - 1
A somewhat less ambitious thing to do would be to give the transition probablilies
Pr{X(t) = j | X(0) = i} = pij(t)
for each i and j and each t > 0. Even though we shall see how to calculate these probabilities in section 4.5,
it is usually difficult to give these at the start of the modeling process. It most real world situations it is
more natural to describe the transitions from one state to another. One way to do this would be as follows.
Let
Ti = time until the next transition after the system enters state i
Zi = the following state after the next transition after the system enters state i
One would specify the joint distribution of Ti and Zi and also suppose that these variables are independent
of the past history of the system. This along with the state the system enters at time t = 0 would define the
X(t), although calculating the joint probability mass functions of the X(t) for all choices of finitely many t
would usually be difficult. Let's look at a specific way of doing this in Example 1.
Example 1 (continued). Suppose
T1 = time the copier remains in good condition after it enters this state
1
= an exponential random variable with mean = 5 days / transition
v1
1
(or rate v1 = = 0.2 transition per day)
5
Z1 = next state after the copier leaves the good condition state
r12 = Pr{Z1 = 2} = probability copier is in poor condition after it is good = 1/4
r13 = Pr{Z1 = 3} = probability copier is broken after it is good = 3/4
T2 = time the copier remains in poor condition after it enters this state
1
= an exponential random variable with mean = 2 days / transition
v2
1
(or rate v2 = transition per day)
2
Z2 = next state after the copier leaves the poor condition state
r21 = Pr{Z2 = 1} = probability copier is in good condition after it is poor = 1/25
r23 = Pr{Z2 = 3} = probability copier is broken after it is poor = 24/25
T3 = time the copier remains broken after it enters this state
1
3
= an exponential random variable with mean = 5/3 days (or rate v3 = transition per day)
v3
5
Z3 = next state after the copier leaves the broken condition state
r31 = Pr{Z3 = 1} = probability copier is in good condition after it is broken = 4/5
r32 = Pr{Z3 = 2} = probability copier is in poor condition after it is broken = 1/5
The assumption that the Ti are exponential random variables and the Ti and Zi are all independent is
equivalent to the stochastic process X(t) having the Markov property which is discussed in the next section.
We shall assume this is the case from now on. Let
4.1 - 2
1
mean of Ti
(1)
vi = rate at which system is leaving state i =
(2)
rij = probability that the next state is j after making a transition out of state i = Pr{Zi = j}
(3)
qij = rijvi = rate at which the system is making transitions from state i to state j if i  j
(4)
qii = - vi = - (rate at which system is making transitions out of state i)
Note that the units of vi and qij are transitions per unit time. One way to interpret vi and qij is as follows.
Suppose we have a large number M of identical systems that all start out in state i at time t = 0. Let’s look
at these systems for a small time interval h. Then approximately viMh will make transitions out of state i
with rijviMh = qijMh going to state j for each j different from i.
The matrix
21
q12  q1N
q22  q2N
N1
qN2  qNN
q
q

q
11
(5)
Q =




21 2
b12v1  b1Nv1
- v2  b2Nv2
N1 N
bN2vN  - vN
-v
b v

b v
1
=




= generator matrix
called the generator matrix, contains the transition rates and, as we shall see in sections 4.4 and 4.5, plays
an important role in constructing the actual probabilities pij(t) of transitions from one state to another in a
given time.
Note that the rows of Q sum to zero and the probabilities of transitions from one state to another are the
ratios of the elements of Q to the absolute value of the diagonal element of that row.
(6)
vi = qi1 +  + qi,i-1 + qi,i+1 +  + qi,n
rij =
qij
vi
Example 1 (continued). In Example 1
v1 =
1
= 0.2
5
q12 =
1 1

= 0.05
4 5
v2 =
1
= 0.5
2
q21 =
1 1

= 0.02
25 2
q23 =
24 1

= 0.48
25 2
v3 =
1
= 0.6
5/3
4 3

= 0.48
5 5
q32 =
1 3

= 0.12
5 5
q31 =
q13 =
3 1

= 0.15
4 5
For example, suppose we have M = 1000 identical copiers that all start out in poor condition (state i = 2) at
time t = 0. Let’s look at these copiers for a small time interval h = 0.1 days. Then approximately
v2Mh = 50 will make transitions out of the poor condition state with 2 going to good condition and 48 going
to broken.
4.1 - 3
The generator matrix is
 - 0.2
Q =  0.02
 0.48
0.05
0.05
0.15 
- 0.5
0.48 
0.12 - 0.6 
Good - 1
Poor - 2
0.02
0.15
0.48
0.48
The qij are often displayed in a transition diagram
0.12
with an arrow from state i to state j labeled by qij.
Broken - 3
For Example 1 it is at the right.
There is also a Markov chain associated with the Markov process called the embedded Markov chain where
one ignores time and just follows the transition from one state to the next. Its transition matrix R contains
the rij, i.e.
R =
21
r12  r1N
0  r2N
N1
rN2 
0
r

r
1
In Example 1 one has R = 25
4
5
0
1
4
0
1
5
0





.
0 
3
4
24
25
In the above example it was easiest to measure the mean times vi in each state and the probabilities rij of
transitions from one state to another. From these we calculate the transition rates qij = rijvi from one state to
another. In some situations it is easier to measure the qij directly and calculate the vi and rij from the qij
using (6).
Example 2. The time between customer arrivals at Sam's Barber Shop is an exponential random variable
with mean 1/2 hour. However, customers will only wait if there is no one already waiting. The time it takes
Sam to give a haircut is an exponential random variable with mean 1/3 hour. Assume all times are
independent of all the other times. Let Nt be the number of customers in the shop including the one
currently getting a haircut. Thus Nt can either be 0 (no customers in the shop), 1 (just the customer getting a
haircut in the shop) or 2 (the customer getting a haircut and one waiting in the shop). Let
 = 2 / hr = rate of customer arrivals
 = 3 / hr = rate at which customers get haircuts
Then
q01 = q12 =  = 2
q10 = q21 =  = 3
4.1 - 4
q02 = q20 = 0
R =
=
q01
 - (q01 + q02)
q10
- (q10 + q12)


q20
q21
q02

q12
 =
- (q20 + q21) 
-
 
 0
-2

- ( + )

0
2
-5
3
  =  3
 0
-
 - ( + 0)



0

- ( + )



- (0 + ) 
0

0
2
-3
When one describes a Markov process by giving the rates qij of transitions from states i to states j then one
can calculate the rates of transitions out of each state and probabilities of going from one state to another
using the formulas (6) above, i.e.
vi = qi1 +  + qi,i-1 + qi,i+1 +  + qi,n
rij =
qij
vi
In doing this one is using the following propositions.
Proposition 1. (a) If S and T are independent exponential random variables with means 1/ and 1/
respectively and R = min(S, T) then R is an exponential random variable with mean 1/( + ). (b) More
generally, if T1, ..., Tn are independent exponential random variables with means 1/1, …, 1/n respectively
and R = min(T1, ..., Tn) then R is an exponential random variable with mean 1/(1 +  + n).
Proof. S and T have density functions fS(s) = e-s and fT(t) = e-t for s  0 and t  0 and fS(t) = fT(t) = 0 for
s < 0 and t < 0. It follows that S and T have survival functions Pr{S > s} = e-s and Pr{T > t} = e-t for s  0
and t  0. So the survival function of R = min(S, T) is
Pr{R > r} = Pr{ min(S, T) > r} = Pr{ S > r, T > r} = Pr{ S > r} Pr{T > r}
= e-re-r = e-(+)r
From this it follow that the density function of R is (+)e-(+)r and hence R is exponential with mean  + .
(b) follows by applying (a) n – 1 times. //
Proposition 2. If S and T are independent exponential random variables with means 1/ and 1/
respectively then
(7)
Pr{S < T} =

+
More generally, if T1, ..., Tn are independent exponential random variables with means 1/1, …, 1/n
respectively, then
(8)
Pr{Ti = min{T1, ..., Tn}} =
i
1 +  +  n
4.1 - 5
Proof. S and T have density functions fS(s) = e-s and fT(t) = e-t for s  0 and t  0 and fS(t) = fT(t) = 0 for
s < 0 or t < 0. Since S and T are independent, it follows that
-s -t
 e e
fS,T(s,t) = 
0
if s  0 and t  0
if s < 0 or t < 0
If A = {(s,t): s < t}, then
Pr{S < T} =



0
s
0
-s -t
-s -s

 e e ds

 f(s,t) dsdt = 
[
 e e dt ] ds = 
A

=

-(+)s
ds = e-(+)s

 e
+
|
0

s=0
=

+
This proves (7). We prove (8) by induction. It suffice to prove (8) for i = 1. The case n = 2 is (7).
Suppose it is true for n – 1. Note that Pr{T1 = min{T1, ..., Tn}} = Pr{T1 < R} where R = min{T2, ..., Tn}. It
follows from Proposition 1 that R is an exponential random variable with mean 1/(2 +  + n). So (8)
follows from (7). //
Proposition 3. (a) Let S and T be independent exponential random variables with means 1/ and 1/
respectively. Let R = min(S, T) and Z = 1 if S < T and Z = 2 if T  S. Then R and Z are independent.
(b) More generally, let T1, ..., Tn be independent exponential random variables with means 1/1, …, 1/n
respectively and R = min(T1, ..., Tn) and Z = j if Tj = R. Then R and Z are independent.
Proof. To prove (a) it suffices to show Pr{R > r, Z = 1} = Pr{R > r} Pr{Z = 1}. From Propositions 1 and
2, one has Pr{R > r} Pr{Z = 1} = e-(+)r


. So we need to show Pr{R > r, Z = 1} e-(+)r
. If
+
+
A = {(s,t): r < s < t}, then
Pr{ R > r, Z = 1} =
A

=
 

r s
r
-t -t
-t -t

 e e dsdt

 f(s,t) dsdt = 

 e e dsdt = 

-(+)t
dsdt = e-(+)t

 e
+
r
|

r
=
 -(+)r
e
+
which proves (a). We prove (b) by induction. The case n = 2 is part (a). Suppose it is true for n – 1. It
suffices to show Pr{R > r, Z = j} = Pr{R > r} Pr{Z = j}. It suffices to show this for j = 1, i.e.
Pr{R > r, Z = 1} = Pr{R > r} Pr{Z = 1}. From Propositions 1 and 2, one has
…+ )r
n
Pr{R > r} Pr{Z = 1} = e-(1+
1
. Note that R = min{T1, S} where S = min{T2, ..., Tn} and S is
1 +  + n
exponential with rate 2 + … + n by Proposition 1. Also, Pr{R > r, Z = 1} = Pr{R > r, T1 < S}. It follows
…+ )r
n
from part (a) that Pr{R > r, T1 < S} = e-(1+
1
which we needed to show. //
1 +  + n
4.1 - 6