Download Lecture 13

STAT253/317 Winter 2014 Lecture 13
6.5. Limiting Probabilities
Definition. Just like discrete-time Markov chains, if the probability
that a continuous-time Markov chain will be in state j at time t,
Pij (t), converges to a limiting value Pj independent of the initial
state i, for all i ∈ X
Pj = lim Pij (t) > 0
t→∞
k∈X ,k6=j
If we let t → ∞, and assume that we can interchange limit and
summation, we obtain


X
′
lim Pij (t) = lim 
Pik (t)qkj  − νj Pij (t)
t→∞
then we say Pj is the limiting probability of state j. If Pj exists for
all j ∈ X , we say {Pj }j∈X is the limiting distribution of the chain.
Remark. If limt→∞ Pij (t) exists, we must have
lim P ′ (t)
t→∞ ij
Recall the forward equations are


X
′
Pij (t) = 
Pik (t)qkj  − νj Pij (t)
= 0.
t→∞
↓
0
k∈X ,k6=j
↓
P
=
k∈X ,k6=j Pk qkj
↓
− νj Pj
Hence we get the balanced equations.
X
νj Pj =
Pk qkj for all j ∈ X
k∈X ,k6=j
Lecture 13 - 1
Interpretation of the Balanced Equations
Lecture 13 - 2
Examples
◮
νj Pj =
X
Pk qkj
Poisson processes: µn = 0, λn = λ for all n ≥ 0
νi = λ, Pi,i+1 = 1,
k∈X ,k6=j
qi,i+1 = νi Pi,i+1 = λ
Balanced equations:
νj Pj = rate at which the process leaves state j
X
νj Pj = Pj−1 qj−1,j
Pk qkj = rate at which the process enters state j
k∈X ,k6=j
Balanced equations means that the rates at which the process
enters and leaves state j are equal.
The limiting distribution {Pj }j∈X can be obtained
by solving the
P
balanced equations along with the equation j∈X Pj = 1.
Remarks. Just like discrete-time Markov chains, a sufficient
condition for the existence of a limiting distribution is that the
chain is irreducible and positive recurrent.
◮
◮
single-server service station
◮ Poisson arrival of customers with rate λ
◮ Upon arrival, customer
◮ goes into service if the server is free (queue length = 0)
◮ joins the queue if 1 ≤ queue length < N, or
◮ walks away if queue length ≥ N
◮ Service times are i.i.d. ∼ Exp(µ)
Q: What fraction of potential customers are lost?
X (t) = # of customers in the station. This is a birth-death
process with


ν0 = λ, νN = ν, νn = µ + λ, 1 ≤ n < N
P0,1 = 1, PN,N−1 = 1
µ0 = 0,


µ
λ
µn = µ, 1 ≤ n ≤ N,
Pn,n+1 = µ+λ
, Pn,n−1 = µ+λ
,1 ≤ n < N
⇒ 

λn = λ, 0 ≤ n < N
q0,1 = λ, qN,N−1 = µ
λN = 0
qn,n+1 = νn Pn,n+1 = λ,


qn,n−1 = νn Pn,n−1 = µ, 1 ≤ n < N
◮
Lecture 13 - 5
λPj = λPj−1
⇒
Pj = Pj−1
No limiting distribution exists. The chain is not irreducible.
All states are transient.
Pure birth processes with λn > 0 for all n
No limiting distribution exists. The chain is not irreducible.
All states are transient.
Pure birth processes with
λn > 0 for n ≤ 10, and λn = 0 for all n > 10.
State space X = {0, 1, 2, . . . , 10}.
State 10 is the only absorbing state. All others are transient.
Lecture 13 - 3
Variation of Example 5.5 (M/M/1 Queueing)
⇒
Lecture 13 - 4
Variation of Example 5.5 (M/M/1 Queueing)
Balanced equations
λP0 = µP1
(µ + λ)Pn = λPn−1 + µPn+1 ,
1≤n ≤N −1
µPN = λPN−1
P1 = (λ/µ)P0
P2 = (λ/µ)P1 = (λ/µ)2 P0
..
.
i = 1, 2, . . . , N
Pi = (λ/µ)i P0 ,
P
Plugging Pi = (λ/µ)i P0 to N
i=1 Pi = 1, one can solve for P0 and
get
1 − λ/µ
Pi =
(λ/µ)i
1 − (λ/µ)N+1
Ans: Fraction of customers lost = PN =
Lecture 13 - 6
1−λ/µ
(λ/µ)N
1−(λ/µ)N+1
Limiting Distribution for Birth and Death Processes
For a birth and death process,
ν 0 = λ0 ,
ν i = λi + µi ,
P01 = 1,
i
Pi,i+1 = λi λ+µ
,
i
i
Pi,i−1 = λi µ+µ
,
i
Pi,j = 0
Limiting Dist’nP
for Birth and Death Processes (Cont’d)
∞
n=0 Pn
Using the fact
= 1, we obtain
P0 + P0
i >0
i >0
i >0
if |i − j| > 1
⇒
qi,i+1 = νi Pi,i+1 = λi , i ≥ 0
qi,i−1 = νi Pi,i−1 = µi , i ≥ 1
∞
X
λn−1 λn−2 · · · λ0
n=1
µn µn−1 · · · µ1
=1
One can see that, to have a limiting distribution, it is necessary
that
∞
X
λ0 λ1 · · · λn−1
<∞
µ1 µ2 · · · µn
n=1
Balanced equations
This condition also may be shown to be sufficient.
If that is finite we can see that the limiting distribution is
λ0 P 0 = µ 1 P 1
(µi + λi )Pi = λi−1 Pi−1 + µi+1 Pi+1 ,
i >0
⇒ λn Pn = µn+1 Pn+1 .
Pn =
λn−1
λn−1 λn−2
λn−1 λn−2 · · · λ0
Pn−1 =
Pn−2 = . . . =
P0
µn
µn µn−2
µn µn−1 · · · µ1
P0 =
and
Pk =
Lecture 13 - 7
Lemma: (Ratio Test) If an ≥ 0 for all n, then
(
∞
X
< ∞ if limn→∞ an /an−1 < 1
an
=
∞ if limn→∞ an /an−1 ≥ 1
n=1
λ0 λ1 · · · λn−1
, an /an−1 = λn−1 /µn . By the ratio test, if
µ1 µ2 · · · µn
λn−1
lim
< 1,
n→∞ µn
P∞ λ0 λ1 ···λn−1
P∞
then n=1 an = n=1 µ1 µ2 ···µn < ∞, the limiting distribution
exists.
Example 6.4 Linear Growth Model with Immigration
For an =
µn = nµ,
λn = nλ + θ
Using the Ratio Test,
λn−1
(n − 1)λ + θ
λ
lim
= lim
=
n→∞ µn
n→∞
nµ
µ
So the linear growth model has a limiting distribution if and only
λ < µ.
Since T0 ∼ Exp(λ0 ), E[T0 ] = 1/λ0 .
Using the recursive formula λi E[Ti ] = 1 + µi E[Ti−1 ], we have
1
E[T0 ] =
λ0
1
µ1
1
µ1
E[T1 ] =
+ E[T0 ] =
+
λ1 λ1
λ1 λ1 λ0
1
µ2 1
µ1
µ2
µ2 µ1
1
E[T2 ] =
+
+
+
+
=
λ2 λ2 λ 1 λ 1 λ 0
λ2 λ2 λ1 λ2 λ1 λ0
..
.
1
µi
1
µi
µi µi−1 · · · µ2 µ1
+ E[Ti−1 ] =
+
+ ··· +
E[Ti ] =
λi
λi
λi
λi λi−1
λi λi−1 · · · λ2 λ1 λ0


k
i Y
X
µi−j+1 
1 
=
1+
λi
λi−j
1+
λ0 λ1 ···λn−1
n=1 µ1 µ2 ···µn
λ0 λ1 · · · λk−1 /(µ1 µ2 · · · µk )
,
P
λ0 λ1 ···λn−1
1+ ∞
n=1 µ1 µ2 ···µn
Duration Times for Birth and Death Processes
Let
Ti = time to move from state i to state i + 1,
i = 0, 1, . . . .
Then Ti , i = 0, 1, . . . are independent random variables.
E[Ti ] = E[Ti |i → i + 1]Pi,i+1 + E[Ti |i → i − 1]Pi,i−1
1
µi
1
λi
=
+
+ E[Ti−1 ] + E[Ti ]
λi + µ i λi + µ i
λi + µ i
λi + µ i
1
µi
=
+
(E[Ti−1 ] + E[Ti ])
λi + µ i
λi + µ i
We obtain the recursive formula
λi E[Ti ] = 1 + µi E[Ti−1 ]
Lecture 13 - 10
6.6. Time Reversibility
Definition. A continuous-time Markov chain with state space X is
time reversible if
Pi qij = Pj qji ,
for all i, j ∈ X
(detailed balanced equation)
If a distribution {Pj } on X satisfies the detailed balanced equation,
then it is a stationary distribution for the process.
Example. One can use the detailed balanced equation to find the
stationary distribution of a Birth and Death process.
λn Pn = µn+1 Pn+1 ,
k=1 j=1
Lecture 13 - 11
k≥1
Lecture 13 - 8
Lecture 13 - 9
Duration Times for Birth and Death Processes (Cont’d)
1
P∞
Lecture 13 - 12
n≥0