Download Transient Analysis of Markov Processes

Lecture 4: Algorithmic Methods for
G/M/1 and M/G/1 type models
Dr. Ahmad Al Hanbali
Department of Industrial Engineering
University of Twente
[email protected]
1
Lecture 4


This Lecture deals with continuous time Markov
chains with infinite state space as opposed to finite
space with skip-free in one direction as opposed to
QBDs Lecture 3
Objective: To find equilibrium distribution of the
Markov chain
Lecture 4: G/M/1 and M/G/1 type models
2
Background (1): G/M/1 queue

Interarrival time of jobs is arbitrary distribution, 𝐹𝐴 (𝑡) with
mean 1/𝜆, and inter-arrival times are iid

Jobs service times are iid exponential rvs with rate 𝜇. Interarrivals and service times are independent

Service discipline can be Fisrt-In-First-Out (FIFO)

Under above assumptions, the nbr of jobs in G/M/1 queue at
arbitrary time 𝑡 DOES NOT form a Markov chain

Let 𝑁(𝑡𝑖 ) denote the number of jobs in the queue just before
the 𝑖-th arrival. The process {𝑁(𝑡𝑖 ), 𝑖 = 0,1, … } is a discretetime Markov chain

Let 𝑎𝑛 denote the probability that exactly 𝑛 jobs are served
during an inter-arrival time given there are at least 𝑛 jobs
present at the start of inter-arrival. Then 𝑎𝑛 reads
𝑎𝑛 =
∞ 𝜇𝑡 𝑛 −𝜇𝑡
𝑒 𝑑𝐹𝐴 (𝑡) , 𝑛
0
𝑛!
Lecture 4: G/M/1 and M/G/1 type models
= 0,1, …, and 𝑏𝑖 =
𝑘>𝑖 𝑎𝑘
3
Background (2): G/M/1 queue
𝑎0
𝑎1
0
i-1
1
𝑏𝑖
𝑎𝑖
i
i+1
𝑎2
The transition probability matrix is given
𝑏0 𝑎0 0 0 0 …
𝑏1 𝑎1 𝑎0 0 0 …
𝑏2 𝑎2 𝑎1 𝑎0 0 …
𝑃=
.
…
𝑎
𝑏3 3 𝑎2 𝑎1 𝑎0
𝑏4 𝑎4 𝑎3 𝑎2 𝑎1 ⋱
⋮ ⋮ ⋮ ⋮ ⋮⋱
For stable case with 𝜆 < 𝜇 we find
𝑝𝑖 = 1 − 𝜎 𝜎 𝑖 , (Geometric distribution)
where 𝜎 is the unique root in (0,1) of 𝜎 = 𝐸 𝑒 −𝜇 1−𝜎
Lecture 4: G/M/1 and M/G/1 type models
𝐴
.
4
Definition G/M/1-type processes:
skip-free process to the right




A 2-dimensional irreducible continuous time Markov
process with states (𝑖, 𝑗), where 𝑖 = 0, … , ∞ and 𝑗 =
0, … , 𝑚 − 1
Subset of state space with common 𝑖 entry is called
level 𝑖 (𝑖 > 0) and denoted 𝑙(𝑖) = {(𝑖, 0), (𝑖, 1), … , (𝑖, 𝑚 − 1)}.
𝑙(0) = {(0,0), (0,1), … , (𝑖, 𝑚0 − 1)}. This means state space
is ∪𝑖≥0 𝑙(𝑖)
Transition rate from (𝑖, 𝑗) to (𝑖′, 𝑗′) is equal to zero for
𝑖′ − 𝑖 ≥ 2
For 𝑖 > 0, transition rate between states in 𝑙(𝑖) and
from 𝑙(𝑖) to 𝑙 𝑖 + 1 , 𝑙 𝑖 − 1 , … , 𝑙(0) are independent of 𝑖
Lecture 4: G/M/1 and M/G/1 type models
5
Skip-free to the right process
Order the states lexicographically, i.e.,
0,0 , … , 0, 𝑚0 , 1,0 , … , 1, 𝑚 , 2,0 , … , 2, 𝑚 , …, the
generator of the skip-free process has the following form:
𝐵00 𝐵01 0 0 0 …
𝐵10 𝐵11 𝐴0 0 0 ⋱
𝑄 = 𝐵20 𝐴2 𝐴1 𝐴0 0 ⋱
𝐵30 𝐴3 𝐴2 𝐴1 𝐴0 ⋱
⋮ ⋱ ⋱ ⋱ ⋱⋱
where 𝐴0 and 𝐴2 are nonnegative 𝑚-by-𝑚 matrices; 𝐴𝑖 is
square matrices of size 𝑚; 𝐵00 is square matrix of size
𝑚0 ; 𝐵𝑖0 𝑚0 -by-𝑚 and 𝐵01 𝑚-by-𝑚0 nonnegative matrices.
Note, (𝐵00 + 𝐵01 )𝑒 = 0, (𝐵10 + 𝐵11 + 𝐴0 )𝑒 = 0, and (𝐵0𝑖 +
𝑖
𝑙=0 𝐴𝑙 )𝑒 = 0
Lecture 4: G/M/1 and M/G/1 type models
6
Stability of G/M/1-type process
Let 𝐴 = ∞
𝑖=0 𝐴𝑖 . 𝐴 is the generator describing transitions of the
M/G/1-type process between level states (i.e., in the vertical
direction)
Theorem: Assume the Markov chain with generator 𝐴 is
irreducible with equilibrium distribution, 𝜋𝐴 = 0, 𝜋𝑒 = 1. The
G/M/1-type process is ergodic if and only if
∞
𝑖 − 1 𝐴𝑖 𝑒 (mean drift condition)
𝜋𝐴0𝑒 < 𝜋
𝑖=1
Interpretation: 𝜋𝐴0𝑒 is mean drift from level 𝑖 to 𝑖 + 1.
𝜋 ∞
𝑖=1 𝑖 − 1 𝐴𝑖 𝑒 is the mean drift from level 𝑖 to levels smaller
than 𝑖 for large 𝑖
Lecture 4: G/M/1 and M/G/1 type models
7
Equilibrium distribution of G/M/1type processes
Let 𝑝𝑛 = (𝑝(𝑛, 0), . . , 𝑝(𝑛, 𝑚 − 1)) and 𝑝 = (𝑝0, 𝑝1, … ) then
equilibrium equation 𝑝𝑄 = 0 reads
∞
∞
𝑝
=
0,
𝑝
+
𝑝
+
0𝐵01
1𝐵11
𝑖=0 𝑖 𝐵𝑖0
𝑖=2 𝑝𝑖 𝐴𝑖 = 0,
∞
𝑖=0 𝑝𝑛+𝑖 𝐴𝑖 = 0, 𝑛 ≥ 1
Theorem: if the G/M/1-type process is ergodic the
equilibrium probability distribution then reads
𝑝𝑛 = 𝑝1𝑅𝑛−1 , 𝑛 ≥ 1,
where 𝑅 is the minimal nonnegative solution of the
matrix equation
∞
𝑖
𝑅
𝐴𝑖 = 0
𝑖=0
Interpretation: 𝑅 = 𝐴0𝑁 same as in QBD process
Lecture 4: G/M/1 and M/G/1 type models
8
Equilibrium distribution of G/M/1type processes(cnt'd)
A direct result of the previous theorem is that spectral
radius of 𝑅 is < 1 and (𝐼 − 𝑅) is nonsingular
Lemma: The stationary probability vectors 𝑝0 and 𝑝1
is the normalized unique solution of 𝑥0 , 𝑥1 𝐵 𝑅 = 0,0 ,
where 𝐵 𝑅 is the generator given by
𝐵00
𝐵01
𝐵𝑅 =
.
∞
∞
𝑖−1
𝑖−1
𝐵𝑖0
𝐵11 + 𝑖=1 𝑅 𝐴𝑖
𝑖=1 𝑅

Normalization is done by letting 𝑥0 𝑒𝑚0 + 𝑥1 𝐼 − 𝑅 −1 𝑒𝑚 = 1.
Note 𝑒𝑚0 and 𝑒𝑚 are column vectors of ones with size 𝑚0 and
𝑚, respectively
Proof: follows by inserting 𝑝𝑛 = 𝑝1𝑅 𝑛−1 in the balance equations
Lecture 4: G/M/1 and M/G/1 type models
9
Finding R

Rearrange the equation of the rate matrix:
∞
𝑅𝑖 𝐴𝑖 𝐴1−1
𝑅 = − 𝐴0 +
𝑖=2



Fixed point equation solved by successive substitution
𝑖 𝐴 )𝐴−1 , with 𝑅 = 0
𝑅𝑘+1 = −(𝐴0 + ∞
𝑅
𝑖 1
0
𝑖=2 𝑘
It can be shown that 𝑅𝑘 → 𝑅 for 𝑘 → ∞
In many queueing systems for large 𝐾, 𝐴𝐾 ≈ 0.
𝑖𝐴 ≈ 𝐾
𝑖𝐴
Truncating ∞
𝑅
𝑅
𝑖
𝑖
𝑖=2 𝑘
𝑖=2 𝑘
Lecture 4: G/M/1 and M/G/1 type models
10
Special case: GI/PH/1 queue

Consider the case 𝐵01 = 𝐴0 and 𝐵11 = 𝐴1

The matrix 𝐵𝑖0 is a rank one matrix satisfying
𝐵𝑖0 = −
𝑖
𝑗=0 𝐴𝑗 𝑒𝛽.
where 𝛽 is the initial state probability vector of the
service time phase-type distribution
Lemma: The stationary probability vector of GI/PH/1
embedded at the moment of arrivals
𝑝𝑖 = 𝑝0 𝑅𝑘 ,
where 𝑝0 = 𝛽 𝐼 − 𝑅
−1
𝑒
−1
𝛽.
Proof: in this case 𝐵 𝑅 will be of rank one
Lecture 4: G/M/1 and M/G/1 type models
11
Definition of M/G/1-type processes:
skip-free process to the left




A 2-dimensional irreducible continuous time Markov
process with states (𝑖, 𝑗), where 𝑖 = 0, … , ∞ and 𝑗 =
0, … , 𝑚 − 1
Subset of state space with common 𝑖 entry is called
level 𝑖 (𝑖 > 0) and denoted 𝑙(𝑖) = {(𝑖, 0), (𝑖, 1), … , (𝑖, 𝑚 − 1)}.
𝑙(0) = {(0,0), (0,1), … , (𝑖, 𝑚0 − 1)}. This means state space
is ∪𝑖≥0 𝑙(𝑖)
Transition rate from (𝑖, 𝑗) to (𝑖′, 𝑗′) is equal to zero for
𝑖 ′ − 𝑖 ≤ − 2.
For 𝑖 > 0, transition rate between states in 𝑙(𝑖) and
from 𝑙(𝑖) to 𝑙 𝑖 − 1 , 𝑙 𝑖 + 1 , …, are independent of 𝑖
Lecture 4: G/M/1 and M/G/1 type models
12
Skip-free to the left process
Order the states lexicographically, i.e., 0,0 , … , (0, 𝑚0 −
Lecture 4: G/M/1 and M/G/1 type models
13
Stability of M/G/1-type processes
Let 𝐴 = ∞
𝑖=0 𝐴𝑖 . 𝐴 is the generator describing transitions
of the M/G/1-type process in the vertical direction.
Theorem: Assume the Markov chain with generator 𝐴 is
irreducible with equilibrium distribution, 𝑖. 𝑒. 𝜋𝐴 = 0, 𝜋𝑒 = 1,
and with ∞
𝑖=1 𝑖𝐵0𝑖 𝑒 finite. The M/G/1-type process is
ergodic if and only if
𝜋𝐴0𝑒 > 𝜋 ∞
𝑖=2 𝑖 − 1 𝐴𝑖 𝑒 (mean drift condition)
Lecture 4: G/M/1 and M/G/1 type models
14
Equilibrium distribution of skip-free
to the left processes

Assume M/G/1-type process is ergodic. The minimal
nonnegative solution 𝐺 of
∞
𝑖
𝐴
𝐺
𝑖=0 𝑖
= 0,
is then stochastic, i.e., 𝐺𝑒 = 𝑒,
Interpretation: 𝑘, 𝑙 -element of 𝐺 represents probability
to jump for the first time to level 𝑖 − 1 by entering state
𝑙, 𝑖 ≥ 1, given process starts in (𝑖, 𝑘) at time 0
Theorem (Matrix analytic): Assume M/G/1-type process
is ergodic then equilibrium probability of 𝑙 𝑖 gives
𝑝𝑖 = − 𝑝0 𝐵𝑖 +
where, 𝐵𝑖 =
𝑖−1
𝑘=1 𝑝𝑘
∞
𝑘, 𝐴
𝐵
𝐺
𝑖
𝑘=0 0𝑖+𝑘
Lecture 4: G/M/1 and M/G/1 type models
𝐴𝑖+1−𝑘
=
𝐴1
−1 , 𝑖
∞
𝑘, 𝑛
𝐴
𝐺
𝑘=0 𝑖+𝑘
= 2,3, …,
= 1,2, ….
15
Finding 𝐺
In many queueing systems for large 𝐾, 𝐴𝐾 ≈ 0. Using
this, 𝐴𝐾 and 𝐵𝐾 → 0, then the following recursion
(backward) can used
𝐵𝑖 = 𝐵0𝑖 + 𝐵𝑖+1 𝐺
𝐴𝑖 = 𝐴𝑖 + 𝐴𝑖+1 𝐺

In this case, it is reasonable to truncate the infinite sum
of 𝐺𝑙+1 at K

The matrix 𝐺 can be found recursively as follows
𝑖
𝐺𝑙+1 = −𝐴1−1 𝐴0 + 𝐾
𝐴
𝐺
, 𝑙 = 1,2, … , with 𝐺0 = 0.
𝑖=2 𝑖 𝑙


It can be shown that 𝐺𝑘 → 𝐺 for 𝑘 → ∞
Lecture 4: G/M/1 and M/G/1 type models
16
Probabilities 𝑝0 and 𝑝1
Theorem: The stationary probability vectors 𝑝0 and 𝑝1
is the normalized unique solution of 𝑥0 , 𝑥1 𝐵 𝐺 = 0,0 ,
where 𝐵 𝐺 is the generator of an irreducible chain given by
𝐵00
𝐵𝐺 =
𝐵10


∞
𝑘
𝐵
𝐺
𝑘=0 01+𝑘
∞
𝑘
𝐴
𝐺
𝑘=0 1+𝑘
done such ∞
𝑖=0 𝑝𝑖 𝑒
.
Normalization should be
=1
𝐵 𝐺 is M/G/1-type process restricted to 𝑙 0 & 𝑙(1), 𝐵 𝐺 e=0
Special case: 𝐵10 = 𝐴0


Recursion in the theorem in slide 15 will also hold from for
𝑖 = 1.
𝑘 . Moreover, 𝑝 is the normalized unique
𝐵𝐺= ∞
𝐵
𝐺
0𝑘
𝑘=0
0
solution of 𝑥0 𝐵 𝐺 = 0
Lecture 4: G/M/1 and M/G/1 type models
17
Finding 𝑝0 : special case 𝐵10 = 𝐴0
∞
∞
Let 𝐵 = ∞
𝐵
,
𝑏
=
𝑖𝐵
𝑒,
𝑎
=
0𝑖
𝑖=0 0𝑖
𝑖=1
𝑖=1 𝑖𝐴𝑖 𝑒

Assume 𝐴 = ∞
𝑖=0 𝐴𝑖 is irreducible and let 𝜋𝐴 = 0, 𝜋𝑒 = 1
𝜇 = 𝜋𝑎 (load measure)
Theorem: Assume the generator of skip-free to the left
process and 𝐴 are irreducible. If the skip-free to the left
process is ergodic, then
𝜇 = 𝑝0 𝜇𝑒 − 𝑏 + 𝐵 𝐴 + 𝑒𝜋 −1 𝑎
Proof: Similar Theorem 4.8 in Bini et al. (2005)

Using 𝑝0 𝐵 𝐺 = 0 and additional equation in the
previous Theorem, then 𝑝0 can be uniquely determined

It is also possible to find 𝐸𝐿 = ∞
𝑖=1 𝑖𝑝𝑖 𝑒 in closed form
as function of 𝑝0 , 𝜋, 𝑎, 𝑏, …
18

Lecture 4: G/M/1 and M/G/1 type models
𝑥
Example: Geo /D/1 with slot reservations in
care pathway patients

Priority patient reserves time slot between L and H slots in future:
1. If time slot L is occupied by a regular patient, the priority patient
takes the position of the regular patient, and all regular patients
from position L onward are shifted to the next time slot
2. If slot L is occupied by priority patient, then new priority patient
moves to the first available slot in the interval L + 1, . . . , H that is
not occupied by a priority patient, and takes that position.
3. If all slots in the interval L, . . . , H are occupied by priority patients,
the new priority patient is blocked
19
Lecture 4: G/M/1 and M/G/1 type models
𝑥
Example: Geo /D/1 with slot reservations in
care pathway patients



𝑁2 (𝑖 ), 𝑖 ≥ 1, total number of regular patients waiting in
the queue at the end of each time slot (level state)
𝑉(𝑖 ), 𝑖 ≥ 1, a state vector of length H with entries
denoting which slot in the set {1, . . . , H} contains a
priority patient (phase state)
Two-dimensional process (𝑁2 (𝑖 ), 𝑉(𝑖 )) is discrete time
Markov chain with transition probability matrix M/G/1type form
Lecture 4: G/M/1 and M/G/1 type models
20
𝑥
Example: Geo /D/1 with slot reservations in
care pathway patients




One-step transition of the priority patients state 𝑉 𝑖 are
independent of the number of regular patients present
𝑗
𝑞2 𝑞2
Let 𝑎𝑗 = 1 −
and 𝐷 the one-step transition
probability matrix of priority patients 𝑉 𝑖 → 𝑉 𝑖 + 1
𝐴𝑗 can be rewritten as 𝑎𝑗 𝐴𝐷, where 𝐴 is the diagonal
matrix, 𝐵0 can be written as 𝑎0 𝐵𝐷, where 𝐵 is the
diagonal matrix, and 𝐶𝑗 can be written as 𝑎𝑗 𝐷
We find that the equilibrium probability vector of is
matrix geometric
𝜋𝑙 = −𝜋0 𝑆 𝑞2 𝐼 − 𝐴𝑆 𝑙−1 , 𝑙 = 1,2, …,
where 𝑆 = 𝐶1 𝐴0 − 𝐼 + 𝑞2 𝑋 −1 and 𝑋 is minimal
nonnegative solution of quadratic matrix equation
Lecture 4: G/M/1 and M/G/1 type models
21
References





Bini, Latouche, Meini, “Numerical Methods for Structured
Markov Chains” Oxford University Press, 2005.
G. Latouche and V. Ramaswami (1999), Introduction to
Matrix Analytic Methods in Stochastic Modeling. SIAM.
M.F. Neuts (1989), Structured Stochastic Matrices of
M/G/1 Type and Their Applications. Marcel Dekker, INC.
M.F. Neuts (1981), Matrix-geometric solutions in
stochastic models. The John Hopkins University Press,
Baltimore.
M.E. Zonderland, R.J. Boucherie, A. Al Hanbali. Appointments
in care pathways: the Geo^x/D/1 queue with slot reservations.
Queueing Systems, vol. 79, issue 1 (2015).
Lecture 4: G/M/1 and M/G/1 type models
22