Download Stochastic processes

Stochastic processes
 Definition
 A time-oriented, physical process that is controlled by a random
mechanism
 A sequence of random variables [Xt], where t ∈ T is a time or sequence
index
 State: mutually exclusive and collectively exhaustive description
of attributes of the system
 Topics to be covered tonight …
 Markov process
 Chapman-Kolmogrov equations
 Markov chains
 Birth-death equations
 Queuing
1
ETM 620 - 09U
Markov process
 Markovian property
P
{
X

j
|
X

i
}

P
{
X

j
|
X

i
,
X

i
,
X

i
,
..
X

i
}
t

1
t
t

1
t
t

1
1
t

2
2
0
t
 Interpretation …

 Examples:
 Arrival of customers at the bank
 Time required to inspect a passenger’s carry-on at the airport
 Etc.
2
ETM 620 - 09U
Three classic examples
 Marketing
Brand
A
Other
 Maintenance
Operational
Org.
Maint.
Depot
 Hospital (staffing)
3
ETM 620 - 09U
Probabilities in Markov processes
 One-step transition probability
P{Xt+1= j|Xt = i} = pij
 This is called stationary if
P{Xt+1= j|Xt = i} = P{X1 = j|X0 = i}
for t = 0, 1, 2, …
 Our example:
Operational
4
Org.
Maint.
Depot
ETM 620 - 09U
The matrix
 One-step transition matrix
 Properties:
 0 < pij < 1
From
 ∑Mpij = 1
0.8
0.6
0.3
To
0.2
0.3
0
0
0.1
0.7
 N-step transition matrix
pij(n) = P{Xt+n = j|Xt = i}
= P{Xn = j|X0 = i}
5
ETM 620 - 09U
Finite-state Markov chain
 Definition
 Stochastic process
 Finite number of states
 Markovian
 Stationary transition probabilities
 Initial set of probabilities
 Chapman-Kolmogrov equations
pij(n) = l=0∑Mpil(v)plj(n-v)
6
i = 0, 1, 2, …, M
j = 0, 1, 2, …, M
0<v<M
ETM 620 - 09U
Building the n-step transition matrix
 Start with 1-step probabilities
 Initial probability matrix

0.8 0.2 0


P
0.6 0.3 0.1

0.3 0 0.7


 The 2-step transition matrix is now


7




0
.
8
0
.
20
0
.
8
0
.
20




2
P

P

P

0
.
6
0
.
3
0
.
1

0
.
6
0
.
3
0
.
1









0
.
300
.
7
0
.
300
.
7




ETM 620 - 09U
Building the matrices in Excel
 Create the matrix P
 Copy this one or two rows below, call the copy Q
 Multiply P×Q using =mmult
 Steps:
 Similarly, P3=P×P×P
 Results from above multiplied by P
 And P4=P×P×P×P
 Etc.
8
ETM 620 - 09U
Classification of states and chains
 First passage time
 Length of time (number of steps) for the process to go from state i to
state j.
 E.g., go from Operational to Depot
 If fij(n) is the probability that the first passage time from i to j is n, then
fij(n)=pij(n) - fij(1)*pjj(n-1 )- fij(2)*pjj(n-2) - … - fij(n-1)*pjj
 First return time (recurrence time)
 Length of time (number of steps) to return to i, i.e., fii(n)
(
n
)
f
1
 If, 
n

1ii 
then state i is a recurrent state
9

ETM 620 - 09U
Classification of states and chains (cont.)
 Absorbing state
 pii = 1
 Once entered, never leave this state.
 Transient state
(
n
)

f
1
n

1ii 
 There is some probability that the process will never return to this
state.

10
ETM 620 - 09U
Expected first passage time



1


p


ij
l

j
i l lj
 To find all expected first passage times, solve as a series of
simultaneous linear equations, e.g. (for a 3×3)

11
ETM 620 - 09U
States and chains
 Periodic state
 only returns to the state in τ, 2τ, 3τ, … steps, where τ>1
 Accessible
 j is accessible from i if pij(n) > 0 for some n
 i and j COMMUNICATE if
 j is accessible from i and i is accessible from j
 Irreducible
 Only 1 class (partitioned set of the state space) exists in which all
states communicate
12
ETM 620 - 09U
States and chains
 Ergodic – a state i in a class that is
 Not periodic
 Positive recurrent (i.e., μii<∞)
 Steady state probabilities
 Irreducible, ergodic Markov chains will reach a “steady state”
probability,
A(n+1) = An*P
 Mean recurrence time
μjj = 1/pj
13
ETM 620 - 09U
Continuous-time Markov chains
(
t
)

P
[
X
(
t

s
)

j
|
X
(
s
)

i
]
p
ij
 Time is a continuous parameter
 State space (range of values for t) is discrete
 Chapman-Kolmogrov equations become

pij(t)= l=0∑mpil(v) * plj(t-v)
 Intensity* of transition, given the state j


1

p
(

t
)
d
jj
u

lim


p
(
t
)
 
j
jj
t

0

t

0

t 
dt

 Intensity* of passage from state i to state j


 uij  lim pij (t)  d pij (t)
t 0
t 0 t
 dt
14
ETM 620 - 09U
Example: Maintenance & repair
 Two identical, redundant modules in a control mechanism.
 Failure rate:
fT(t)= λe-λt
t ≥0
 Repair time:
rT(t)= μe-μt
t ≥0
 The transition intensities are:
u0  2
u1    
u 01  2 
u 12  
u 02  0
u 20  0
u 10  
u 21  2 
u 22  2 
15
ETM 620 - 09U
Example (cont.)
 From eq. 18-17
0: __________________
1: __________________
2: ___________________
since p0 = p1 = p2 = 1
p0 = ________________
p1 = ________________
p2 = ________________
16
ETM 620 - 09U
Example (cont.)
 Let’s assume MTBF = 1/5 day  λ = _________
MTTR = 1/6 day  µ = _________
we can now compute the probabilities …
p0 = ________________
p1 = ________________
p2 = ________________
17
ETM 620 - 09U
Birth-death processes
 Used in queueing theory
 birth = arrival to the system
 interarrival times are commonly assumed to be
exponentially distributed
 λn=arrival rate given that there are n customers in
the system
 death = departure from the system
 queueing system – the queue and the service facility
 µn= service rate given that there are n customers in
the system
 Transition diagrams
 similar to what we’ve shown in our example,
but with transitions
18
 Transition matrix
ETM 620 - 09U
Transitions …
 Transition matrix
1



t


t
0
0
...
0


0
0



t
1

(



)

t


t
0
...
0
1
1
1
1






t
1

(



)

t


t
...
0
2
2
2
2
P



0


t
1

(



)

t
...
0
2
2
2




...
0


t
...
...
3


0
...
...
...
...




 Time dependent behavior (eq. 18-23, 18-24)

p


(



)
p
(
t
)


p
(
t
)


p
(
t
)
'
p


p
(
t
)

p
(
t
)
0
0
0
1
1
'
j
19
j
j j
j

1
j

1
j

1
j

1
ETM 620 - 09U
Steady state equations
 Steady-state equations (18-25)
1 p1  0 p0
0 p0  2 p2  (1  1) p1
1 p1  3 p3  (2  2) p2

j2 pj2  j pj  (j1  j1) pj1
j1 pj1  j1 pj1  (j  j ) pj

 Solving
0
p0
1


p2  1 p1  1 0 p0
2
21
p1 

20
j
jj1 0
pj1 
p 
p
j1 j j1j 1 0
j1j2
0
let
C
j
jj1

1
1
and
solving
,we
getp
0

1
C
j
j
1
ETM 620 - 09U
Considerations in queuing models
 If service and arrival rates are constant and
 L = expected number of customers in the queueing system
 Lq = expected queue length
 W = expected time in the system (including service time)
 Wq = expected waiting time in the queue (including service time)
 if λ is constant, then
 L = λW
 Lq = λWq
 System utilization coefficient (fraction of time servers are busy)
 ρ = λ/sµ
21
ETM 620 - 09U
Basic single-server model (constant rates)
 Assume:
s=1
 unlimited potential queue length
 exponential interarrivals with constant λ
 independent exponential service times with constant µ
 Length of queue

L

1


 
L

1



(



)
2
2
q
 Time in system and time in queue
L

1
W
  W


(
1


)





(



)
q
q
22
ETM 620 - 09U
Recall our maintenance example …
 MTBF = 1/5 day  λ = _________
MTTR = 1/6 day  µ = _________
ρ = ___________
23
L = ___________
W = ______________
Lq = ___________
Wq = ______________
ETM 620 - 09U
Other models
 Limited queue length
 Multiple servers with unlimited queue
 Others …
24
ETM 620 - 09U