Download p 0

Lecture on Markov Chain
Definition : Markov Chain is a discrete time discrete alphabet random process
{Xn, n>0} such that
Pr( X n1  j | X n  i , X n1  in1 ,..., X 0  i0 )  Pr( X n1  j | X n  i )
 X n1 is conditiona lly independen t of X 0 , X 1 ,..., X n1 , given X n .
 Let   0,1,2,... be the discrete set of values that X n takes.
  is called the state space of X n 
  is finite or countably infinite.
Definition : Initial distribution is the probability distribution of
X 0, such that  0 (i )  Pr( X 0  i ).
Associated with any Markov Chain is a stochastic matrix
(or transi tion matrix) P such that the (i,j) th element is p(i,j)  Pr (X n1  j|X n  i)
where p(i,j) is the probabilit y of going from state i to state j.
(Note p(i,j) is independen t of n, i.e.,
p(i,j)  Pr (X n 1  j|X n  i)  Pr (X m1  j|X m  i)
This type of Markov Chain is called homogeneou s.
P   p(i, j )i,j and Π0  (Π0( 0 ),Π0( 1 ),Π0( 2 ),...) specify th e random process, i.e.,
the probabilit y disributio n for the finite collection of the random variables
X 1 , X 2 , X 3 ,..., X m can be found from P and  0 .
Pr( X 0  i0 , X 1  i1 , X 2  i2 ,..., X m  im )
 Pr( X 0  i0 ) Pr( X 1  i1 | X 0  i0 ) 
Pr( X 2  i2 | X 0  i0 , X 1  i1 )  
  Pr( X m  im | X 0  i0 , X 1  i1 ,..., X m1  im1 )
  0 (i0 ) p(i0 , i1 ) p(i1 , i2 )... p(im1 , im )
Prove
Pr( X m n  j | X n  i, X n1  in1 ,..., X 0  i0 )  p m (i, j )
p m (i, j ) is the (i, j )th component of the matrix P m  
P 
P

P
 ...

P.
m times
Definition : P is a stochastic matrix means P has all non negative components
and each row adds up to 1.
Example: Bernoulli Process
N n ,
n  0,1,2,...
X n ' s are i.i.d and Pr( X n  0)  q, Pr( X n  1)  p, and p  q  1.
n
Let N n   X i  number of successes up to time n.
i 0
Then N n , n  0,1,2,... is a Markov Chain.
P(N n 1  j|N 0  i0 ,N1  i1,...N n  i)  P(N n 1  j|N n  i)
 p if j  i  1
 q if j  i
 0 else
P=
q
0
0
0
.
.
.
p
q
0
0
.
.
.
0
p
q
0
.
.
.
0
0
p
q
…
…
…
…
.…
.…
.…
Example
Let Tn be the time of the nth success.
Tn , n  0 is a Markov Chain
P(Tn1  j | T0  i0 , T1  i1 ,...Tn  i )  P(Tn1  j | Tn  i )
 pq j i 1 if
 0 else
P=
j  i 1
0
p
pq
pq2
…
0
0
p
pq
…
0
0
0
p
…
.
.
.
.…
.
.
.
.…
.
.
.
.…
Example
X n  is a discrete alphabet i.i.d. process.
Then X n  is a Markov Chain.
(Show it for yourself.)
Example
X n  is a discrete alphabet i.i.d. process with
Pr( X n  i )  pi .
n
Let Yn   X i .
i 0
Then , Yn is a Markov Chain.
Find P for Yn .
Classification of States
1. A state i ε  is recurrent if
Pr( X n  i for some n  1 | X 0  i )  1.
Otherwise it is called transient , i.e.,
Pr( X n  i for some n  1 | X 0  i )  1.
2. Given X 0  i, let
Tii  min n such that n  0 X n  i.
Tii is the time of first visi t back to state i.
Tii is a random variable.
Let μi  E(Tii ) be the mean recurrent time.
A recurrent state i is called positive recurrent if
1   and null recurrent if 1  .
3. A recurrent state i is periodic with period  ( 2)
if Pr(Tii  nδ for some n  1 | X 0  i )  1.
If there is no such  , it is called aperiodic.
4. Two states i and j communicat e (i  j ), if for some n
and m, p n(i,j)  0, and p m(j,i)  0. If i  j, then the y are of the
same type; if i is recurrent so is j , if i is transient , so is j ,
if i is periodic, so is j , etc.
5. If all the states in a Markov Chain communicat e, the
Markov Chain is called irreducibl e.
Fact : A Markov Chain is irreducibl e if there exists n such
that all the elements of P are positive.
6. A stationary distributi on Π   Π (0), Π (1), Π (2),... for a Markov Chain
is such that with trans ition matrix P,
Π  ΠP
7. For an irreducibl e and aperiodic Markov Chain with
transition matrix P, there exists
lim p n (i,j)
n 
and it is independen t of i. We call this limit Π(j).
Then, one of the two conditions below is true.
- Π(j)  0 for all j. Then, the Markov Chain has no stationary
distributi on and the chain is either tra nsient or recurrent null.
- Π(j)  0 for all j. Π   Π (0), Π (1), Π (2),... is the statinary distributi on,
and it is unique. Then the Markov Chain is positive recurrent.
1/Π ( j ) is equal to the mean recurrent time of state j.
Little’s Formula
•
•
•
•
•
•
Scenario:
Customers arrive at random times. Or equivalently, packets are assigned to links at
random times.
The probability distribution of time between two arrives is given.
Service time is random.
The probability distribution of service time is given
We are interested in finding out:
Average number of customers in queue
Average delay seen by customers
Packets arriving with
random packet lengths
Packets transmitted
at a constant rate
pn(t) : probability of n customers in queue or under service at time t.
pn(0): Initial probability of having n customers. Is assumed to be known.
N(t): number of customers at time t.
Tk : delay of k th customers, i.e., time spent in system by k th customer.

E N(t)  N(t)   npn(t)
n 0
T k  E(Tk )
Assume that T  lim k- E(Tk ) exists and is constant w here
T is the average delay per customer. Let
Tn 
1 n
 Ti
n i 1
T n is the average delay of the first n customers.
We can assume that T  lim n T n
N : Average number of customers in the system
T : Average delay per customer
 : Average customer arrival rate
Little’s Theorem
N  λT
E[number of arrivals in [0 ,t ]]
t 
t
N (t )  average number of customers in the system at time t.
  lim
N (t ) and pn(t) depends on t and pn( 0 ),n  0 ,1,2 ,...
Usually system reaches equilibriu m, i.e .,
lim pn(t)  pn , n  1,2 ,...
t 

lim N(t)  N   npn
t 
n 0
Also time average of N(t)
1 t
N t  0 N( )d
t
In general, many random processes are known to
possess the ergodic property w ith constant N , i.e.,
lim N t  lim N (t )  N
t 
t 
Queuing Models
Probability distribution of service time
(M: memoryless or exponential, D:deterministic, etc.,)

 /  / Number of servers

Nature of arrival process
(M:memoryless or Poisson, G:general, D:deterministic)
For M/M/1, the arrival process is Poisson with rate λ, the service time is
exponential random variable with mean 1/μ, and there is one server.
Poisson Process
A(t ), t  0is a Poisson process with
rate  if A(t) takes on non
negative integers and
1. A( 0 )  0
2. A(t)-A(s)  number of arrivals in (s,t .
3.The number of arrivals in disjoint intervals are independen t.
For t1  t 2  t3  t 4 ,
A(t 2 )  A(t1 ) is independen t of A(t 4 )  A(t3 ).
4. For and t and  , A(t  τ)  A(t) is a Poisson random variable
with parameter  . I.e.,
Pr (A(t  τ)-A(t)  k)  e
 
 k
k!
k  0 ,1,2 ,...
Properties of Poisson Process
1. Probabilit y of having k arrivals from 0 to t :
Pr (A(t)  k)  e
 t
t k
k  0 ,1,2,...
k!
o(δ )
 0.
δ 0 δ
2. Pr (A(t  δ)  A(t)  0 )  1  λδ  o(δ ) where lim
3. Pr (A(t  δ)  A(t)  1 )  λδ  o(δ )
4. Pr (A(t  δ)  A(t)  2 )  o(δ )
5. Inter  arrival times
Let T0 , T1, T2 ,... denote the time of arrivals.
Let  n  Tn 1  Tn .
Then,  0 , 1 , 2 ,... are i.i.d. exponentia l random variables with
Pr( n  s )  1  e s where s  0.
E ( n ) 
1

and var( n ) 
1

2
.
Fact : If Ai(t),t  0, i  1,2 ,3,...,n are independen t
Poisson processes with rates 1 , 2 , 3 ,..., n . Then
n
A(t)   Ai(t)
i 1
is a Poisson random process with parameter
n
λ   λi
i 1
Customer Service Time
Sn  Service time of nth customer
Pr(Sn  s )  1  e  s where s  0
Also service times are independent of each other and of inter-arrival times.
For exponential random variables,
Pr( n  t   |  n  t)  Pr( n   )
Pr(Sn  t   | sn  t)  Pr(Sn   )
Memoryless Property of Exponential Distribution
The probability that there will be no arrivals in the next τ seconds is the same as the
conditional probability of no arrival in the next τ seconds given that there have
been no arrivals for the past t seconds for any t.
M/M/1 Queuing Model
Balance equations
1    
λδ
n
1    
n+1
μδ
Consider (kδ, (k+1)δ] for k0. The number of transitions from state n to state n+1 is at most
one, and the number of transitions from state n+1 to state n is also at most 1.
In steady state, consider that system goes from state n to state n+1.
 pn   pn1
 pn 1 

pn  pn

 pn 1   n1 p0 for n  0,1,2,...
 p0  lim k  Pr( N k  0)

We know  pn  1. Then we can solve for p0 .
n 0
Assume that   1 , and therefore    .
Since N is the average number of customers in the system,

N   npn
N
n 0

  n(1   )  n
n 0

1- 


 

N
0

1 

By Little’s theorem, T, the average delay in the system, i.e., queue time plus service
time, is equal to T  N  1

 
Average delay in queue = average delay in system - average service time
W

1
1

 
    
NQ = average number of customers in queue

2
 W 

   1 
Number of customers in queue
0 if N k  0



 N k  1 if N k  0
2
 N Q   (n  1)  P0 
(same as before)
1 
n 1

n
Note :  is the utilization factor, i.e., long term proportion of time the server is busy.
Then  =1- p0 where p0 is the probability that the system is empty.
Example
Suppose we increase the arrival and transmission rates by the same factor k.
  k (faster arrivels )
1
1

(faster service )

k
N



1    
N
1
T 

k k (    )
Therefore, the delay is reduced by a factor of k while the number of customers are the same.
4
Arrival
Original system
N(t)
Departure
3
2
1
t
0
The reason is that the time is compressed by a factor of k and the customers move out faster.
4
3
System with faster
arrivals and departures
Arrival
N(t)
Departure
2
1
0
t
Multiplexing
Assume m Poisson arrivals of rate of /m each for the total transmission capacity of rate .
Two possible scenarios:
1.
Divide capacity into m, and transmit each arrival process over the divided one. Then
we have m M/M/1 queues with arrival rate /m with service rate  /m.
1
m
T

   

m m
Time division multiplexing (TDM) or frequency division multiplexing (FDM)
accomplishes this.
2.
Merge the m arrival processes into one Poisson process of rate  and use the entire
transmission capacity for it.
1
T
 
Statistical multiplexing achieves this.