Download Lecture 11: Introduction to Markov Chains

Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Lecture 11:
Introduction to Markov Chains
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Copyright G. Caire (Sample Lectures)
321
Discrete-time random processes
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• A sequence of RVs indexed by a variable n 2 {0, 1, 2, . . .} forms a discretetime random process {Xn} = {Xn : n = 0, 1, 2, . . .}.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• The process if defined by the collection of all joint distributions of order m,
FXn1 ,...,Xnm (x1, . . . , xm)
for all instants {n1, . . . , nm} and for all m = 1, 2, 3, . . ..
• If the process has the property that, given the present, the future is
independent of the past, we say that the process satisfies the Markov
property, or, it is a Markov chain.
• Furthermore, we are mainly interested in the case where Xt takes on values
in some countable set S, called state space.
Copyright G. Caire (Sample Lectures)
322
Markov chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Definition 44. The process {Xn} is a Markov chain if it satisfies the Markov
property:
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
P(Xn = j|X0 = x0, . . . , Xn
for all i, j, x0, . . . , xn
2
1
= i) = P(Xn = j|Xn
1
= i)
⌃
2 S and for all n = 1, 2, 3, . . ..
• The Markov property implies that:
P(Xnk = j|Xn0 = x0, . . . , Xnk
for all k, n, all n0  n1  · · ·  nk
1
1
= i) = P(Xnk = j|Xnk
1
= i)
 nk and all i, j, x0, . . . , xnk 1 .
• Also
P(Xn+m = j|X0 = x0, . . . , Xm = i) = P(Xn+m = j|Xm = i)
Copyright G. Caire (Sample Lectures)
323
Transition probability
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• The evolution of a markov chain is defined by its transition probability, defined
by P(Xn+1 = j|Xn = i) (where without loss of generality we may assume that
S is an integer set.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Definition 45. The chain {Xn} is called homogeneous if its transition
probabilities do not depend on the time, i.e.,
P(Xn+1 = j|Xn = i) = P(X1 = j|X0 = i)
for all n, i, j. The transition probability matrix P = [pi,j ] is the |S| ⇥ |S| matrix of
the transition probabilities, such that
pi,j = P(Xn+1 = j|Xn = i)
⌃
Copyright G. Caire (Sample Lectures)
324
Property of the transition matrix
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Theorem 46. The transition matrix P of a Markov chain is a stochastic matrix,
that is, it has non-negative elements such that
X
pi,j = 1
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
j2S
(sum of the elements on each row yields 1)
• In order to characterize the probability for n steps transitions, we introduce
the n-step transition probability matrix with elements
pi,j (m, m + n) = P(Xm+n = j|Xm = i)
• By homogeneity, we have that P(m, m + 1) = P.
• Furthermore, P(m, m + n) = P(n) does not depend on m.
Copyright G. Caire (Sample Lectures)
325
Chapman-Kolmogorov equation
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Theorem 47.
pi,j (m, m + n + r) =
X
pi,k (m, m + n)pk,j (m + n, m + n + r)
k
Therefore, P(m, m + n + r) = P(m, m + n)P(m + n, m + n + r). It follows that
for homogeneous Markov chains, P(m, m + n) = Pn, i.e., P(n) = Pn.
Copyright G. Caire (Sample Lectures)
326
Long-term evolution
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We let u(n) denote the pmf of Xn, that is, for each n we have that u(n) is a
vector with |S| non-negative components that sum to 1.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Lemma 33. u(m + n) = u(m)Pn, and hence u(n) = u(0)Pn. This describes
the pmf of Xn in terms of the initial state pmf u(0).
Copyright G. Caire (Sample Lectures)
327
First step and last step conditioning
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We have a MC with transition matrix P. Let A ⇢ S denote a subset of states.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• For i 2
/ A, we are interested in the probability
P(Xk 2 A for some k = 1, . . . , m|X0 = i) = P ([m
k=1 {Xk 2 A}|X0 = i)
• Define the stopping time N = min{n
Wn =
⇢
1 : Xn 2 A}, and the new MC
Xn n < N
a
n N
where a is a special symbol.
Copyright G. Caire (Sample Lectures)
328
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• {Wn} has transition probability matrix
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
qi,j
qi,a
= pi,j if i 2
/ A, j 2
/A
X
=
pi,j if i 2
/A
j2A
qa,a = 1
• Notice that the original MC enters a state in A by time m if and only if Wm = a,
then
m
P ([m
k=1 {Xk 2 A}|X0 = i) = P(Wm = a|W0 = i) = qi,a (m) = [Q ]i,a
Copyright G. Caire (Sample Lectures)
329
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• Now, we are interested in the probability of never entering the subset of states
A in the times 0, . . . , m, i.e., for i, j 2
/ A we wish to compute
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
1
P {Xm = j} \m
/ A}|X0 = i
k=1 {Xk 2
• The above condition is equivalent to Wm = j conditionally on W0 = i, hence
1
P {Xm = j} \m
/ A}|X0 = i = qi,j (m)
k=1 {Xk 2
Copyright G. Caire (Sample Lectures)
330
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
• Next, we wish to consider the first entrance probability, for i 2
/ A and j 2 A
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
1
P {Xm = j} \m
/ A}|X0 = i =
k=1 {Xk 2
=
X
r 2A
/
=
X
P {Xm = j} \ {Xm
P (Xm = j|Xm
r 2A
/
=
X
pr,j qi,r (m
1
1
2
= r} \m
/ A}|X0 = i
k=1 {Xk 2
= r) P {Xm
1
2
= r} \m
/ A}|X0 = i
k=1 {Xk 2
1)
r 2A
/
• When i 2 A and j 2
/ A, we can determine
1
P {Xm = j} \m
/ A}|X0 = i
k=1 {Xk 2
Copyright G. Caire (Sample Lectures)
331
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• By conditioning on the first transition:
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
1
P {Xm = j} \m
/ A}|X0 = i =
k=1 {Xk 2
=
X
r 2A
/
=
X
r 2A
/
=
X
1
P {Xm = j} \m
/ A} \ {X1 = r}|X0 = i
k=2 {Xk 2
1
P {Xm = j} \m
/ A}|X1 = r P(X1 = r|X0 = i)
k=2 {Xk 2
qr,j (m
1)pi,r
r 2A
/
Copyright G. Caire (Sample Lectures)
332
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• Finally, we can calculate the conditional probability of Xm = j given that the
chains starts at X0 = i and has not entered the subset A in any time till time
m. For i, j 2
/ A, we have
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
P(Xm = j|{X0 = i} \m
/ A}) =
k=1 {Xk 2
P({Xm = j} \m
/ A}|X0 = i)
k=1 {Xk 2
=
P(\m
/ A}|X0 = i)
k=1 {Xk 2
1
P({Xm = j} \m
/ A}|X0 = i)
k=1 {Xk 2
=P
m 1
P
({X
=
r}
\
/ A}|X0 = i)
m
r 2A
/
k=1 {Xk 2
qi,j (m)
=P
r 2A
/ qi,r (m)
Copyright G. Caire (Sample Lectures)
333
Classification of states (1)
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Definition 46. A state i 2 S is called persistent (or recurrent) if
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
P(Xn = i for some n
1|X0 = i) = 1
Otherwise, if the above probability is strictly less than 1, the state is called
transient.
⌃
• We are interested in the first passage probability
fi,j (n) = P(X1 6= j, X2 6= j, . . . , Xn
• We define fi,j =
fj,j = 1.
P1
Copyright G. Caire (Sample Lectures)
n=1 fi,j (n).
1
6= j, Xn = j|X0 = i)
Notice: state j is persistent if and only if
334
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We define the generating functions
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Pi,j (s) =
1
X
pi,j (n)sn,
Fi,j (s) =
n=0
with the conditions: pi,j (0) =
1
X
fi,j (n)sn
n=0
i,j ,
fi,j (0) = 0 for all i, j 2 S.
• We shall assume |s| < 1, since in this way the generating functions converge,
and occasionally evaluate limits for s " 1 using Abel’s theorem.
• Notice that fi,j = Fi,j (1).
Theorem 48. a) Pi,i(s) = 1 + Pi,i(s)Fi,i(s),
b) Pi,j (s) = Fi,j (s)Pj,j (s), for i 6= j.
Copyright G. Caire (Sample Lectures)
335
Classification of states (2)
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
P
Corollary
6. a) State j is persistent if n pj,j (n) = 1. If this holds, then
P
n pi,j (n) = 1 for all states
P i such that fi,j > 0.
P
b) State j is transient if n pj,j (n) < 1. If this holds then n pi,j (n) < 1 for all
i.
c) If j is transient, then pi,j (n) ! 0 as n ! 1.
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• Let N (i) denote the number of times a chain visit state i given that its starting
point is i. It is intuitively clear that
⇢
1 i is persistent
P(N (i) = 1) =
0 i is transient
• Let Tj = min{n : Xn = j} denote the time to the first visit to state j. This is a
random variable, with the convention that Tj = 1 is the visit never occurs.
• We have that P(Tj = 1|X0 = j) > 0 if and only if j is transient an in this
case E[Tj |X0 = j] = 1.
Copyright G. Caire (Sample Lectures)
336
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We define the mean recurrence time µj of a state j as
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
µj = E[Tj |X0 = j] =
⇢ P
1
n nfj,j (n)
j is persistent
j is transient
Notice: µj may be infinity even though the state is persistent!
Definition 47. A persistent state j is called null if µj = 1 and non-null or
positive if µj < 1.
⌃
Theorem 49. A persistent state i is null if and only if pi,i(n) ! 0 as n ! 1. If
this holds, then pj,i(n) ! 0 for all j.
Copyright G. Caire (Sample Lectures)
337
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• Let d(i) denote the period of state i, defined as
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
d(i) = GCD{n : pi,i(n) > 0}
• We call i periodic, if d(i) > 1, and aperiodic if d(i) = 1.
Definition 48. A state is called ergodic if it is persistent, non-null and aperiodic.
⌃
Copyright G. Caire (Sample Lectures)
338
Classification of chains (1)
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We say that state i communicates with state j (written i ! j) if pi,j (n) > 0 for
some n
0. In this case, state j can be reached from state i with positive
probability.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• We say that state i and j intercommunicate if i ! j and j ! i. In this case,
we write i $ j.
• Clearly, i $ i, since pi,i(0) = 1.
• Also, i ! j if and only if fi,j > 0.
• Intercommunication $ induces an equivalence relation on the state space.
In fact, if i $ j and j $ k then i $ k. Hence, S can be partitioned into
equivalence classes of mutually intercommunicating states.
Copyright G. Caire (Sample Lectures)
339
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Theorem 50. If i $ j then:
a) i and j have the same period;
b) i is transient if and only if j is transient;
c) i is null persistent if and only if j is null persistent;
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Copyright G. Caire (Sample Lectures)
340
Classification of chains (2)
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Definition 49. A set C ✓ S of states is called:
a) closed if pi,j = 0 for all i 2 C and j 2
/ C.
b) irreducible if i $ j for all i, j 2 C.
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
⌃
• Once the chain enters a closed set of states, it will never leave it (trapped).
• A closed set of cardinality 1 (single state) is called absorbing state.
• For example, the state 0 in a branching process is an absorbing state.
• Equivalence classes with respect to the relation $ are irrducible.
• We call an irreducible set C aperiodic, persistent null, persistent etc .. if all
states in the set have the property.
Copyright G. Caire (Sample Lectures)
341
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Theorem 51. Decomposition theorem. The state space S can be uniquely
partitioned as
S = T [ C1 [ C2 [ · · ·
where T is the set of transient states, and Ci are irreducible closed sets of
persistent states.
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Copyright G. Caire (Sample Lectures)
342
Qualitative behavior of Markov chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• In light of Theorem 51 we can identify the following behaviors:
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• If the chain starts in some state i 2 Cr , then it stays in Cr forever.
• If the chain starts in T , then either it stays in T forever or eventually will end
up into some Cr .
• If S is finite (finite-state Markov chain), then the first possibility cannot occur:
Lemma 34. If S is finite, then at least one state is persistent and all persistent
states are non-null.
Copyright G. Caire (Sample Lectures)
343
Example
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Let S = {1, 2, 3, 4, 5, 6} and consider the transition matrix
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
2
6
6
6
6
P=6
6
6
6
4
Copyright G. Caire (Sample Lectures)
1
2
1
4
1
4
1
4
1
2
3
4
1
4
3
0 0 0 0
0 0 0 0 7
7
7
1
1
7
4
4 0 0 7
0 14 14 0 14 7
7
7
0 0 0 0 12 12 5
0 0 0 0 12 12
344
Stationary distribution
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Definition 50. The vector ⇡ is called a stationary distribution of the chain if it
has entries {⇡j : j 2 S}Psuch that:
a) ⇡j 0 for all j, and j2S ⇡j = 1.P
b) it satisfies ⇡ = ⇡P, that is, ⇡j = i ⇡ipi,j , for all j 2 S.
⌃
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• This is called “stationary distribution” since if X0 is distributed with u(0) = ⇡,
then all Xn will have the same distribution, in fact
u(n) = u(0)Pn = ⇡Pn = ⇡PPn
1
= ⇡Pn
1
= ··· = ⇡
• Given the classification of chains and the decomposition theorem, we shall
assume that the chain is irreducible, that is, its state space is formed by a
single equivalence class of intercommunicating (persistent) states C, or by
the class of transient states T .
Copyright G. Caire (Sample Lectures)
345
Stationary distribution and mean recurrence time
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
Theorem 52. A irreducible chain has a stationary distribution ⇡ if and only if
all states are non-null persistent. In this case, ⇡ is unique and satisfies ⇡j = µ1j ,
where µj is the mean recurrence time of state j.
Copyright G. Caire (Sample Lectures)
346
Solution of the linear equation x = xP
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• Fix a state k 2 S.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• Let ⇢i(k) denote the mean number of visits of the chain to state i between
two successive visits to state k.
• Defining
Ni =
1
X
n=1
I{Xn=i}\{Tk
n}
where Tk is the time of first passage to state k, we have
⇢i(k) = E[Ni|X0 = k] =
Copyright G. Caire (Sample Lectures)
1
X
P(Xn = i, Tk
n|X0 = k)
n=1
347
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
• Since k is persistent, then
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
⇢k (k) =
=
1
X
n=1
1
X
P(Xn = k, Tk
n|X0 = k)
P(Tk = n|X0 = k) =
n=1
1
X
fk,k (n) = 1
n=1
P
• It is also clear that Tk = i2S Ni, since the time between two visits to state
k must be spent somewhere else. Taking expectations, we find
X
µk =
⇢i(k)
i2S
this shows that the vector ⇢(k) = {⇢i(k) : i 2 S} contains the terms whose
sum is equal to the mean recurrence time of state k, denoted as usual by µk .
Copyright G. Caire (Sample Lectures)
348
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Lemma 35. For any state k of an irreducible persistent chain, the vector ⇢(k)
satisfies ⇢i(k) < 1 for all i and furthermore ⇢(k) = ⇢(k)P.
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• For an irreducible persistent chain, we have proved that the equation x = xP
has a solution ⇢(k) with non-negative entries, that sum to µk .
• If state k is non-null, then µk < 1, and therefore the vector ⇡ = ⇢(k)/µk is
non-negative, with sum 1, and solution of the equation x = xP.
• It follows that ⇡ = ⇢(k)/µk is a stationary distribution.
• This shows that every non-null persistent irreducible chain has a stationary
distribution, summarized as follows:
Theorem 53. If the chain is irreducible and persistent, there exists a positive
root x of the equation xP= xP which is unique
P up to a multiplicative constant.
The chain is non-null if i xi < 1 and null if i xi = 1.
Copyright G. Caire (Sample Lectures)
349
Example: irreducible class of the previous example
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Let S = {1, 2} and consider the transition matrix
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
P=
Copyright G. Caire (Sample Lectures)
"
1
2
1
4
1
2
3
4
#
350
Criterion for non-null persistency of irreducible chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• Consider an irreducible chain (notice: irreducibility must be checked directly
by inspection).
• In order to check if the chain is non-null persistent, we just have to look for a
stationary distribution.
• Since Theorem 52 is a necessary and sufficient condition, if we find ⇡, then
we conclude that the chain is non-null persistent.
Copyright G. Caire (Sample Lectures)
351
Criterion for transience of irreducible chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Theorem 54. Let s 2 S be a state of an irreducible chain. The chain is transient
if an only if there exists a non-zero solution {yj : j 6= s} satisfying |yj |  1 for all
j, to the system of equations
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
yi =
X
j:j6=s
Copyright G. Caire (Sample Lectures)
pi,j yj ,
8 i 6= s
352
Criterion for persistency and null persistency
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• It follows from Theorem 54 that an irreducible chain is persistent if and only
if the only bounded solution to the system of equations in Theorem 54 is the
zero vector yj = 0 for all j.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• If this happens, but the chain has no stationary distribution, then we conclude
that the chain is null persistent.
• Another condition for null persistency is given as follows:
Theorem 55. Let s 2 S be a state of an irreducible chain with S = {0, 1, 2, . . .}.
The chain is persistent if there exists a solution {yj : j 6= s} to the system of
inequalities
X
yi
pi,j yj , 8 i 6= s
j:j6=s
such that yi ! 1 as i ! 1.
Copyright G. Caire (Sample Lectures)
353
Ergodic behavior of Markov chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• We are interested in the limit of the n-th order transition probabilities, pi,j (n),
for large n.
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
• Periodicity yields some difficulties, as the following examples shows:
Example: consider S = {1, 2}, with p1,2 = p2,1 = 1. Then,
p1,1(n) = p2,2(n) =
⇢
0 if n is odd
1 if n is even
• It is clear that if states are periodic, the sequence of n-th order transition
probabilities may not converge.
Copyright G. Caire (Sample Lectures)
354
Ergodic theorem for Markov chains
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Theorem 56. For an irreducible aperiodic chain, we have that
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
lim pi,j (n) =
n!1
1
µj
8 i, j 2 S
Remarks:
• If the chain is transient or null-persistent, then pi,j (n) ! 0 for all i, j since
µj = 1.
• If the chain is non-null persistent, then pi,j (n) !
i, where ⇡ is the unique stationary distribution.
1
µj
= ⇡j > 0, independent of
• From Theorem 56 we see that in any case, for an irreducible aperiodic chain,
the limit of pi,j (n) is independent of X0 = i.
Copyright G. Caire (Sample Lectures)
355
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
• Therefore, the chain forgets its starting point. It follows that
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
P(Xn = j) =
X
i
1
P(X0 = i)pi,j (n) !
µj
irrespectively of the initial pmf vector u(0) = {P(X0 = i), i 2 S}.
• If the chain {Xn} is periodic, it turns out that a “sampling of the chain at
integer multiples of the period” {Yn = Xdn : n 0} is aperiodic. In this case,
letting qj,j (n) = P(Yn = j|Y0 = j), it turns out that
qj,j (n) = P(Xnd
d
= j|X0 = j) = pj,j (nd) !
µj
or, equivalently, that the mean recurrence time of the new chain {Yn} is equal
to 1/d times the mean recurrence time of the original chain.
Copyright G. Caire (Sample Lectures)
356
Faculty of
Electrical Engineering and
Computer Systems
Department of Telecommunication
Systems
Information and Communication
Theory
Prof. Dr. Giuseppe Caire
Einsteinufer 25
10587 Berlin
Telefon +49 (0)30 314-29668
Telefax +49 (0)30 314-28320
[email protected]
Sekretariat HFT6
Patrycja Chudzik
Telefon +49 (0)30 314-28459
Telefax +49 (0)30 314-28320
[email protected]
End of Lecture 11
Copyright G. Caire (Sample Lectures)
357