Download Chapter 3 Asymptotic Equipartition Property

Chapter 3
Asymptotic Equipartition Property
Peng-Hua Wang
Graduate Inst. of Comm. Engineering
National Taipei University
Chapter Outline
Chap. 3 Asymptotic Equipartition Property
3.1 Asymptotic Equipartition Property Theorem
3.2 Consequences of the AEP: Data Compression
3.3 High-Probability Sets and Typical Set
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 2/17
3.1 Asymptotic Equipartition Property
Theorem
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 3/17
Definition of convergence
Given a sequence of random variables, X1 , X2 , . . . we say that the
sequence X1 , X2 , . . . converges to a random variable X
■
In probability if for every ǫ
> 0,
lim Pr {|Xn − X| > ǫ} = 0
n→∞
or, equivalently,
lim Pr {|Xn − X| < ǫ} = 1
n→∞
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 4/17
Definition of convergence
■
In mean square if
lim E[|Xn − X|2 ] = 0
n→∞
■
With probability 1 or called almost surely if
Pr
Peng-Hua Wang, April 2, 2012
n
lim Xn = X
n→∞
o
=1
Information Theory, Chap. 3 - p. 5/17
Weak law of large numbers
For i.i.d. random variables X1 , X2 , . . . , Xn with common mean m, we
have
That is, for any ǫ
n
X
1
Xi → m in probability.
n i=1
> 0,
( n
)
1 X
X i − m > ǫ = 0
lim Pr n→∞
n
i=1
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 6/17
AEP
Theorem 3.1.1 (AEP) If X1 , X2 , . . . are i.i.d.
∼ p(x), then
1
− log p(X1 , X2 , . . . , Xn ) → H(X) in probability
n
Proof. Let Zi
= − log p(Xi ) be i.i.d. random variables. That is, Zi = − log p[Xi = x] if
Xi = x, we have
E[Zi ] = −
X
p[Xi = x] log p[Xi = x] = H(Xi ) = H(X)
Now, by the weak law of large numbers,
1X
Zi → H(X) in probability
n i
1X
log p(Xi ) → H(X) in probability
⇒−
n i
⇒−
Peng-Hua Wang, April 2, 2012
1
log p(X1 , X2 , . . . , Xn ) → H(X) in probability n
Information Theory, Chap. 3 - p. 7/17
Interpretation of AEP
■
When n is sufficient large, p(X1 , X2 , . . . , Xn )
= 2−nH(X) with
high probability.
■
For example, Let the random number Xi with probability
P [Xi = 1] = p and P [Xi = 0] = 1 − p = q . If X1 , X2 , . . . , Xn
are i.i.d.,
p(X1 , X2 , . . . , Xn ) = p
When n
P
Xi n−
q
P
Xi
.
→ ∞,
p(X1 , X2 , . . . , Xn ) → pnp q nq = 2−nH .
It means that the number of 1’s in the sequence is close to np, and all
such sequences have roughly the same probability 2−nH .
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 8/17
Interpretation of AEP
■
Thus for large n we can divide the sequences X1 , X2 , . . . , Xn into
two types: the typical type consisting of sequences each with
probability roughly 2−nH , and another type, consisting of other
sequences.
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 9/17
Typical set
(n)
Definition (Typical set) The typical set Aǫ
set of sequence (x1 , x2 , . . . , xn )
with respect to p(x) is the
∈ X n with the property
2−n(H(X)+ǫ) ≤ p(x1 , x2 , . . . , xn ) ≤ 2−n(H(X)−ǫ) .
Theorem 3.1.2
1. If (x1 , x2 , . . . , xn )
(n)
∈ Aǫ , then
1
H(X) − ǫ ≤ − log p(x1 , x2 , . . . , xn ) ≤ H(X) + ǫ
n
.
Proof. By the definition of typical set.
Peng-Hua Wang, April 2, 2012
.
Information Theory, Chap. 3 - p. 10/17
Theorems
Theorem 3.1.2
2.
(n)
Pr{Aǫ } > 1 − ǫ for n sufficiently large.
Proof. This property follows directly from Theorem 3.1.1, since the
convergence in the mean can be written as
1
Pr − log p(X1 , X2 , . . . , Xn ) − H(X) < ǫ > 1 − δ
n
Setting δ
= ǫ, we obtain the desired result. Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 11/17
Theorems
Theorem 3.1.2
(n)
|Aǫ | ≤ 2n(H(X)+ǫ) , where |A| denotes the number of elements in
the set A.
3.
Proof.
1=
X
p(x) ≥
x∈X n
≥
X
X
p(x)
(n)
x∈Aǫ
2−n[H(X)+ǫ] = 2−n[H(X)+ǫ] |A(n)
ǫ | (n)
x∈Aǫ
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 12/17
Theorems
Theorem 3.1.2
4.
(n)
|Aǫ | ≥ (1 − ǫ)2n(H(X)−ǫ) for n sufficiently large.
(n)
Proof. For n sufficiently large, Pr{Aǫ
1−ǫ<
Pr{A(n)
ǫ }
≤
X
} > 1 − ǫ, so that,
2−n[H(X)−ǫ] = 2−n[H(X)−ǫ] |A(n)
ǫ | (n)
x∈Aǫ
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 13/17
3.2 Consequences of the AEP: Data
Compression
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 14/17
Typical set and source coding
■
There are |X |n elements in the whole set.
■
There are |Aǫ
(n)
| ≈ 2n(H+ǫ) elements in the typical set. We need
n(H + ǫ) + 1 bits to encode these elements, and one addition bit to
indicate they are typical sequences.
■
There are |X |n
(n)
− |Aǫ | elements in the nontypical set. We can use
n log |X | + 1 bits to encode them, and one addition bit to indicate
they are non-typical sequences.
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 15/17
Average length of codeword
n
E[l(X )] =
X
p(xn )l(xn )
xn
=
X
n
n
p(x )l(x ) +
(n)
≤
p(xn )l(xn )
(n) c
]
xn ∈Aǫ
X
X
xn ∈[Aǫ
p(xn )[n(H + ǫ) + 2]
(n)
xn ∈Aǫ
+
X
p(xn )[n log |X | + 2]
(n)]c
xn ∈[Aǫ
(n) c
= Pr{A(n)
ǫ }[n(H + ǫ) + 2] + Pr{[Aǫ ] }[n log |X | + 2]
≤ n(H + ǫ) + ǫn log |X | + 2
= n(H + ǫ′ )
where ǫ′
= ǫ + ǫ log |X | +
Peng-Hua Wang, April 2, 2012
2
n
Information Theory, Chap. 3 - p. 16/17
Theorems
Theorem 3.2.1 Let X n be i.i.d.
∼ p(x). Let ǫ > 0. Then there exists
a code that maps sequences xn of length n into binary strings such that
the mapping is one-to-one (and therefore invertible) and
1
E
l(X n ) ≤ H(X) + ǫ
n
for n sufficiently large.
Peng-Hua Wang, April 2, 2012
Information Theory, Chap. 3 - p. 17/17