Download 4 Entropy as Missing Information

Entropy as Missing Information 1
4
Entropy as Missing Information
Here we generalize the idea of information to encompass physical systems. The
idea of missing information is seen to be formally equivalent to entropy. Much of
the unit is devoted to technical details of combinatorics and manipulating factorial
expressions.
General Equation
The basic idea is to recast
I  k G ln n
(1)
into a form that is expressed in terms of the number of possible messages Wthat is,
information depends on the number of messages that could have been constructed. A
physical counterpart is to regard W as the number of possible states of a system. The
number of messages is thereafter regarded as a special case of the number of possible
states.
Consider a message consisting of G symbols of an n-letter alphabet. Since each
symbol can have n values, the total number of possible combinations is nG:
n  n  n ... n  n  n G

G of these symbols
The number of “messages” W that could be written in this scheme is therefore given by
W  nG
and Eq.(1) becomes
I  k ln W
(2)
1.
Find the information in the genetic message ACT CCG using I  k ln W . Does it
agree with simple counting and I  k G ln n ?
2.
Starting with I  k ln W , show that the information in a message of G “letters”
from an n-letter alphabet can be written as I  k G ln n .
Entropy and Information
Equation (2) represents the information in a system when the system is fully
specified. Most often, however, the microscopic details of a physical system are not
specified and then (2) tells us how much information is missing. In the cases where
Entropy as Missing Information 2
Eq.(2) represents missing information, I is formally identical to the quantity physicists
call entropy, S. As you know, entropy is a technical measure of randomness. The
entropy
(2’)
S  k ln W
is often called the Boltzmann entropy. It can be regarded as the most fundamental
entropy expression.
Calculating W with Factorials
Various applications of Eq. (2) in information theory and in statistical mechanics
require us to manipulate factorials.
First consider how many states can be formed from a row of G distinguishable
items. The first item can be selected G ways, the next can be selected in G-1 ways, and so
on
...
1
G
G 1
first item
second item
last item
The product is G!, the number of ways G distinguishable items can be distributed in G
cells.
3.
In how may arrangements can 5 different color balls be placed in a row? Find the
missing information in the (unspecified) array. [ans. 120, 6.9 bits]
We now pose the question of how many states can be formed from a row of G
items consisting of subsets containing G1, G2 , and GN identical items. For example, if an
arrangement of 7 balls consists of 5 black and 2 red balls, then G = 7, Gblack = 5, and
Gred = 2. Note that we have
G  G1  G2 G N
(3)
In this case there are less than G! arrangements because we must divide out the G1!
arrangements that are indistinguishable from each other. Similar statements can be made
for G2, G3 ,...,GN. We have
W
G!
G1 ! G2 ! G N !
(4)
4.
How many configurations may be made of 3 black and 2 red balls placed in a
row? List all the arrangements. [ans. 10]
5.
A 5-letter word is known to contain 2 e’s, 2 t’s, and one r. How many
combinations can be formed from these letters and what is the information in the
word (in bits)? [ans. 30, 4.9 bits]
Entropy as Missing Information 3
Sterling’s Formula
For a large number, N, the following expression is highly accurate:
ln N !  N ln N  N .
(5)
This is Sterling’s Formula*. Most of the numbers we deal with in information theory and
statistical mechanics are large enough to treat Eq. (5) as an exact expression.
Derivation:
ln N !  ln N  ln N  1 ln 2  ln 1
N
  ln N dN
1
 N ln N  N
The first term usually dominates and we can use
ln N! N ln N
6.
(5’)
A particular polymer consists of a chain of G molecular units. Each molecular
unit can have a bent or straight configuration. Find an expression for the entropy
(in units of k) when there are m bends distributed randomly over the polymer
Hint: Treat the polymer as a “message” of m bent and G-m straight molecular
units. Use Eq. (5) and calculate the value of m that gives the maximum entropy.
[ans. m=G/2] (Later you will see that the maximum entropy or missing
information corresponds to the most probable state of the system.)
We saw that in order to conform to thermodynamic relations the quantity k in
S  k ln W must be taken as Boltzmann’s constant, 1.3810-23 J/K. This is the alteration
made when we examine the entropy of physical systems rather than messages,
The following problem introduces the simplest model for a rubber band or other elastic
polymer. Often, the relation S  k ln W is best suited for theoretical developments, but
some systems are amenable to applying it directly as in this model.
*
There are additional terms in the full Sterling expression that increase its accuracy. However, these can be
ignored for the large numbers treated in statistical physics.
Entropy as Missing Information 4
7.
Consider a rubber band to be comprised of N links of length a that may point left
or right. The whole chain has a length L as shown.
a
L
L
(a) Let NR and NL represent the number of links pointing right and left
respectively. Express the multiplicity of states W in terms of N, NR, and NL .
Use the expression S  k ln W and apply the Sterling approximation assuming
all numbers are large.
(b) Note that the length and total number of links are constrained. In particular,
L  N R  N L  a
N  NR  NL
Show that NR and NL can now be expressed in terms of N and L,
N R  12 N  L a 
N L  12 N  L a 
and substitute these into S from part (a). This is the full expression for the
entropy of the rubber band as a function of N and L. The following sections
wring physical results from this entropy.
(c) The results will be easier to recognize if you approximate your result for small
L relative to Na (this is physically reasonable). Show that expanding S to
kL2
second order in L gives S  Nk ln 2 
. (This is most easily done using a
2 Na 2
series application with a math program like Maple. By hand, you might use
ln 1  x   x  12 x 2 for small x.)
(d) The fundamental thermodynamic relation applied to this system is
dU  TdS  fdL
where f is the tension applied to the rubber band. Solve this for dS. The result
is a physical expression for entropy as a function of U and L. Compare this
with the mathematical identify,
 S 
 S 
dS  
 dU    dL
 U  L
 L U
to find the relation of S to f:
 S 
f  T  
 L U
Use this to evaluate f. From your result, (i) does the tension increase or
decrease for increased temperature? (ii) Note that the tension obeys
Hooke’s Law. What is the spring constant? [ans. kT / Na 2 ]