Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Chapter 7 Mathematical Foundations 7-1 Notions of Probability Theory • Probability theory deals with predicting how likely it is that something will happen. • The process by which an observation is made is called an experiment or a trial. • The collection of basic outcomes (or sample points) for our experiment is called the sample space (Omega). • An event is a subset of the sample space. • Probabilities are numbers between 0 and 1, where 0 indicates impossibility and 1 certainty. • A probability function/distribution distributes a probability mass of 1 throughout the sample space. : F 0,1 1 7-2 Example the event of interest an experiment • A fair coin is tossed 3 times. What is the chance sample space of 2 heads? – ={HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} probabilistic function – uniform distribution 1 P(basic outcome)= 8 The chance of getting 2 heads when tossing 3 coins A={HHT, HTH, THH} P(A)= A 3 8 a subset of 7-3 Conditional Probability • Conditional probabilities measure the probability of events given some knowledge. • Prior probabilities measure the probabilities of events before we consider our additional knowledge. • Posterior probabilities are probabilities that result from using our additional knowledge. 1st Event B: is H Event A: 2 Hs in 1st, 2nd and 3rd 4 P(B)= 8 P(AB)= 2 8 1st H 2 P(A|B)= 4 2nd H T 3rd T H 2 4 7-4 P(A B) A | B P(B) A B AB The multiplication rule P(A B) P(B)P(A | B) P(A)P(B | A) The chain rule P(A1 A 2 ... A n ) P(A1 )P(A 2 | A1 )P(A 3 | A1 A 2 ) P(A n | in11 A i ) (used in Markov models, …) 7-5 Independence • The chain rule relates intersection with conditionalization (important to NLP) • Independence and conditional independence of events are two very important notions in statistics – independence: P( A) P( A | B) P( A B) P( A) P( B) – conditional independence P( A B | C ) P( A | C ) P( B | C ) 7-6 Bayes’ Theorem • Bayes’ Theorem lets us swap the order of dependence between events. This is important when the former quantity is difficult to determine. P(B A) P(A | B) (B) B | A P(A) P(A) • P(A) is a normalizing constant. 7-7 An Application • Pick the best conclusion given some evidence – (1) evaluate the probability P(c|e) <--- unknown – (2) Select c with the largest P(c|e). P(c | e) P (c ) * P ( e | c ) P(e) P( c ) and P(e|c) are known. • Example. – Relative probability of a disease – how often a symptom is associated 7-8 Bayes’ Theorem P(A | B) (B) arg max P(A) arg max P(A | B)(B) B B A B AB P( A) P( A B) P( A B) P( A | B) P( B) P( A | B) P( B) 7-9 Bayes’ Theorem ( j | ) if ( | j ) ( j ) ( ) A i 1 Bi , P(A) > 0, n ( | j ) ( j ) n i 1 ( | i )( i ) B B i j (ij) The group of sets Bi partition A (used in noisy channel model) 7 - 10 An Example • A parasitic gap occurs once in 100,000 sentences. • A complicated pattern matcher attempts to identify sentences with parasitic gaps. • The answer is positive with probability 0.95 when a sentence has a parasitic gap, and the answer is positive with probability 0.005 when it has no parasitic gap. • When the test says that a sentence contains a parasitic gaps, what is the probability that this is true? P(G)=0.00001, P (G ) 0.99999 , P(T|G)=0.95, P(T | G ) 0.005 P(T | G ) P(G ) P(G | T ) P(T | G ) P(G ) P(T | G ) P(G ) 0.95 0.00001 7 - 11 0.002 0.95 0.0001 0.005 0.99999 Random Variables • A random variable is a function X: (sample space) Rn – Talk about the probabilities that are related to the event space • A discrete random variable is a function X: (sample space) S where S is a countable subset of R. • If X: (sample space) {0,1}, then X is called a Bernoulli trial. • The probability mass function (pmf) for a random variable X gives the probability that the random variable has different numeric values. pmf where p( x) P( X x) P( Ax ) Ax { : X ( ) x} 7 - 12 Events: toss two dice and sum their faces ={(1,1), (1,2), …, (1,6), (2,1), (2,2), …, (6,1), (6,2), …, (6,6)} S={2, 3, 4, …, 12} X: S First die 2 Second die 3 4 5 1 6 6 7 8 9 10 11 12 5 6 7 8 9 10 11 4 5 6 7 8 9 10 3 4 5 6 7 8 9 2 3 4 5 6 7 8 1 2 3 4 5 6 7 x 2 3 4 5 6 p(X=x) 1/36 1/18 1/12 1/9 7 5/36 1/6 8 9 5/36 1/9 10 11 12 1/12 1/18 1/36 1 pmf p(3)=p(X=3)=P(A3)=P({(1,2),(2,1)})=2/36 where A3={: X()=3}={(1,2),(2,1)} 7 - 13 Expectation • The expectation is the mean (, Mu) or average of a random variable. E ( X ) xp( x ) x • Example – Roll one die and Y is the value on its face 6 6 1 21 1 E (Y ) xp( y) y 3 6 y 1 6 2 y 1 E ( g (Y )) g ( y ) p( y ) y E(aY+b)=aE(Y)+b E(X+Y)=E(X)+E(Y) 7 - 14 Variance • The variance (2, Sigma ) of a random variable is a measure of whether the values of the random variable tend to be consistent over trials or to vary a lot. 2 Var( X ) E (( X E ( X )) 2 ) E ( X 2 ) E 2 ( X ) • standard deviation () – square root of the variance 7 - 15 X: a random variable that is the sum of the numbers on two dice First die 2 Second die 3 4 5 1 6 6 7 8 9 10 11 12 5 6 7 8 9 10 11 4 5 6 7 8 9 10 3 4 5 6 7 8 9 2 3 4 5 6 7 8 1 2 3 4 5 6 7 x 2 3 4 5 6 p(X=x) 1/36 1/18 1/12 1/9 7 5/36 1/6 8 9 5/36 1/9 10 11 12 1/12 1/18 1/36 1 E( X ) 2 1 1 1 1 5 1 5 1 1 1 1 3 4 5 6 7 8 9 10 11 12 7 36 18 12 9 36 6 36 9 12 18 36 1 1 E ( X ) E (Y Y ) E (Y ) E (Y ) 3 3 7 2 2 var( X ) E (( X E ( X )) 2 ) p( x)( x E ( X )) 2 5 x X: toss two dice, and sum of their faces, I.e., X=Y+Y Y: toss one die, and its value 5 6 E ( g (Y )) g ( y ) 7p-(16 y) y Joint and Conditional Distributions • More than one random variable can be defined over a sample space. In this case, we talk about a joint or multivariate probability distribution. • The joint probability mass function for two discrete random variables X and Y is: p(x, y)=P(X=x, Y=y) • The marginal probability mass function totals up the probability masses for the values of each variable separately. p X ( x) p( x, y) p Y ( y ) p ( x, y ) y x • If X and Y are independent, then p( x, y) pX ( x) pY ( y) 7 - 17 Joint and Conditional Distributions • Similar intersection rules hold for joint distributions as for events. p X |Y ( x | y) p ( x, y ) pY ( y ) • Chain rule in terms of random variables p( w, x, y, z ) p( w) p( x | w) p( y | w, x) p( z | w, x, y ) 7 - 18 Estimating Probability Functions • What is the probability that sentence “The cow chewed its cud” will be uttered? Unknown P must be estimated from a sample of data. • An important measure for estimating P is the relative frequency of the outcome, i.e., the proportion of times a certain outcome occurs. C (u ) fu N • Assuming that certain aspects of language can be modeled by one of the well-known distribution is called using a parametric approach. • If no such assumption can be made, we must use a non-parametric approach or distribution-free 7 - 19 approach. parametric approach • Select an explicit probabilistic model • Specify a few parameters to determine a particular probability distribution • The amount of training data required is not great, and can be calculated to make good probability estimates 7 - 20 Standard Distributions • In practice, one commonly finds the same basic form of a probability mass function, but with different constants employed. • Families of pmfs are called distributions and the constants that define the different possible pmfs in one family are called parameters. • Discrete Distributions: the binomial distribution, the multinomial distribution, the Poisson distribution. • Continuous Distributions: the normal distribution, the standard normal distribution. 7 - 21 Binomial distribution • A series of trials with only two outcomes, each trial is independent from all others • The number r of successes out of n trials given that the probability of success in any trial is p: variable b( r; n, p) nr p r (1 p)nr n r n! (n r)! r! expectation: np parameters 0r n variance: np(1-p) 7 - 22 7 - 23 The normal distribution • two parameters for mean and standard deviation , the curve is given by 1 ( x ) 2 /( 2 2 ) n( x; , ) e 2 • standard normal distribution – =0, =1 7 - 24 7 - 25 Baysian Statistics I: Bayesian Updating • frequentist statistics vs. Bayesian statistics – Toss a coin 10 times and get 8 heads 8/10 (maximum likelihood estimate) – 8 heads out of 10 just happens sometimes given a small sample • Assume that the data are coming in sequentially and are independent. • Given an a priori probability distribution, we can update our beliefs when a new datum comes in by calculating the Maximum A Posteriori (MAP) distribution. • The MAP probability becomes the new prior and the process repeats on each new datum. 7 - 26 Frequentist Statistics m : the model that asserts P(head)=m s: a particular sequence of observations yielding i heads and j tails P( s | m ) mi (1 m) j Find the MLE (maximum likelihood estimate) arg max P(s | ) m m by differentiating the above polynomial. i i j 8 heads and 2 tails: 8/(8+2)=0.8 f (m) mi (1 m) j f ' (m) i mi 1 (1 m) j j mi (1 m) j 1 (i mi 1 (1 m) j mi ) (1 m) j 1 (i mi 1 i m i j mi ) (1 m) j 1 (i (i j )m) m i 1 (1 m) j 1 7 - 27 belief in the fairness of a coin: Bayesian Statistics a regular, fair one a priori probabilistic distribution: P( m ) 6m(1 m) P( s | m ) mi (1 m) j a particular sequence of observations: i heads and j tails New belief in the fairness of a coin?? P ( s | m ) P( m ) P ( m | s ) P( s ) m (1 m) 6m(1 m) P( s ) i j 6mi 1 (1 m) j 1 P( s ) arg max P( m | s) m f (m) 6m i 1 (1 m) j 1 f ' (m) 6(i 1)m i (1 m) j 1 6( j 1)m i 1 (1 m) j 6m i (1 m) j (i 1)(1 m) ( j 1)m i 1 m i j2 new priori arg max P( m | s ) m (when i=8,j=2) 3 4 7 - 28 Bayesian Statistics II: Bayesian Decision Theory • Bayesian Statistics can be used to evaluate which model or family of models better explains some data. • We define two different models of the event and calculate the likelihood ratio between these two models. 7 - 29 Entropy • The entropy is the average uncertainty of a single random variable. • Let p(x)=P(X=x); where x X. Toss two coins and count the number of heads p(0)=1/4, p(1)=1/2, p(2)=1/4 • H(p)= H(X)= - xX p(x)log2p(x) • In other words, entropy measures the amount of information in a random variable. It is normally measured in bits. 7 - 30 Example • Roll an 8-sided die 8 8 1 1 1 H ( X ) p(i ) log p(i ) log( ) log log 8 3bits 8 8 i 1 i 1 8 • Entropy (another view) – the average length of the message needed to transmit an outcome of that variable 1 2 3 4 5 6 7 8 001 010 011 100 101 110 111 000 – Optimal code to send a message of probability p(i): log p(i) 7 - 31 Example • Problem: send a friend a message that is a number from 0 to 3. How long a message must you send ? (in terms of number of bits) • Example: watch a house with two occupants. Case Situation Probability1 Probability2 0 no occupants 0.25 0.5 1 1st occupant 0.25 0.125 2 2nd occupant 0.25 0.125 3 both occupant 0.25 0.25 0 1 2 3 Probability Code Probability2 Code 0.25 00 0.5 0 0.25 01 0.125 110 0.25 10 0.125 111 0.25 11 0.25 10 7 - 32 • Variable-length encoding 0-No occupants 10-both occupants 110-First occupant 111-Second occupant • Code tree: – (1) all messages are handled. – (2) when one message ends and the next starts. 1 1 1 1 *1bit * 2bits * 3bits * 3bits 1.75bits ( 2bits ) 2 4 8 8 – Fewer bits for more frequent messages more bits for less frequent messages. 7 - 33 W : random variable for a message V(W) : possible messages P : probability distribution • lower bound on the number of bits needed to encode such a message: H (W ) P ( w) log P ( w) wV (W ) (entropy of a random variable) H (Wocc ) 1 1 1 1 1 1 1 1 ( log log log log ) 2 2 4 4 8 8 8 8 1 1 1 1 ( ( 1) ( 2 ) ( 3) ( 3)) 2 4 8 8 1.75 P ( w)bits required ( w) wV (W ) P ( w)( log wV (W ) P ( w)) 7 - 34 • (1) entropy of a message: – lower bound for the average number of bits needed to transmit that message. • (2) encoding method: – using -log P(w) bits • entropy (another view) – a measure of the uncertainty about what a message says. – fewer bits for more certain message more bits for less certain message 7 - 35 H ( p) H ( X ) p( x ) log 2 p( x ) xX 1 H ( X ) p( x ) log p( x ) xX E ( X ) xp( x ) x 1 H ( X ) E log p( X ) 7 - 36 Simplified Polynesian • p 1/8 t 1/4 k 1/8 a 1/4 i 1/8 u 1/8 a 01 i 110 u 111 • per-letter entropy H ( P) P(i) log P(i) i{ p ,t , k , a ,i ,u } 1 1 1 1 [4 log 2 log ] 8 8 4 4 1 2 bits 2 • p 100 t 00 k 101 Fewer bits are used to send more frequent letters. 7 - 37 Joint Entropy and Conditional Entropy • The joint entropy of a pair of discrete random variables X, Y ~ p(x,y) is the amount of information needed on average to specify both their values. H ( X , Y ) p( x, y ) log p( x, y ) xX yY • The conditional entropy of a discrete random variable Y given another X, for X, Y ~ p(x,y), expresses how much extra information you still need to supply on average to communicate Y given that the other party knows X. H (Y | X ) p( x )H (Y | X x ) xX p( x ) p( y | x ) log p( y | x ) xX yY p( x, y ) log p( y | x ) xX yY 7 - 38 Chain rule for entropy H ( X , Y ) H ( X ) H (Y | X ) H ( X 1,..., X n ) H ( X 1 ) H ( X 2 | X 1 ) ... H ( X n | X 1,..., X n1 ) Proof: H ( X , Y ) E p ( x , y ) (log p( x, y )) E p ( x , y ) (log( p( x) p( y | x)) E p ( x , y ) (log p( x) log p( y | x)) E p ( x , y ) (log p( x)) E p ( x , y ) (log p( y | x)) H ( X ) H (Y | X ) 7 - 39 Simplified Polynesian revised • Distinction between model and reality • Simplified Polynesian has syllable structure • All words are consist of sequences of CV (consonant-vowel) syllables. • A better model in terms of two random variables C and V. 7 - 40 t k 1 16 3 8 1 16 1 16 0 3 16 3 16 1 16 P(C,•) 1 8 3 4 1 8 p t k a i u 1 16 3 1 16 1 1 8 1 8 P(V,C) v o w e l consonant a i u p 8 P(•,V) 1 2 1 4 1 4 0 4 1 1 3 3 1 1 1 3 H (C ) ( log log log ) 2 3 (2 log 3) 8 8 4 4 8 8 8 4 9 3 7 - 41 log 3bits 1.061bits 4 4 H (V | C ) p(C c) H (V | C c) c p ,t , k 1 1 1 3 1 1 1 1 1 1 H ( , ,0) H ( , , ) H ( ,0, ) 8 2 2 4 2 4 4 8 2 2 1 1 1 1 3 1 16 16 16 16 (log ) (log ) 0 1 1 1 1 8 4 8 8 8 8 1 1 1 1 1 16 16 16 16 1 (log 1 ) 0 1 (log 1 ) 8 8 8 8 8 3 8 (log 3 4 3 3 8 ) 16 (log 3 3 4 4 3 3 16 ) 16 (log 3 3 4 4 3 16 ) 3 4 1 3 1 1 1 1 1 1 H (C,V ) H (C ) H (V | C ) 8 4 2 2 2 8 1.061bits 1.375bits 2.44bits 11 bits 1.375bits 8 7 - 42 p t k a i u 1 16 3 1 16 1 1 8 1 8 8 4 1 1 3 3 1 1 1 1 1 1 1 1 log log log log log log ) 16 16 8 8 16 16 4 4 8 8 8 8 2 1 3 3 1 1 2 1 ( log log log log ) 16 16 8 8 4 4 8 8 8 3 2 6 (3 log 3) 16 8 4 8 3.2825 H ( P ) ( p t k 1 8 1 1 8 1 H ( P) 2 bits 2 4 a 1 4 i u 1 8 1 8 1 H (2 P) 2 2 bits 5bits 2 7 - 43 Short Summary • Better understanding means much less uncertainty (2.44bits < 5 bits) • Incorrect model’s cross entropy is larger than that of the correct model (W1,n ) (W1,n , M ) def (W1, n , M ) P ( w1, n ) log M ( w1, n ) w1, n (W1, n ) : correct model M (W1, n ) : approximate model 7 - 44 Entropy rate: per-letter/word entropy • The amount of information contained in a message depends on the length of the message 1 1 H rate H ( X 1n ) p( x1n ) log p( x1n ) n n x1n • The entropy of a human language L 1 H rate( L) lim H ( X 1 , X 2 ,..., X n ) n n 7 - 45 Mutual Information • By the chain rule for entropy, we have H(X,Y) = H(X)+ H(Y|X) = H(Y)+H(X|Y) • Therefore, H(X)-H(X|Y)=H(Y)-H(Y|X) • This difference is called the mutual information between X and Y. • It is the reduction in uncertainty of one random variable due to knowing about another, or, in other words, the amount of information one random variable contains about another. 7 - 46 H(X,Y) H(X|Y) I(X;Y) H(X) I ( X ;Y ) H ( X ) H ( X | Y ) H(Y|X) H(Y) p X ( x) p( x, y) y p ( y ) p ( x, y ) H ( X ) H (Y ) H ( X , Y ) 1 1 p( x ) log p( y ) log p( x, y ) log p( x, y ) p( x ) y p( y ) x , y x Y p ( x, y ) p( x, y ) log p( x ) p( y ) x, y x 7 - 47 It is 0 only when two variables are independent, but for two dependent variables, mutual information grows not only with the degree of dependence, but also according to the entropy of the variables Conditional mutual information I ( X ;Y ) H ( X ) H ( X | Y ) I ( X ;Y | Z ) I (( X ;Y ) | Z ) H ( X | Z ) H ( X | Y , Z ) Chain rule for mutual information I ( X 1n ; Y ) I ( X 1; Y ) I ( X 2 ; Y | X 1 ) ... I ( X n ; Y | X 1 ,..., X n1 ) n I ( X i ;Y | X 1 ,..., X i 1 ) i 1 7 - 48 Pointwise Mutual Information p ( x, y ) I ( x, y ) log p( x ) p( y ) Applications: clustering words word sense disambiguation 7 - 49 • Clustering by Next Word (Brown, et al., 1992) 1. Each word was characterized by the word that immediately followed it. c(wi) ==<|w1|,|w2|, ...,|ww|> wj total … wi wj … in the corpus 2. Define the distance measure on such vectors. mutual information I(x, y) the amount of information one outcome gives us about the other def I(x, y) == (-log P(x)) (-log P(x|y)) x uncertainty == log x gives y uncertainty P ( x, y ) P( x) P( y ) certainty (x gives y) 7 - 50 • Example. How much information the word “pancake” gives us about the following word “syrup” P( pancake, syrup ) I ( pancake; syrup ) log P( pancake) P( syrup ) 7 - 51 Physical meaning of MI (1) wi and wj have no particular relation to each other. I(x; y) 0 P(wj| wi) = P(wj) x與y相互之間不會給對方信息 I ( wi ; w j ) log log log P ( wi , w j ) P ( wi ) P( w j ) P ( w j | wi ) P( w j ) P( w j ) P( w j ) 0 7 - 52 Physical meaning of MI (2) wi and wj are perfectly coordinated. i j P ( w , w ) i j I ( w ; w ) log P ( wi ) P( w j ) P ( wi ) log P ( wi ) P ( w j ) 1 log a very large number j P( w ) I(x; y) >> 0 當知道y後,提供很多有關x的信息 7 - 53 Physical meaning of MI (3) wi and wj are negatively correlated a very small negative number I(x; y) << 0 王不見王 7 - 54 The Noisy Channel Model • Assuming that you want to communicate messages over a channel of restricted capacity, optimize (in terms of throughput and accuracy) the communication in the presence of noise in the channel. n o i s y c h a n n e l W Message from a finite alphabet Encoder X Input to channel Channel p(y|x) Y Decoder Output from channel W Attempt to reconstruct message based on output • A channel’s capacity can be reached by designing an input code that maximizes the mutual information between the input and output over all possible input distributions. C max I ( X ; Y ) p( X ) 7 - 55 A binary symmetric channel. A 1 or 0 in the input gets flipped on transmission with probability p. 0 1-p 0 p I(X;Y) = H(Y) – H(Y|X) = H(Y) – H(p) max I ( X ; Y ) 1 H ( p) 1 1 1- p The channel capacity is 1 bit only if the entropy H(p) is 0. I.E., If p=0 the channel reliability transmits a 0 as 0 and 1 as 1. If p=1 it always flips bits p( X ) H(p)=-p(x)log2p(x) The channel capacity is 0 when both 0s and 1s are transmitted with equal probability as 0s and 1s (i.e., p=1/2). 1 1 1 1 log log 1 2 2 2 2 Completely noisy binary channel 7 - 56 The noisy channel model in linguistics I Noisy Channel p(o|i) O I Decoder p (i ) p (o | i ) I arg max p(i | o) arg max arg max p(i ) p(o | i ) p(o) i i i p(i): language model p(o|i): channel probability 7 - 57 Speech Recognition • Find the sequence of words W1,n that maximizes P(W1,n w1,n | Speech Signal) P(W1,n ) P(Speech Signal | W1,n ) P( Speech Signal) • Maximize P( W1,n) P(Speech Signal |W1,n ) language model signal acoustic aspects of speech W1 W2 W3 a1 b1 c1 c2 a2 a3 a2 b1 c4 b2 c3 c4 P(a2 , b1 , c4 ) P(a2 , b1 , c4 | Speech Signal) 7 - 58 The big dog... pig Assume P(big | the) = P(pig | the). P(the big dog) = P(the)P(big | the)P(dog | the big) P(the pig dog) = P(the)P(pig | the)P(dog | the pig) P(dog | the big) > P(dog | the pig) “the big dog” is selected. => “dog” selects “big” 7 - 59 7 - 60 Relative Entropy or Kullback-Leibler Divergence • For 2 pmfs, p(x) and q(x), their relative entropy is: p( x) D( p || q) p( x) log q( x) xX • The relative entropy (also known as the KullbackLeibler divergence) is a measure of how different two probability distributions (over the same event space) are. • The KL divergence between p and q can also be seen as the average number of bits that are wasted by encoding events from a distribution p with a code based on a not-quite-right distribution q. D( p || q ) 0 i.e., no bits are wasted when p=q 7 - 61 p( X ) D( p || q) E p log q( X ) D( p || q) p( x) log xX Recall p( x) q( x) E ( X ) xp( x ) x A measure of how far a joint distribution is from independence: I ( X ;Y ) D( p( x, y ) || p( x ) p( y )) I ( X ; Y ) p( x, y ) log x, y p ( x, y ) p( x) p( y ) Conditional relative entropy: p( y | x ) D ( p( y | x ) || q( y | x )) p( x ) p( y | x ) log q( y | x ) x y Chain rule for relative entropy: D( p( x, y ) || q( x, y )) D( p( x) || q( x)) D( p( y | x) || q( y | x)) Application: measure selectional preferences in selection 7 - 62 The Relation to Language: Cross-Entropy • Entropy can be thought of as a matter of how surprised we will be to see the next word given previous words we already saw. • The cross entropy between a random variable X with true probability distribution p(x) and another pmf q (normally a model of p) is given by: H ( X , q) H ( X ) D( p || q) p( x) log q( x) x 1 E p (log ) q( x) H ( X ) D( p || q ) 1 p ( x) p ( x) log p ( x) log p( x) q ( x) 1 p ( x) p ( x) log ( log ) p ( x) q( x) • Cross-entropy can help us find out what our average surprise for the next word is. 7 - 63 Cross Entropy • Cross entropy of a language L=(Xi)~p(x) 1 H ( L, m) lim p( x1n ) log m( x1n ) n n x1n • The language is “nice” 1 H ( L, m) lim log m( x1n ) n n • Large body of utterances available 1 H ( L, m) log m( x1n ) n 7 - 64 How much good does the approximate model do ? (W1,n ) (W1,n , M ) def (W1, n , M ) P ( w1, n ) log M ( w1, n ) w1, n (W1, n ) : correct model M (W1, n ) : approximate model 7 - 65 (W1,n ) (W1,n , M ) proof: log e X X 1 n x i 1 i ….(1) n y 1 i 1 i 1 ….(2) (correct model) (approximate model) n yi xi log 2 ( ) xi i 1 1 log e 2 y=x-1 n i 1 xi log e ( n 1 log e 2 1 log e 2 n i 1 xi ( (y i 1 i yi ) xi yi 1) xi xi ) y log e x … (1) e … (2) 0 7 - 66 n xi log 2 ( i 1 n yi ) xi n xi log 2 yi xi log 2 xi 0 i 1 n x i 1 n i log 2 yi i 1 n x i i 1 log 2 xi n xi log 2 xi xi log 2 yi i 1 i 1 (W1,n ) (W1,n , M ) cross entropy of a language L give a model M def 1 ( w1,n ) log M ( w1,n ) n n w1,n ( L, PM ) lim ( w1,n ) ? (“all” English test) infinite representative samples of English text 7 - 67 H ( L, PM ) 1 lim P(W1,n ) log PM (W1,n ) n n 1 lim log PM (W1,n ) n n 7 - 68 Cross Entropy as a Model Evaluator • Example: – Find the best model to produce message of 20 words. • Correct probabilistic model. Message M1 M2 M3 M4 Probability 0.05 0.05 0.05 0.10 Message M5 M6 M7 M8 Probability 0.10 0.20 0.20 0.25 7 - 69 • Per-word cross entropy 1 * (0.05 log P(M1) + 0.05 log P(M2)+ 20 0.05 log P(M3) + 0.10 log P(M4)+ 0.10 log P(M5) + 0.20 log P(M6)+ 0.20 log P(M7) + 0.25 log P(M8) ) • Approximate model 100 samples M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M3 M3 M3 M3 M3 M4 M4 M4 ... M4 M5 M5 M5 ... M5 M6 M6 M6 ... M6 M7 M7 M7 ... M7 M8 M8 M8 ... M8 (5) (5) (5) (10) (10) (20) (20) (25) Each message is independent of the next 7 - 70 PM = M1 : P(M1) * P(M1) * … * P(M1) 5 times M2 : . . . M8 : P(M2) * P(M2) * … * P(M2) 5 times * ... P(M8) * P(M8) * … * P(M8) 25 times Log(PM ) = 5 log P(M1) + 5 log P(M2) + 5 log P(M3)+ 10 log P(M4) + 10 log P(M5) + 20 log P(M6)+ 20 log P(M7) + 25 log P(M8) 7 - 71 • per-word cross entropy 1 (log PM ) 20 100 1 5 5 5 ( log P ( M 1) log P( M 2) log P( M 3) 20 100 100 100 10 10 20 log P ( M 4) log P( M 5) log P( M 6) 100 100 100 20 25 log P ( M 7) log P( M 8)) 100 100 • the test suite of 100 examples was exactly indicative of the probabilistic model. • To measure the cross entropy of a model works only if the test sequence has not been used by model builder. closed test (inside test) open test (outside test) 7 - 72 • The Composition of Brown and LOB Corpora • Brown Corpus: – Brown University Standard Corpus of Present-day American English. • LOB Corpus: – Lancaster/Oslo-Bergen Corpus of British English. 7 - 73 Text Category Number of Texts Brown LOB A Press:reportage 報導文學 44 44 B Press:editorial 社論 27 27 C Press:reviews 書評 17 17 D Religion 宗教性 17 17 E Skills and hobbies 技藝,商業性,娛樂性 36 38 F Popular lore 名間傳奇 48 44 G Belles lettres,biography,memoirs,etc純文學 75 77 H Miscellaneous(mainly government documents) 雜類(包括政府、出版品、基金會報告等) 30 30 J Learned(including suence and technology) 學術性和科學論文 80 80 K General fiction 一般小說 29 29 L Mystery and detective 神秘及偵探小說 24 24 M Suence fiction 科幻小說 6 6 N Adverture and western fiction 探險及西部小說 29 29 P Romancer and love story 浪漫愛情小說 29 29 R Humour 幽默體 9 9 7 - 74 The Entropy of English • We can model English using n-gram models (also known a Markov chains). • These models assume limited memory, i.e., we assume that the next word depends only on the previous k ones [kth order Markov approximation]. P( X n xn | X n1 xn1 ,..., X 1 x1 ) P( X n xn | X n1 xn1,..., X nk xnk ) 7 - 75 P(W1 w1 ,W2 w2 ,...,Wn wn ) P(W1,n w1,n ) P(w1,n) P( w1 ) P( w2 | w1 ) P( w3 | w1, 2 )...P( wn | w1,n 1 ) bigram: P(W1 w1 ,W2 w2 ,...,Wn wn ) P( w1 ) P( w2 | w1 ) P( w3 | w2 )...P( wn | wn 1 ) trigram: P(W1 w1 ,W2 w2 ,...,Wn wn ) P( w1 ) P( w2 | w1 ) P( w3 | w1, 2 )...P( wn | wn 2,n 1 ) 7 - 76 The Entropy of English • What is the Entropy of English? Model Cross entropy zeroth order 4.76 first order 4.07 second order 208 Shannon's experiment 1.3(1.34) (bits) (uniform model, so log27) (Cover and Thomas 1991:140) 7 - 77 Perplexity • A measure related to the notion of cross-entropy and used in the speech recognition community is called the perplexity. perplexity ( x1n , m) 2 H ( x1n ,m ) m( x1n ) 1 n • A perplexity of k means that you are as surprised on average as you would have been if you had had to guess between k equiprobable choices at each step. 7 - 78