Download ppt

CMSC 723 / LING 645: Intro to
Computational Linguistics
September 22, 2004: Dorr
Porter Stemmer,
Intro to Probabilistic NLP
and N-grams (chap 6.1-6.3)
Prof. Bonnie J. Dorr
Dr. Christof Monz
TA: Adam Lee
Computational Morphology
(continued)
 The Rules and the Lexicon
–
–
–
–
General versus Specific
Regular versus Irregular
Accuracy, speed, space
The Morphology of a language
 Approaches
– Lexicon only
– Lexicon and Rules
• Finite-state Automata
• Finite-state Transducers
– Rules only
Lexicon-Free Morphology:
Porter Stemmer
 Lexicon-Free FST Approach
 By Martin Porter (1980)
http://www.tartarus.org/%7Emartin/PorterStemmer/
 Cascade of substitutions given specific conditions
GENERALIZATIONS
GENERALIZATION
GENERALIZE
GENERAL
GENER
Porter Stemmer
 Definitions
– C = string of one or more consonants, where a consonant is anything other
than A E I O U or (Y preceded by C)
– V = string of one or more vowels
– M = Measure, roughly with number of syllables
– Words = (C)*(V*C*)M(V)*
•
•
•
M=0 TR, EE, TREE, Y, BY
M=1 TROUBLE, OATS, TREES, IVY
M=2 TROUBLES, PRIVATE, OATEN, ORRERY
 Conditions
– *S - stem ends with S
– *v* - stem contains a V
– *d - stem ends with double C, e.g., -TT, -SS
– *o - stem ends CVC, where second C is not W, X or Y, e.g., -WIL, HOP
*<S> = ends with <S>
*v* = contains a V
Porter Stemmer
*d = ends with double C
*o = ends with CVC
second C is not W, X or Y
Step 1: Plural Nouns and Third Person Singular Verbs
SSES  SS
IES  I
caresses
ponies
ties
caress
cats
SS  SS
S 





caress
poni
ti
caress
cat
Step 2a: Verbal Past Tense and Progressive Forms
(M>0) EED  EE

i
(*v*) ED
ii
(*v*) ING 
feed  feed, agreed  agree
plastered 
motoring
plaster, bled  bled
 motor, sing  sing
Step 2b: If 2a.i or 2a.ii is successful, Cleanup
AT  ATE
conflat(ed)  conflate
BL  BLE
troubl(ed)  trouble
IZ  IZE
siz(ed)  size
(*d and not (*L or *S or *Z))
 single letter
(M=1 and *o)  E
hopp(ing) 
hop, tann(ed)  tan
hiss(ing)  hiss, fizz(ed)  fizz
fail(ing)  fail, fil(ing)  file
*<S> = ends with <S>
*v* = contains a V
Porter Stemmer
*d = ends with double C
*o = ends with CVC
second C is not W, X or Y
Step 3: Y  I
(*v*) Y  I
happy  happi
sky  sky
Porter Stemmer
Step 4: Derivational Morphology I: Multiple Suffixes
(m>0) ATIONAL ->
(m>0) TIONAL ->
ATE
TION
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
ENCE
ANCE
IZE
ABLE
AL
ENT
E
OUS
IZE
ATE
ATE
AL
IVE
FUL
OUS
AL
IVE
BLE
ENCI
ANCI
IZER
ABLI
ALLI
ENTLI
ELI
OUSLI
IZATION
ATION
ATOR
ALISM
IVENESS
FULNESS
OUSNESS
ALITI
IVITI
BILITI
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
relational
->
conditional
->
rational
->
valenci
->
hesitanci
->
digitizer
->
conformabli
->
radicalli
->
differentli
->
vileli
- >
analogousli
->
vietnamization ->
predication
->
operator
->
feudalism
->
decisiveness
->
hopefulness
->
callousness
->
formaliti
->
sensitiviti
->
sensibiliti
->
relate
condition
rational
valence
hesitance
digitize
conformable
radical
different
vile
analogous
vietnamize
predicate
operate
feudal
decisive
hopeful
callous
formal
sensitive
sensible
Porter Stemmer
Step 5: Derivational Morphology II: More Multiple Suffixes
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
(m>0)
ICATE
ATIVE
ALIZE
ICITI
ICAL
FUL
NESS
->
->
->
->
->
->
->
IC
AL
IC
IC
triplicate
formative
formalize
electriciti
electrical
hopeful
goodness
->
->
->
->
->
->
->
triplic
form
formal
electric
electric
hope
good
*<S> = ends with <S>
Porter Stemmer
*v* = contains a V
*d = ends with double C
*o = ends with CVC
second C is not W, X or Y
Step 6: Derivational Morphology III: Single Suffixes
(m>1) AL
->
(m>1) ANCE ->
(m>1) ENCE ->
(m>1) ER
->
(m>1) IC
->
(m>1) ABLE ->
(m>1) IBLE ->
(m>1) ANT
->
(m>1) EMENT ->
(m>1) MENT ->
(m>1) ENT
->
(m>1 and (*S or *T)) ION ->
(m>1) OU
->
(m>1) ISM
->
(m>1) ATE
->
(m>1) ITI
->
(m>1) OUS
->
(m>1) IVE
->
(m>1) IZE
->
revival
allowance
inference
airliner
gyroscopic
adjustable
defensible
irritant
replacement
adjustment
dependent
adoption
homologou
communism
activate
angulariti
homologous
effective
bowdlerize
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
->
reviv
allow
infer
airlin
gyroscop
adjust
defens
irrit
replac
adjust
depend
adopt
homolog
commun
activ
angular
homolog
effect
bowdler
*<S> = ends with <S>
Porter Stemmer
*v* = contains a V
*d = ends with double C
*o = ends with CVC
second C is not W, X or Y
Step 7a: Cleanup
(m>1) E 
(m=1 and not *o) E 
probate  probat
rate
 rate
cease  ceas
Step 7b: More Cleanup
(m > 1 and *d and *L)
 single letter
controll  control
roll  roll
Porter Stemmer
 Errors of Omission
–
–
–
–
–
European
analysis
matrices
noise
explain
Europe
analyzes
matrix
noisy
explanation
 Errors of Commission
–
–
–
–
–
organization
doing
generalization
numerical
university
From Krovetz ‘93
organ
doe
generic
numerous
universe
Why (not) Statistics for NLP?
Pro
– Disambiguation
– Error Tolerant
– Learnable
Con
– Not always appropriate
– Difficult to debug
Weighted Automata/Transducers
 Speech recognition: storing a pronunciation lexicon
 Augmentation of FSA: Each arc is associated with a
probability
Pronunciation network
for “about”
Noisy Channel
Probability Definitions
 Experiment (trial)
– Repeatable procedure with well-defined possible
outcomes
 Sample space
– Complete set of outcomes
 Event
– Any subset of outcomes from sample space
 Random Variable
– Uncertain outcome in a trial
More Definitions
Probability
– How likely is it to get a particular outcome?
– Rate of getting that outcome in all trials
Probability of drawing a spade from 52 well-shuffled playing cards:
Distribution: Probabilities associated with
each outcome a random variable can take
– Each outcome has probability between 0 and 1
– The sum of all outcome probabilities is 1.
Conditional Probability
What is P(A|B)?
First, what is P(A)?
– P(“It is raining”) = .06
Now what about P(A|B)?
– P(“It is raining” | “It was clear 10 minutes ago”) = .004
P( A, B)
P( A | B) 
P( B)
A
A,B B
Note: P(A,B)=P(A|B) · P(B)
Also: P(A,B) = P(B,A)
Independence
 What is P(A,B) if A and B are independent?
 P(A,B)=P(A) · P(B) iff A,B independent.
– P(heads,tails) = P(heads) · P(tails) = .5 · .5 = .25
– P(doctor,blue-eyes)
= P(doctor) · P(blue-eyes)
= .01 · .2 = .002
 What if A,B independent?
– P(A|B)=P(A) iff A,B independent
– Also: P(B|A)=P(B) iff A,B independent
Bayes Theorem
P( A | B) P( B)
P( B | A) 
P( A)
• Swap the order of dependence
• Sometimes easier to estimate one
kind of dependence than the other
What does this have to do with
the Noisy Channel Model?
(O)
(H)
P(H)
P(O|H)
Best H
Best H = argmax P(H|O) = argmax P(O | H ) P( H )
H
H
P(O)
likelihood prior
Noisy Channel Applied to
Word Recognition
argmaxw P(w|O) = argmaxw P(O|w) P(w)
Simplifying assumptions
– pronunciation string correct
– word boundaries known
Problem:
– Given [n iy], what is correct dictionary word?
What do we need?
[ni]: knee, neat, need, new
What is the most likely
word given [ni]?
 Compute prior P(w)
Word
freq(w)
P(w)
new
2625
.001
neat
338
.00013
need
1417
.00056
knee
61
.000024
 Now compute likelihood P([ni]|w), then multiply
Word
P(O|w)
P(w)
P(O|w)P(w)
new
.36
.001
.00036
neat
.52
.00013
.000068
need
.11
.00056
.000062
knee
1.00
.000024
.000024
Why N-grams?
 Compute likelihood P([ni]|w), then multiply
Word
P(O|w)
P(w)
P(O|w)P(w)
new
.36
.001
.00036
P([ni]|new)P(new)
neat
.52
.00013
.000068
P([ni]|neat)P(neat)
need
.11
.00056
.000062
P([ni]|need)P(need)
knee
1.00
.000024
.000024
P([ni]|knee)P(knee)
Unigram approach: ignores context
Need to factor in context (n-gram)
- Use P(need|I) instead of just P(need)
- Note: P(new|I) < P(need|I)
Next Word Prediction
[borrowed from J. Hirschberg]
From a NY Times story...
– Stocks plunged this ….
– Stocks plunged this morning, despite a cut in
interest rates
– Stocks plunged this morning, despite a cut in
interest rates by the Federal Reserve, as Wall ...
– Stocks plunged this morning, despite a cut in
interest rates by the Federal Reserve, as Wall
Street began
Next Word Prediction (cont)
– Stocks plunged this morning, despite a cut in
interest rates by the Federal Reserve, as Wall
Street began trading for the first time since last …
– Stocks plunged this morning, despite a cut in
interest rates by the Federal Reserve, as Wall
Street began trading for the first time since last
Tuesday's terrorist attacks.
Human Word Prediction
Domain knowledge
Syntactic knowledge
Lexical knowledge
Claim
A useful part of the knowledge needed to
allow Word Prediction can be captured
using simple statistical techniques.
Compute:
– probability of a sequence
– likelihood of words co-occurring
Why would we want to do this?
Rank the likelihood of sequences
containing various alternative alternative
hypotheses
Assess the likelihood of a hypothesis
Why is this useful?
Speech recognition
Handwriting recognition
Spelling correction
Machine translation systems
Optical character recognizers
Handwriting Recognition
Assume a note is given to a bank teller,
which the teller reads as I have a gub. (cf.
Woody Allen)
NLP to the rescue ….
– gub is not a word
– gun, gum, Gus, and gull are words, but gun
has a higher probability in the context of a
bank
Real Word Spelling Errors
 They are leaving in about fifteen minuets to go to
her house.
 The study was conducted mainly be John Black.
 The design an construction of the system will take
more than a year.
 Hopefully, all with continue smoothly in my
absence.
 Can they lave him my messages?
 I need to notified the bank of….
 He is trying to fine out.
For Spell Checkers
Collect list of commonly substituted words
– piece/peace, whether/weather, their/there ...
Example:
“On Tuesday, the whether …’’
“On Tuesday, the weather …”
Language Model
Definition: Language model is a model
that enables one to compute the probability,
or likelihood, of a sentence S, P(S).
Let’s look at different ways of computing
P(S) in the context of Word Prediction
Word Prediction:
Simple vs. Smart
 Simple:
Every word follows every other word w/ equal probability (0-gram)
– Assume |V| is the size of the vocabulary
– Likelihood of sentence S of length n is = 1/|V| × 1/|V| … × 1/|V|
n times
– If English has 100,000 words, probability of each next word is 1/100000 =
.00001
 Smarter:
Probability of each next word is related to word frequency (unigram)
– Likelihood of sentence S = P(w1) × P(w2) × … × P(wn)
– Assumes probability of each word is independent of probabilities of other words.
 Even smarter:
Look at probability given previous words (N-gram)
– Likelihood of sentence S = P(w1) × P(w2|w1) × … × P(wn|wn-1)
– Assumes probability of each word is dependent on probabilities of other words.
Chain Rule
 Conditional Probability
– P(A1,A2) = P(A1) · P(A2|A1)
 The Chain Rule generalizes to multiple events
– P(A1, …,An) =
P(A1) P(A2|A1) P(A3|A1,A2)…P(An|A1…An-1)
 Examples:
– P(the dog) = P(the) P(dog | the)
– P(the dog bites) = P(the) P(dog | the) P(bites| the dog)
Relative Frequencies and
Conditional Probabilities
 Relative word frequencies are better than equal
probabilities for all words
– In a corpus with 10K word types, each word would
have P(w) = 1/10K
– Does not match our intuitions that different words are
more likely to occur (e.g. the)
 Conditional probability more useful than
individual relative word frequencies
– Dog may be relatively rare in a corpus
– But if we see barking, P(dog|barking) may be very
large
For a Word String
 In general, the probability of a complete string of
words w1…wn is:
n
P(w1 )
= P(w1)P(w2|w1)P(w3|w1..w2)…P(wn|w1…wn-1)
n
=  P(wk | w1k 1)
k 1
But this approach to determining the
probability of a word sequence is not very
helpful in general….
Markov Assumption
 How do we compute P(wn|w1n-1)?
Trick: Instead of P(rabbit|I saw a), we use P(rabbit|a).
– This lets us collect statistics in practice
– A bigram model: P(the barking dog) =
P(the|<start>)P(barking|the)P(dog|barking)
 Markov models are the class of probabilistic
models that assume that we can predict the
probability of some future unit without looking
too far into the past
n
n
– Specifically, for N=2 (bigram): P(w1) ≈ Π P(wk|wk-1)
k=1
 Order of a Markov model: length of prior context
– bigram is first order, trigram is second order, …
Counting Words in Corpora
What is a word?
–
–
–
–
–
e.g., are cat and cats the same word?
September and Sept?
zero and oh?
Is seventy-two one word or two? AT&T?
Punctuation?
How many words are there in English?
Where do we find the things to count?
Corpora
 Corpora are (generally online) collections of text
and speech
 Examples:
–
–
–
–
–
–
Brown Corpus (1M words)
Wall Street Journal and AP News corpora
ATIS, Broadcast News (speech)
TDT (text and speech)
Switchboard, Call Home (speech)
TRAINS, FM Radio (speech)
Training and Testing
 Probabilities come from a training corpus, which
is used to design the model.
– overly narrow corpus: probabilities don't generalize
– overly general corpus: probabilities don't reflect task
or domain
 A separate test corpus is used to evaluate the
model, typically using standard metrics
– held out test set
– cross validation
– evaluation differences should be statistically
significant
Terminology
 Sentence: unit of written language
 Utterance: unit of spoken language
 Word Form: the inflected form that appears in the
corpus
 Lemma: lexical forms having the same stem, part
of speech, and word sense
 Types (V): number of distinct words that might
appear in a corpus (vocabulary size)
 Tokens (N): total number of words in a corpus
 Types seen so far (T): number of distinct words
seen so far in corpus (smaller than V and N)
Simple N-Grams
An N-gram model uses the previous N-1
words to predict the next one:
P(wn | wn-N+1 wn-N+2… wn-1 )
–
–
–
–
unigrams: P(dog)
bigrams: P(dog | big)
trigrams: P(dog | the big)
quadrigrams: P(dog | chasing the big)
Using N-Grams
 Recall that
– N-gram: P(wn|w1n-1) ≈ P(wn|wn-1
n-N+1)
– Bigram:
P(wn1)
n
≈Π
P(wk|wk-1)
k=1
 For a bigram grammar
– P(sentence) can be approximated by multiplying all the
bigram probabilities in the sequence
 Example:
P(I want to eat Chinese food) =
P(I | <start>) P(want | I) P(to | want) P(eat | to)
P(Chinese | eat) P(food | Chinese)
A Bigram Grammar Fragment
from BERP
Eat on
.16
Eat Thai
.03
Eat some
.06
Eat breakfast
.03
Eat lunch
.06
Eat in
.02
Eat dinner
.05
Eat Chinese
.02
Eat at
.04
Eat Mexican
.02
Eat a
.04
Eat tomorrow
.01
Eat Indian
.04
Eat dessert
.007
Eat today
.03
Eat British
.001
Additional BERP Grammar
<start> I
<start> I’d
<start> Tell
<start> I’m
.25
.06
.04
.02
Want some
Want Thai
To eat
To have
.04
.01
.26
.14
I want
I would
I don’t
.32
.29
.08
To spend
To be
British food
.09
.02
.60
I have
.04
British restaurant
.15
Want to
Want a
.65
.05
British cuisine
British lunch
.01
.01
Computing Sentence Probability
 P(I want to eat British food) = P(I|<start>)
P(want|I) P(to|want) P(eat|to) P(British|eat)
P(food|British) = .25×.32×.65×.26×.001×.60 =
.000080
 vs. I want to eat Chinese food = .00015
 Probabilities seem to capture “syntactic” facts,
“world knowledge”
– eat is often followed by a NP
– British food is not too popular
 N-gram models can be trained by counting and
normalization
BERP Bigram Counts
I
Want
To
Eat
Chinese
Food lunch
I
8
1087
0
13
0
0
0
Want
3
0
786 0
6
8
6
To
3
0
10
860 3
0
12
Eat
0
0
2
0
19
2
52
Chinese
2
0
0
0
0
120
1
Food
19
0
17
0
0
0
0
Lunch
4
0
0
0
0
1
0
BERP Bigram Probabilities:
Use Unigram Count
Normalization: divide bigram count by
unigram count of first word.
I
Want
To
Eat
Chinese
Food Lunch
3437
1215
3256
938
213
1506
459
Computing the probability of I I
– P(I|I) = C(I,I)/C(I) = 8 / 3437 = .0023
A bigram grammar is an NxN matrix of
probabilities, where N is the vocabulary size
Learning a Bigram Grammar
The formula P(wn|wn-1) = C(wn,wn-1)/C(wn-1) is
used for bigram “parameter estimation”
Relative Frequency
Maximum Likelihood Estimation (MLE):
Parameter set maximizes likelihood of
training set T given model M — P(T|M).
What do we learn about the
language?
 What's being captured
with ...
–
–
–
–
–
P(want | I) = .32
P(to | want) = .65
P(eat | to) = .26
P(food | Chinese) = .56
P(lunch | eat) = .055
 What about...
– P(I | I) = .0023
– P(I | want) = .0025
– P(I | food) = .013
Readings for next time
J&M Chapter 5, 7.1-7.3