Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
NSF-ULA Sense tagging and Eventive Nouns Martha Palmer, Miriam Eckert, Jena D. Hwang, Susan Windisch Brown, Dmitriy Dligach, Jinho Choi, Nianwen Xue University of Colorado Departments of Linguistics and Computer Science Institute for Cognitive Science 1 OntoNotes http://www.bbn.com/NLP/OntoNotes Participating Sites: BBN University of Colorado University of Pennsylvania USC/ISI 2 OntoNotes goals Develop a skeletal representation of the literal meaning of sentences – Add to a frame-based (PropBank) representation of predicates and their arguments: • Referring expressions and the textual phrases they refer to • Terms disambiguated by coarse-grained word sense in an ontology – Encodes the core, skeletal meaning – Moves away from strings to terms that a reasoning system can use Find a “sweet spot” in space of – Inter-tagger agreement – Productivity – Depth of representation 3 Text Treebank Word Sense wrt Ontology PropBank OntoNotes Annotated Text Co-reference Creating a Sense Inventory that Supports High Quality Annotation A large scale annotation effort as part of the OntoNotes project Two Steps – Grouping subtle, fine-grained WordNet senses into coherent semantic sense groups based on syntactic and semantic criteria For Example: WordNet Sense1: I called my son David WordNet Sense 12: You can call me Sir are grouped together – Annotation 4 Example Grouping: Order Group Characteristics Examples 1 “give a command” NP1[+human] ORDER NP2[+animate] to V The victim says that the owner ordered the dogs to attack. where NP1 has some authority over NP2 2 “request something to be made, supplied, or delivered" 3 “organize” NP1[+human] ORDER NP2 I just ordered pizza from panhandle pizza. I ordered the papers before the meeting, –5 5 Annotation Process 6 Verb sense groups are created based on WordNet senses and online resources (VerbNet, PropBank, FrameNet, dictionaries) Newly created verb sense groups are subject to sampleannotation Verbs higher than 90% ITA (or 85% after regrouping) go to actual annotation Verbs with less that 90% ITA are regrouped and sent back into sample-annotation tasks. Regroupings and resample annotations are not done by the original grouper and taggers. Verbs that complete actual annotation are adjudicated The Grouping and Annotation Process 7 WSD with English OntoNotes Verbs Picked 217 sense group annotated verbs with 50+ instances each (out of 1300+ verbs) – 35K instances total (almost half the data) – WN polysemy range: 59 to 2; Coarse polysemy range: 16 to 2 – Test: 5-fold cross-validation – Automatic performance approaches human performance! 8 WN Avg. Polysemy Onto Avg. Baseline Polysemy 10.4 5.1 0.68 ITA MaxEnt 0.825 0.827 SVM 0.822 ITA > System Performance 1 0.9 0.8 0.7 0.6 ITA 0.5 Baseline 0.4 Performance 0.3 0.2 0.1 0 form 9 count deal order ITA < System Performance 1 0.9 0.8 0.7 0.6 ITA 0.5 Baseline 0.4 Performance 0.3 0.2 0.1 0 decide 10 keep throw mean Discussion Coarse-grained sense distinctions improve both ITA and system performance Data set Baseline Acc. System Acc. ITA SENSEVAL-2 verbs 0.407 0.646 0.713 OntoNotes verbs 0.680 0.827 0.825 Linguistically motivated features contributed to high system accuracy 11 ALL w/o SEM. w/o SEM+SYN 0.827 0.816 0.789 Eventive nouns ISI sense tags nouns Some nouns have eventive senses – party – development Given a list of nouns and tagged instances, we nombank just those. A few thousand at most. Last meeting we reported very poor ITA with Adam’s nombank annotation 12 Comparison of NomBank and PropBank Frames 107 Frames examined: the ISI eventive nouns that have frame files in NomBank 47 of those showed differences between the NomBank and the PropBank frames. Types of differences: 13 1. No PropBank equivalent 2. No PropBank equivalent for some NomBank senses 3. No NomBank equivalent for some PropBank senses 4. NomBank equivalent has extra Args No PropBank equivalent 15 cases breakdown; downturn; illness; oath; outcome; pain; repercussion; stress; transition; turmoil; unrest 14 No related PropBank equivalent for some NomBank senses 7 cases No PB equivalent start start.02 “attribute/housing-starts” PB equivalent of unrelated name appointment appointment.02 “have a date” Equivalent: meet.03, no equivalent sense of “to appoint” • PB equivalent of related name has different sense numbering plea plea.02 “beg” Equivalent: plead.01, source is listed as appeal.02 15 solution solution.02 “mix, combine”; source mix.01 Arg0: agent, mixer Arg1: ingredient one Arg2: ingredient two Related PB equivalent: dissolve.01 “cause to come apart”?? Arg0: causer, agent Arg1: thing dissolving Arg2: medium “salt water solution” “rubber solution” “chemical solution” 16 No related NomBank equivalent for some PropBank rolesets 10 cases harassment harassment.01 “bother” Source: harass.01 “bother” “Police and soldiers continue to harass Americans.” “the harassment of diplomats and their families” harass.02 “cause an action” “John harassed Mary into giving him some ice cream.” No NB equivalent. 17 NomBank equivalent has extra Args 6 cases allegation PB: allege.01 Arg0: speaker, alleger Arg1: utterance, allegation Arg2: hearer NB: allegation.01 Arg3: person against whom something is alleged “Fraud allegation against Wei-Chyung Wang” “Abuse alleged against accused murderer. Conclusion: Consider adding Arg3 to VN frame. 18 answer PB: answer.01 Arg0: replier Arg1: in response to Arg2: answer NB: answer.01 Arg3: asker/recipient of answer “Wang’s marketing department provided the sales forceArg3 answers to [the] questions” In PropBank, this role is often fulfilled by the Arg1. “’I’ve read Balzac’, he answers criticsArg1” 19 attachment attachment.01 Arg0: agent Arg1: theme Arg2: theme2 Arg3: instrument attach.01 Arg0: agent Arg1: thing being tied Arg2: instrument Attach.01 allows two Arg1’s: John attached the apology noteArg1 to his dissertationArg1. 20 Other issues Roles described differently (different label or true difference) utterance.01: Arg0 “agent” utter.01: Arg0 “speaker” score.01 (VN): Arg2 “opponent” score.01 (NB): Arg2 “test/game” “scored against the B teamArg2”; “high testArg2 scores” Different frame numbers “attach, as with glue” bond.02 (NB) bond.01 (VN) 21 Conclusion We can fix these! 22