Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Finding Question-Answer Pairs from Online Forums
Document related concepts
no text concepts found
Transcript
F INDING Q UESTION -A NSWER PAIRS FROM O NLINE F ORUMS O UTLINE Introduction Related Work Algorithms for Question Detection Algorithms for Answer Detection Experiments Conclusions I NTRODUCTION D EFINITION What’s an online forum? A web application It contains valuable user generated content Desire to extract and reuse this content In this paper, we only consider question-answer (QA) pairs from forums. Why mining QA pairs from forums? Essential to many QA services (Yahoo! Answers, Baidu) Questions are the focus of forum discussions : Forum Management Useful for Knowledge Base of Chatbot T HREADS A forum thread Contains an initiating post and a couple of reply posts. Initiating post -> several questions Reply posts -> answers to those questions Has an asynchronous nature : multiple participants to pursue multiple questions in parallel We propose a new method to detect QA pairs Question detection Answer detection D ETECTION Question Detection Objective : Detect all questions in a thread -> Easy ? They are often stated in an informal way -> checking (?) and 5W1H words is not enough Answer Detection Not easy because possible multiple QA pairs in a thread One question may have multiple replies. Which one? Query-document model doesn’t work due to forum-specific features such as distance between QA. Solution : Graph-based approach R ELATED W ORK W ORKS S O FAR No successful works on extracting QA pairs from forum data Closest works : input-reply pairs for chatbot knowledge QA pairs for email summarization Researches done in TREC on short answers for factoid questions Research FAQ and CQA retrieval A LGORITHMS F OR Q UESTION D ETECTION Q UESTION S TRUCTURE 5W1H words and (?) are not adequate 30% questions w/o question marks, 9% sentences ending with question marks are not question Solution : Extract labeled sequential patterns (LSPs) from both questions and nonquestions to characterize them Use the discovered patterns as features to build classifiers for question detection LSP S A LSP, p, is an implication in the form of LHS c, where LHS is a sequence and c is a class label a LSP p₁ is contained by p₂ if the sequence p₁.LHS is contained by p₂.LHS p₁.c = p₂.c The support of p, sup(p), is the percentage of tuples in database D that contain the LSP p The probability of the LSP p being true : “the confidence of p” , where sup(p) is the support of p. M INING LSP S Preprocess each sentence by applying Part-Of-Speech (POS) tagger MXPOST Toolkit to tag each sentence e.g. “where can you find a job“ becomes “where can PRP VB DT NN" Each processed sentence becomes a database tuple The combination of POS tags and keywords allows us to capture representative features for question sentences by mining LSPs <anyone, VB, how> Q <what, do, PRP, VB> Q M INING LSP S C ONT ’ D We will mine LPSs by imposing thresholds Minimum support to ensure that discovered patterns are general Minimum confidence to ensure that all discovered LSPs are discriminating In our experiments, imposed values are 0.5% and 85% respectively Each discovered LSP forms a binary feature as the input for a classification model If a sentence includes a LSP, the corresponding feature is set at 1 We build a classifier to detect questions using Ripper classification algorithm A LGORITHMS F OR A NSWER D ETECTION P RELIMINARY Three IR methods to rank candidate answers for a given forum question Cosine Similarity Query likelihood language model KL-divergence language model Classification based re-ranking First method : Classifiers are built to extract input-response pairs using content features (e.g. the number overlapping words between input and reply post ) and structural features (e.g. is the reply posted by the thread starter) Second method : We can treat each question and candidate answer pair as an instance, compute features for the pair, and train a classifier. The resulting classification scores can be used to rank the candidate answers of a question G RAPH B ASED P ROPAGATION M ETHOD It considers the inter-relationships of candidate answers of which previous methods do not make use It’s been successful in Web search where links are usually obvious Idea behind it: If a candidate answer is related to an authoritative candidate answer with high score, the candidate answer, which may not have a high score, is also likely to be an answer. B UILDING G RAPHS We build a weighted directed graph given a question q and its candidate answer set Use KL-divergence language model to determine the edge between two candidates answers and The problem : how to generate the edge set E KL-divergence captures the asymmetry (opposite in cosine similarity) An edged is formed from ( offspring) to (generator) if the value is larger than a threshold θ (determined empirically) Our methods are not sensitive to θ because we allow self-loop i.e. each candidate answer can be its own generator. W EIGHT C OMPUTATION Straightforward way : use KL-divergence score. We consider two more factors in computing weight to achieve better performance Distance between question and answer denoted by Authority level of authors denoted by The weight for edge Normalized C OMPUTING P ROPAGATED S CORES Propagation w/o initial scores The product of the authority value and the initial ranking score (from previous slide) between candidate answer a and question q will be returned as the final ranking score for a. How to compute authority score Propagation with initial scores , where λ determined empirically is a trade-off between a and its offsprings I NTEGRATION WITH O THER M ETHODS When training data is available the graph-based model can be integrated with classification model in two ways For each candidate QA pair, the results of graph-based methods can be added to classification method to determine if the candidate answer is an answer of the question (ranking) The classification score returned by a classifier is often (or can be transformed into) the probability for a candidate answer being a true answer and can be used as initial score for propagation of graph-based model E XPERIMENTS S OURCE D ATA Data is selected from 3 forums of different scales TripAdvisor -> 1,212,153 threads LonelyPlanet -> 86,772 threads BootsnAll Network -> 25,298 threads Each thread in our corpus contains at least 2 posts (4.46 on avg) 2 annotators tag questions and their answers in each thread with kappa statistic 0.96 Two datasets are generated from the union of the two annotated data for question detection (Q-TUnion and Q-TInter) Five datasets for answer detection E VALUATION Q UESTION D ETECTION ON The result are obtained through 10-fold cross-validation SM is simple rules and the method in a previous related work. E VALUATION ON A NSWER I DENTIFICATION C ONT ’ D The methods and their abbreviations E VALUATION Q UESTION D ETECTION ON Results on A-TUnion data Graph-based methods signficantly outperform their respective counterparts in terms of all the three measures as expected The overall performance deteriorated due to the questions without answers C ONCLUSIONS The extracted QA pairs could be used to enrich the knowledge base of community based QA service, instant answer service and Chatbot Experimental results on real data show that our approach is effective Future work: Detecting questions without answers Revisiting more TREC QA techniques to see if they can help answer detection in forums Modelling the relationship of questions in the same thread to improve answer detection Investigating the effectiveness of our techniques in passage retrieval on TREC QA data R EFERENCES [1] Gao Cong , Long Wang, Chin-Yew Lin, Young-In Song, Yongheng Sun. Finding Question-Answer Pairs from Online Forums. In SIGIR, 2008
