* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Indexing for Searching - UNC School of Information and Library
Survey
Document related concepts
History of genetic engineering wikipedia , lookup
Site-specific recombinase technology wikipedia , lookup
Public health genomics wikipedia , lookup
Gene nomenclature wikipedia , lookup
Epigenetics of human development wikipedia , lookup
Gene desert wikipedia , lookup
Gene expression programming wikipedia , lookup
Genome (book) wikipedia , lookup
Biology and consumer behaviour wikipedia , lookup
Microevolution wikipedia , lookup
Gene expression profiling wikipedia , lookup
Transcript
Major Issues Information is mostly online Information is increasing available in full-text (full-content) There is an explosion in the amount of information being produced. 1 Manual Indexing Human’s read and index content. Fairly good, although not consistent (interobserver, or even intraobserver). Certain fields support costly manual indexing (primary example is Medline). 2 Major Issues For all fields unable to afford manual indexing, and even for biomedical, because there is so much knowledge in the huge amount of literature being produced that we cannot keep track of it, or utilize it. Research Example: Swanson’s undiscovered literature 3 What this means Need ways to index without requiring paid experts – Automatic indexing, classification, keyword extraction, and even relationship and fact extraction. – Need to take advantage of experts who are reading the materials to comment on it and provide rankings, summarizations, keywords, “factoids”. (like Amazon) 4 Why Automatic Classification? Classification is time consuming and expensive Knowledge structuring – To much information Status of automatic classification – Approaching level of human indexing. (NLM’s Metamap). 5 What is Automatic Classification? Automatic manipulation of a document’s contents to support logical grouping with other similar documents for organization and/or retrieval activities. Can include the assignment of, or manipulation of, classification notation. 6 Approaches and Methods Initial approach – Create an inverted file – On-the-fly (natural language processing) Methods – All words, remove stop words – Word frequencies (Wilson’s objective method of determining aboutness) – More sophisticated IR methods • Semantic/linguistical analysis, • co-occurrence/similarity measures, etc. 7 Simple automatic indexes Inverted file: contains all the index terms automatically drawn from the document records according to the indexing technique used. – Position of term - record number Field number Number of occurrences Position in the field (digits 45-57) 8 Pros and Cons of Automatic Indexing Pros – Consistency – Cost reduction – Time reduction Cons / limitations – – – – Human intellect Term relationships Misleading in retrieval Good algorithms, but generally domain-specific 9 How to gauge effectiveness? Recall Number of relevant documents retrieved out of all the possible relevant documents in system. [quantity—did you get it all?] Precision Percentage of documents retrieved that were relevant [quality of what you found] 10 Tradeoff between Recall and Precision We can easily recall everything that matches a particular text string or pattern; however, we cannot search through all the matching results (too many) We can do an OK job limiting to most relevant, but as we “tune” result to be more relevant, we leave out more and more matching results. 11 Future Search Full text searching of content, and of associated annotations on content, and metadata (including reader rankings, tags, etc). Like Connotea, NeoNote, etc. Faceted based searching (Endeca, e.g. Home Depot, NCSU library). Clustered based searching (Clusty) 12 Study on gene name searching Looks at full text searching Tradeoff between precision and recall (Hemminger 2007). 13 Article Discovery Study Schizophrenia + Schizophrenia Gene Genes Found in Metadata Only Schizophrenia Gene Arabidopsis Gene 172 8.58% 3541 20.63% 2712 8.83% 1671 83.38 % 10125 58.99% 5705 18.57% Genes Found in Metadata and Full-text 161 8.03% 3498 20.38% 22305 72.60% Totals for Found Genes 2004 Genes Found in Fulltext Only 17164 30722 14 Article Review Study Two literature cohorts, – Schizophrenia (Pat Sullivan) – Arabidopsis (Todd Vision) Each cohort had three readers Readers are asked to “review the article and judge its relevance to them as someone new to the gene in this biological setting, trying to build an understanding of the state of knowledge 15 in that research area.” Metadata Articles More Valuable In both cases and for all observers, their mean quality rating values were lower (more useful) for the metadata discovered articles. There were statistically significant differences between the mean quality rating for the metadata discovered articles versus the full-text discovered articles for the both the Arabidopsis and Schizophrenia sets16 at the p < 0.05 level Precision and Recall Schizophrenia Recall Precision Arabidopsis Recall Precision Metadata discovered 15.7% (16.6%) 94.7% 84.1% (84.1%) 100% Full-text only discovered 100% 63.7% 100% 69% 17 Article Features that correlate with Value: Number of Hits The number of hits or matches of the search term within the returned document is a commonly used feature to rank returned articles. To test the value of this feature, the number of hits was correlated with the mean quality ranking for each article (averaged across all observers). The results clearly show a relationship where articles with many matches of the search term, tend to be much more highly valued. 18 Improving Relevance for Metadata Searching Repeating the calculations on the schizophrenia and Arabidopsis article review sets, but limited to only matches with high hit counts (Schizophrenia ≥ 20 hits and Arabidopsis ≥ 15 hits) shows that precision for the full text is now the same (100% in Aradidopsis) or slightly better than that of the metadata retrieved articles (95% versus 94.4% in schizophrenia). However, the number of additional cases discovered by fulltext searching is now only slightly better, finding 5% more cases in schizophrenia and 28% more in Arabidopsis. 19 Conclusions This suggests that rather than accepting metadata searching as a surrogate for fulltext searching, it may be time to make the transition to direct full text searching as the standard. This could be accomplished by using certain features of the full-text article, such as number of hits of the search string or whether the search string is found in the metadata (i.e. our current metadata search) as filters that allow us to increase the precision of our results. (and put the user in 20 control of the filtering).