Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Collaborative information seeking wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Logic programming wikipedia , lookup
Ecological interface design wikipedia , lookup
Embodied cognitive science wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Personal knowledge base wikipedia , lookup
Anybody can Contribute Knowledge using KnowBuddy We describe how a group of distributed collaborators have used web-based software to economically curate machine understanding of a chapter of a textbook deeply enough that the resulting artificial intelligence can answers more difficult questions than previously demonstrated in systems such as IBM’s Watson, Apple’s Siri, and Stanford Research Institute’s Aura. We discuss how the approach promises to impact the web and transform education, healthcare, and enterprise intelligence. Paul Haley Automata, Inc. [email protected] (412) 741-6420 Google excels at “recall” • Presenting information in response to simple questions is not a problem. • Does Google “know” what a lipid bilayer is? • Could Google teach? What is the structure of an integral protein? • Presenting information is not the same as understanding or answering questions • Google doesn’t have “knowledge”, per se • Google “recalls” information Will blood cells in a hypertonic environment burst? • A question from a college biology textbook • Google finds Yahoo answers to this common question from biology students • But even those findings are indirect and, in some cases, misleading • KnowBuddy answers “no”, correctly Why can’t search engines do better? • Statistical analysis of even vast amounts of text produces only superficial correlations between anything more complex than simple words. • Statistical analysis requires exponentially more data and processing power for each increase in the depth of what it might discover. • Statistical analysis produces information which may be ranked for relevance but which becomes knowledge only when used in reasoning. IBM on Watson vs. search engines • The bottom line is that the Jeopardy! Challenge poses a different kind of problem than what is solved by web search. • It demands that the computer deeply analyze the question to figure out exactly what is being asked, deeply analyze the available content to extract precise answers, and quickly compute a reliable confidence in light of whatever supporting or refuting information it finds. • IBM believes that an effective and general solution to this challenge can help drive the broader impact of automatic question answering in science and the enterprise. IBM on QA using logic or NLP • Classic knowledge-based AI approaches to QA try to logically prove an answer is correct from a logical encoding of the question and all the domain knowledge required to answer it. Such approaches are stymied by two problems: – the prohibitive time and manual effort required to acquire massive volumes of knowledge and formally encode it as logical formulas accessible to computer algorithms, and – the difficulty of understanding natural language questions well enough to exploit such formal encodings if available. • Techniques for dealing with huge amounts of natural language text, such as Information Retrieval, suffer from nearly the opposite problem in that they can always find documents or passages containing some keywords in common with the query but lack the precision, depth, and understanding necessary to deliver correct answers with accurate confidences. The Knowledge Acquisition Bottleneck • IBM is talking about the typically prohibitive cost of knowledge acquisition (KA) • KA cost has come down only for superficial & approximate knowledge using statistical NLP • IBM implies logic for which KA is expensive is beyond the ability of statistical NLP to discern • IBM lacks such knowledge while admitting it is needed to overcome the limitations of statistical NLP Why not QA using logic and NLP? • What if it was “cheap” to acquire massive volumes of knowledge formally encoded as logical formulas? • What if it was “easy” to understand natural language questions well enough to exploit such formal encodings? What does KnowBuddy do? • KnowBuddy allows English text to be precisely understood as formal logic almost as quickly as such text can be authored in the first place. • KnowBuddy makes it easy to understand natural language sentences - including questions, well enough to exploit formal logic. How does KnowBuddy work? • KnowBuddy helps people curate English documents into bodies of knowledge. • KnowBuddy translates knowledge from English into the formal logic to be exploited. • KnowBuddy answers questions by exploiting the formal logic using reasoning technology. How do people curate knowledge? • KnowBuddy helps people clarify what sentences mean with an easy to use, graphical interface. • The grammatical structure and logical semantics of sentences become increasingly precise as people click on or drag & drop words, phrases, clauses, etc. • People produce high quality logic for moderately complex sentences in 1-4 minutes (on their 1st day). • Its real-time, wiki-like platform facilitates collaborative curation of grammatical, logical, and semantic precision across all the sentences in the knowledge base. How much knowledge? How fast? • Distributed people contribute concurrently. • Several people tend to improve each sentence. • Typical users are more competent in English than logic. • Fully encoding thousands of sentences covering a chapter of a college science textbook took a few minutes per sentence, on average. • High-quality formal logic is produced an order of magnitude more quickly than prior approaches. How is 10x KA productivity possible? • Essentially, KnowBuddy focuses on productivity rather than automating understanding, which is far beyond the state of the art and error prone. – KnowBuddy assists with discerning the logical semantics of English sentences. – KnowBuddy leverages statistical techniques to improve productivity but makes no assumptions or guesses. Why KnowBuddy doesn’t guess… • The “parse” ranked first by machine learning techniques is usually not the one with the right syntax & semantics – below, the 13th ranked interpretation is the correct one >100xKA • What does improving knowledge acquisition productivity by an order of magnitude imply? • What does increasing the number of people who can perform KA by another 10x imply? – – – – 10x as much knowledge per contributor 10x as many contributors <$ per paid contributor 0$ per wiki contributor Implications of >100x KA • Legions of world-wide web collaborators curating scientific, medical, and legal wikis into broader, deeper formal knowledge systems than previously conceivable (e.g., to IBM). – revolutionary in education, science, healthcare, … • Enterprise performance management systems in more direct and reliable compliance with enterprise requirements and regulations. – perhaps $1,000,000,000,000 value w/in 10 years • Artificially intelligent expert systems bringing orders of magnitude more knowledge to bear. – revolutionary in legal, engineering, healthcare, … • Precise answers to increasingly deep questions. – revolutionary in education, the web, man-machine, … A Textbook Example • If a Paramecium swims from a hypotonic environment to an isotonic environment, will its contractile vacuole become more active? • As IBM admits, such questions are far beyond the foreseeable capabilities of statistical NLP to answer reliably. • KnowBuddy can even explain its answer to this question. – The answer is “no”. • Why is it easy for KnowBuddy but impractical for Watson? Cognitive Skills for QA • Educators use the Bloom scale to organize how they teach and how they measure understanding and intelligence. • The base of the Bloom scale distinguishes recalling information from understanding knowledge – search engines & statistical NLP are hardly cognitive • The middle of the Bloom scale distinguishes using knowledge to solve problems from understanding. – skills at and above this level are beyond Watson et al. Where does KnowBuddy fit on Bloom? • The Paramecium question requires cognitive skills ranging from the 3rd to the 5th level. – In the biology KA experiment discussed here, subject matter experts rated it as Bloom level #4. • KnowBuddy can answer some level #5 questions but is currently most competent between levels #3 and #4. – The blood cell question was judged level #3. – A level #5 question near current competence: • Would an animal cell lacking oligosaccharides on the external surface of its membrane have a reduced ability to transport ions against an electrochemical gradient? How do statistical approaches rate? • Search engines fail before level #2 (understanding) • Watson demonstrates level #2 capability • Watson has no innate level #3 capability (applying knowledge) Does this mean an AI is near? • First, measuring cognitive capabilities of systems should not be taken to imply that they are sentient. • Second, performing well at the higher levels of the Bloom scale is less about having knowledge than using it well. Although Watson is impressive and KnowBuddy can make it more so, becoming even more intelligent requires further advances in problem solving methodologies and reasoning technology. • Third, spoken language input for question asking (in addition to spoken answers, such as demonstrated by Watson) requires more forms of inference than have yet been well integrated, including deductive (aka logical), inductive (i.e., statistical), and abductive (e.g., hypothetical) inference. • Then, perhaps by 2020, after millions of sentences formalized by thousands of people are combined with something like Watson combined with something like Cyc, it may seem there is AI in the ether. If not AI, what fruit is near at hand? • Electronic textbooks or web services that measurably improve educational outcomes and standardized or advanced placement test performance. • Information retrieval in which results are more precisely relevant than foreseeable using statistical techniques (e.g., answers). • Legal, governance, risk, compliance , and operational systems that rigorously apply formal logic contained in contracts, legislation, regulation, policy and doctrine. • Decision support systems of greater breadth, depth, reliability and overall competence, such as in financial or healthcare applications. • Reduced application life cycle costs for so-called “business logic”. Case 1: Educational Impact • Inquire: An Intelligent Textbook—“Best Video” Award, AAAI 2012 – New Scientist, August 7, 2012 • Earlier this year, the team recruited 72 first-year students from De Anza College in Cupertino, California, to put the system to the test. Students were given either the full Inquire system, the Inquire system with the query function switched off, or a paper copy of Campbell Biology. They were then asked to spend 60 minutes reading a section of the book, 90 minutes on homework problems, and to take a 20-minute-long quiz. • Students who used the full Inquire system scored a grade better on the quiz, on average, than the other groups. "When we did our assessment, we didn't see any Ds or Fs, which we did see in the control groups," says Debbie Frazier, a high school biology teacher who works on the project. "Our students could use Inquire as a tool and ask it questions that they might be embarrassed to ask a teacher in person because it makes them feel stupid.“ • But Vulcan needed a solution to the following problem: • While such results are promising, perhaps it's a little soon to crown Inquire the future of textbooks. For starters, after two years of work the system is still only half-finished. The team plan to encode the rest of the 1400-page Campbell Biology by the end of 2013, but they expect a team of 18 biologists will be needed to do so. This raises concerns about whether the project could be expanded to cover other areas of science, let alone other subjects. • Which was the genesis of the experiment discussed here. – reducing the time and skill required to encode knowledge by at least an order of magnitude – the business case for knowledge-based textbooks such as Campbell Biology is now compelling