Download Week 8 Lecture MSE614

Week 8
MSE614 – SP 08
Ileana Costea
HW Questions on KA
Due today, Week 8
Assigned last session, Week 7
A few verbal questions
(see Transparency)
HW Question #1
What is meta-knowledge?
Answer:
K about K, or
The system’s K about how it reasons
e.g.,
• how to use K in specific situations;
• how to determine which K is relevant
• when the K is insufficient
Meta-knowledge (ctnd.)
• Meta-K allows the system to examine the operation of
the descriptive (declarative) and procedural K
• Explanation in an ES can be viewed as Meta-K
In the future Meta-K will allow ES more:
– Create the rationale behind individual rules
–
by reasoning from first principles
– Tailor explanations to fit the audience
– Able to change their internal structure through:
– Rule correction
– Reorganization of KB
– System reconfiguration
HW Question #2
Give four reasons why KA is difficult.
• Representation mismatch
between the human Expert and
the program underdevelopment
• Large number of participants
• Transfer via a machine
• Difficulty of experts to describe their K
More … next slide
HW Question #2 (Ctnd.)
Give four reasons why KA is difficult.
•
•
•
•
•
Other reasons, see slide.
Overcoming difficulties:
Research on KA to reduce mismatch:
Develop systems able to accept advice
KA to converse with expert in natural
language
• Simplify the syntax of the rules so that an
expert can build a system without training
HW Question #3
Describe the process of
Protocol Analysis. (Ctnd.)
• The expert is asked to perform a real task.
• Then he/she is asked to verbalize (think aloud)
the thought process.
• A record (protocol) of all the details is made.
• The records are transcribed.
HW Question #3
Describe the process of
Protocol Analysis. (Ctnd.)
Required Web search:
• http://www.psy.fsu.edu/faculty/ericsson/ericsson.proto.th
nk.html
Protocol analysis and Verbal Reports on Thinking
An updated and extracted version from Ericsson (2002)
• http://www.epistemics.co.uk/Notes/176-0-0.htm
Protocol Analysis Techniques
Among others discusses Repertory Grid Technique
• http://en.wikipedia.org/wiki/Protocol_analysis
Protocol Analysis
http://en.wikipedia.org/wiki/Protocol_analysis
(from Wikipedia, the free encyclopedia)
• Protocol analysis is a psychological research method that elicits
verbal reports from research participants. Protocol analysis is used
to study thinking in cognitive psychology, cognitive science, and
behavior analysis. It has found further application in the design of
surveys and interviews, usability testing, and educational
psychology.
• Usability testing is a technique used to evaluate a product by
testing it on users.
• Cognitive psychology is a school of thought in psychology that
examines internal mental processes such as problem solving,
memory, and language.
Cognitive science
(from Wikipidia)
•
•
•
the scientific study either of mind
or of intelligence.
an interdisciplinary study drawing
from relevant fields including
psychology, philosophy,
neuroscience, linguistics,
anthropology, computer science,
and biology.
term coined by Christopher
Longuet-Higgins in his 1973
commentary on the Lighthill
report, which concerned the thencurrent state of Artificial
Intelligence research. In the same
decade, the journal Cognitive
Science and the Cognitive
Science Society began.
Protocol Analysis (Ctnd.)
• in AI – KA it is a method of KA
• it is one of the “tracking methods”
• it is the most common method of formal
tracking.
(See next slide for “tracking”)
Tracking Methods
(Protocol Analysis – Ctnd.)
• Process tracking: a set of techniques that
attempt to track the reasoning process of an
expert.
• Popular method among cognitive psychologists
who:
interested in discovering the expert’s “train of
thought” while he/she reaches a conclusion
• Tracking methods are: informal and formal
Protocol Analysis (Ctnd.)
• Particularly a set of techniques known as
verbal protocol analysis,
• a common method by which KEngineer
acquires detailed K from the expert
Protocol Analysis
(Ctnd.)
• A protocol = a record or documentation of the expert’s step-by-step
info. processing (IP) and decision-making (DM) behavior
• Protocol Analysis Is similar to interviewing but more formal and
systematic
• The expert is asked by the KE to perform a real task and verbalize
his/her thought process
• Expert is asked to “think aloud” while performing the task or solving
a problem under observation
• A recording made while expert “thinks aloud”
• Recording describes every aspect of the IP and DM behavior
• The recording becomes a record, or protocol, of the expert’s
ongoing behavior
Protocol Analysis (ctnd.)
• Recording is:
– transcribed by the KE for further analysis
( e.g., to deduce the decision process)
– Coded by the KE
Protocol Analysis (ctnd.)
• In contrast with interactive interviewing methods,
PA involves mainly a one-way communication.
• KE prepares the scenario and plans the process
• during the session the Expert does most of the
talking as he/she interacts with the data to solve
problem
• KE listens and records the process
• Later KE must: analyze, interpret, and structure
the protocol (or record) into Knowledge
Representation for a review by the Expert.
Protocol Analysis (ctnd.)
• for Procedure of PA
&
• Advantages and Limitations of PA
 see Transparencies
HW Question #4
List the major difficulties of KA from
multiple experts.
• Different experts use different methods of
problem solving, yet all may be correct.
• It is difficult to reconcile these different
methods.
• It is difficult to get all the experts together at
the same time and place.
List the major difficulties of KA from
multiple experts. (Ctnd.)
The four possible scenarios to deal with
multiple experts:
– individual experts
– primary and secondary experts
– small groups
– panels
List the major difficulties of KA from
multiple experts. (Ctnd.)
Major methods of dealing with multiple experts:
• Consensus methods
Reach consensus by using group dynamics or other methodologies
• Analytical approaches:
figure the average estimate of the group
• Selection of an appropriate line of reasoning for each occasion –
don’t mix
• Automation of the process
automatic decision by the computer
• Blackboard system
divide the problem into sub-domains and use one expert for each
sub-domain
HW Question #4
Define evaluation, validation and
verification of K.
AAAI
http://www.aaai.org/home.html
• Association for the Advancement of
Artificial Intelligence (AAAI)
(formerly the American Association for
Artificial Intelligence)
• is a nonprofit scientific society devoted to
advancing the scientific understanding of:
• mechanisms underlying thought & intelligent
behavior, and
• their embodiment in machines
AI Applications
http://www.aaai.org/AITopics/pmwiki/pmwi
ki.php/AITopics/SiteMap
a whole list of links of AI applications
http://www.aaai.org/AITopics/pmwiki/pmwi
ki.php/AITopics/AINews
AI in the News