Download Example: the weather forecasting

Textbook
• Basics of an Expert System:
– “Expert systems: Design and Development,” by: John Durkin, 1994,
Chapters 1-4.
• Uncertainty (Probability, Certainty factor & Fuzzy):
– “Expert systems: Design and Development,” by: John Durkin, 1994,
Chapters 11-13.
– “Artificial intelligence: a guide to intelligent systems,” by: Michael
Negnevitsky, 2005, Chapters 3,4.
1
Chapter 11
Bayesian approach to inexact reasoning
2
What is Uncertainty?
• Information can be incomplete,
inconsistent, uncertain, or all three.
• Uncertainty: lack of the exact knowledge to
reach a perfectly reliable conclusion.
• Classical logic permits only exact reasoning.
IF A is true THEN A is not false
IF A is false THEN A is not true
3
Sources of uncertain knowledge
• Weak implications:
• Need for concrete correlations between IF and THEN parts
• handling vague associations is required
• Imprecise language:
• Our natural language is ambiguous and imprecise. (i.e. often and
sometimes, frequently and hardly ever)
• In 1944, Ray Simpson asked 355 high school and college students
to place 20 terms on a scale between 1 and 100.
• In 1968, Milton Hakel repeated this experiment.
Quantification of
ambiguous and imprecise terms
Ray Simpson (1944)
Term
Mean value
Always
99
Very often
88
Usually
85
Often
78
Generally
78
Frequently
73
Rather often
65
About as often as not
50
Now and then
20
Sometimes
20
Occasionally
20
Once in a while
15
Not often
13
Usually not
10
Seldom
10
Hardly ever
7
Very seldom
6
Rarely
5
Almost never
3
Never
0
Milton Hakel (1968)
Term
Mean value
Always
100
Very often
87
Usually
79
Often
74
Rather often
74
Frequently
72
Generally
72
About as often as not
50
Now and then
34
Sometimes
29
Occasionally
28
Once in a while
22
Not often
16
Usually not
16
Seldom
9
Hardly ever
8
Very seldom
7
Rarely
5
Almost never
2
Never
0
5
Sources of uncertain knowledge
(continued.)
• Unknown data:
• When the data is incomplete or missing, the only solution is to accept the
value “unknown” and proceed to an approximate reasoning with this value.
• Combining the views of different experts:
• Large expert systems usually combine the knowledge and expertise of a
number of experts. Unfortunately, experts often have contradictory
opinions and produce conflicting rules.
• To resolve the conflict, attaching a weight to each expert
6
Solutions to Handle Uncertainty
• Probability approach
Durkin, Ch.11 & Negnevitsky Ch. 3
• Certainty Factor
Durkin Ch. 12 & Negnevitsky Ch. 3
• Fuzzy Logic
(Durkin Ch. 13 & Negnevitsky Ch. 4
7
Basic probability theory
• Probability provides an exact approach for inexact reasoning
• The probability of an event is the proportion of cases in which the
event occurs.
• A scientific measure of chance.
• A range between 0 (impossibility) to 1 (certainty).
8
Basic probability theory
• s: success; f :failure
s
P success  p 
s f
f
P failure  q 
s f
p q 1
• throwing a coin,
• the probability of getting a head s
• the probability of getting a tail f
• P(s) = p(f) =0.5
Conditional Probability
• Let A, B be two events in the world
• A and B are not mutually exclusive (A∩B ≠ null)
• The probability that event A will occur if event B occurs is called
the conditional probability.
• p(A|B): Conditional probability of event A occurring given that
event B has occurred.
p AB 
the number of times A and B can occur
the number of times B can occur
10
Conditional Probability
• P(A∩B): Joint Probability
• Similarly,
• Hence,
11
Bayesian Rule
pBA p A
p AB 
pB
where:
• p(A|B) is the conditional probability that event A occurs given that
event B has occurred;
• p(B|A) is the conditional probability of event B occurring given
that event A has occurred;
• p(A) is the probability of event A occurring;
• p(B) is the probability of event B occurring.
12
Bayesian Rule
• 𝑝(B) = 𝑝(B ∩ 𝐴) + 𝑝(𝐵 ∩ ¬𝐴)
• 𝑝(B) = 𝑝(B 𝐴 𝑝(𝐴) + 𝑝 𝐵 ¬𝐴 𝑝(¬𝐴)
• Substituting in the Bayesian rule:
• The above equation provides the background for the application of
probability theory to manage uncertainty in expert systems.
13
Example: Chest Pain
14
Joint Probability on a number of mutually
exclusive and exhaustive events
• 𝑝 𝐵 ∩ 𝐴1 = 𝑝 𝐵 𝐴1 𝑝 𝐴1
• 𝑝 𝐵 ∩ 𝐴2 = 𝑝 𝐵 𝐴2 𝑝(𝐴2 )
•…
• 𝑝 𝐵 ∩ 𝐴𝑛 = 𝑝 𝐵 𝐴𝑛 𝑝(𝐴𝑛 )
15
Bayesian rule with mutually exclusive
and exhaustive events
• Replacing
• in Bayesian rule
16
Bayesian Reasoning
• Suppose all rules in the knowledge base are represented in the
following form:
If E is true, then H is true {with probability p}
• This rule implies that if event E occurs, then the probability that
event H will occur is p.
• H: hypothesis
• E: evidence
17
The Bayesian rule
in terms of hypotheses and evidence
pHE 
p EH p H
p E H  p H  p E H  p H
• p(H): the prior probability of hypothesis H being true;
• p(E|H): the probability that hypothesis H being true will
result in evidence E;
• p(¬H): the prior probability of hypothesis H being false;
• p(E|¬H ): the probability of finding evidence E even when
hypothesis H is false.
• P(H|E): Posterior probability
18
The Bayesian rule in expert systems
• An expert determines:
• the prior probabilities for possible hypotheses, p(H) and
p(¬H)
• the conditional probabilities for observing evidence E if
hypothesis H is true, p(E|H), and if hypothesis H is
false, p(E|¬H).
• The expert system computes:
• the posterior probability of hypothesis H upon
observing evidence E, p(H|E).
pHE 
p EH p H
p E H  p H  p E H  p H
19
The Bayesian rule with multiple
hypothesis and multiple evidences
• We can take into account both multiple hypotheses 𝐻1 , 𝐻2 , … , 𝐻𝑚
and multiple evidences 𝐸1 , 𝐸2 , … , 𝐸𝑛 .
• Both must be mutually exclusive and exhaustive.
• Single evidence E and multiple hypotheses follow:
• Multiple evidences and multiple hypotheses follow:
20
The Bayesian rule with multiple
hypothesis and multiple evidences
• assumption: conditional independency among different evidences
p H i E1 E2 . . . En 
m
p E1 H i  p E 2 H i  . . .  p En H i  p H i
 p E1 H k
k 1
 p E 2 H k  . . .  p En H k  p H k
21
Example: Ranking potentially true
hypothesis
• Suppose an expert, given three conditionally independent
evidences E1, E2 and E3, creates three mutually exclusive and
exhaustive hypotheses H1, H2 and H3.
Probabilit y
H y p o t h esi s
i 1
i 2 i 3
p Hi
0.40
0.35
0.25
p E1 H i
0.3
0.8
0.5
p E2 H i
0.9
0.0
0.7
p E3 Hi
0.6
0.7
0.9
H1: Cold
H2: Allergy
H3: Flu
E1: Cough
E2: Fever
E3: runny nose
22
Example: Ranking potentially true
hypothesis
• Assume that we first observe evidence E3. The expert system computes
the posterior probabilities for all hypotheses as
p H i E3 
3
p E3 H i  p H i
 p E3 H k
k 1
,
i = 1, 2, 3
 p Hk
0.6  0.40
 0.34
0.6  0.40 + 0.7  0.35 + 0.9  0.25
0.7  0.35
 0.34
p H 2 E3 
0.6  0.40 + 0.7  0.35 + 0.9  0.25
p H1 E3 
p H3 E3 
0.9  0.25
 0.32
0.6  0.40 + 0.7  0.35 + 0.9  0.25
H1: Cold
H2: Allergy
H3: Flu
E1: Cough
E2: Fever
E3: runny nose
• After evidence E3 is observed, belief in hypothesis H1 decreases and
becomes equal to belief in hypothesis H2. Belief in hypothesis H3
increases and even nearly reaches beliefs in hypotheses H1 and H2.
23
Example: Ranking potentially true
hypothesis
• Suppose now that we observe evidence E1. The posterior
probabilities are calculated as
p Hi E1E3 
3
p E1 Hi  p E3 H i  p H i
 p E1 H k
k 1
,
i = 1, 2, 3
 p E3 H k  p H k
0.3 0.6 0.40
0.3 0.6 0.40 + 0.8 0.7  0.35 + 0.5
0.8 0.7  0.35
p H 2 E1E3 
0.3 0.6 0.40 + 0.8 0.7  0.35 + 0.5
 0.25
0.5 0.9  0.25
0.3 0.6 0.40+ 0.8 0.7  0.35 + 0.5
 0.25
p H1 E1E3 
p H3 E1E3 
 0.25
• Hypothesis H2 has now become the most likely one.
 0.19
 0.52
 0.29
H1: Cold
H2: Allergy
H3: Flu
E1: Cough
E2: Fever
E3: runny nose
24
Example: Ranking potentially true
hypothesis
• After observing evidence E2, the final posterior probabilities for
all hypotheses are calculated:
p H i E1E2E3 
3
p E1 Hi  p E2 H i  p E3 H i  p H i
 p E1 H k
k 1
,
i = 1, 2, 3
 p E2 H k  p E3 H k  p H k
0.3 0.9 0.6 0.40
 0.45
0.3 0.9 0.6 0.40+ 0.8 0.0 0.7  0.35+ 0.5 0.7 0.9  0.25
0.8  0.0  0.7  0.35
p H 2 E1E2E3 
0
0.3 0.9  0.6  0.40 + 0.8  0.0  0.7  0.35+ 0.5 0.7  0.9 0.25
p H1 E1E2E3 
p H 3 E1E2E3 
0.5 0.7 0.9  0.25
 0.55
0.3 0.9 0.6 0.40+ 0.8 0.0 0.7  0.35+ 0.5 0.7 0.9  0.25
• Although the initial ranking was H1, H2 and H3, only hypotheses
H1 and H3 remain under consideration after all evidences (E1, E2
and E3) were observed.
25
Uncertain Evidence (LS & LN)
If E {LS, LN}, then H
• LS (likelihood of sufficiency): represents a measure of the expert
belief in hypothesis H if evidence E is present.
• LN (likelihood of necessity): a measure of discredit to hypothesis
H, if evidence E is missing.
26
Uncertain Evidence (LS & LN)
• Note that LN cannot be derived from LS. The domain expert must
provide both values independently
• High values of LS (LS>>1) indicate that the rule strongly supports
the hypothesis if the evidence is observed, and
• low values of LN (0<LN<1) suggest that the rule also strongly
opposes the hypothesis if the evidence is missing.
Review
27
Uncertain Evidence (LS & LN)
• Usually (but not always):
Review
• Example: FORCAST expert system of London:
28
Effects of LS and LN on Hypothesis
Review
29
How to calculate posterior probabilities
from LS & LN
Review
Q: Prove that the above equations are the same Bayesian rule.
pHE 
p EH p H
p E H  p H  p E H  p H
30
Example: the weather forecasting
31
Example: the weather forecasting
(non-probabilistic solution)
• Simple non-probabilistic knowledge base:
• Rule: 1
IF today is rain, THEN tomorrow is rain
• Rule: 2
IF today is dry, THEN tomorrow is dry
• Using these rules we will make only ten mistakes
32
Example: the weather forecasting
(probabilistic solution)
• Simple Probabilistic knowledge base:
33
Example: the weather forecasting
p(tomorrow is rain | today is rain)
34
Example: the weather forecasting
Complete knowledge base
35
Assignment 3, (Durkin, ch. 11)
Due Date: 93/8/29
36