Download A PRACTICAL LOOK AT UNCERTAINTY MODELING

A Click
PRACTICAL
LOOK title
AT
to edit Master
UNCERTAINTY
MODELING
style
Click to edit
Steve
Master
Unwin
subtitle style
Risk & Decision Sciences Group
March 7, 2006
"The fundamental cause of trouble in
the world today is that the stupid are
cock-sure while the intelligent are full of
doubt.“
Bertrand Russell
2
Illuminating the Path
• Visual Analytics Agenda - Recommendations
– Rec. 4.10: Develop new methods and technologies for capturing
and representing information quality and uncertainty
– Rec. 4.11: Determine the applicability of confidence assessment in
the identification, representation, aggregation, and communication
of uncertainties in both the information and the analytical methods
used in their assessment.
– Summary Rec: Develop methods and principles for representing
data quality, reliability, and certainty measures throughout the data
transformation and analysis process
3
Uncertainty Analysis as Resource to
Visual Analytics
•
VA Agenda
•
UA Insight
– Develop new methods and
technologies for capturing and
representing information quality and
uncertainty
– Probabilistic techniques
– Determine the applicability of
confidence assessment in the
identification, representation,
aggregation, and communication of
uncertainties in both the information
and the analytical methods used in
their assessment.
– Nonprobabilistic alternatives
– Develop methods and principles for
representing data quality, reliability,
and certainty measures throughout
the data transformation and analysis
process
• Elicitation methods
• Aggregation methods
• Information-theoretic approaches
• Dempster-Shafer
• Possibility theory
– Uncertainty propagation techniques
• Analytic
• Numerical
– Risk communication
• Risk representation
• Decision-analysis methods
4
MEASURING
UNCERTAINTY
CLASSICAL
METHODS
BAYESIAN
METHODS
NONPROBABILISTIC
METHODS
5
Classical Statistics
• Focus on Aleatory Uncertainty
– random variation inherent in the system
• Sampling produces confidence intervals
• Need a sampling model
– Generally unavailable for many real-world complex
situations
• Confidence intervals are not probability intervals
– Propagation difficulties in even the simplest models
6
Bayesianism
• de Finetti, Ramsey, Savage (1920s-50s)
• Subjectivism – Epistemic Probabilities
– Probability as a degree of belief
• Classicists are coin tossers
• Bayesians are believers
– What is the basis for forming probability?
• “ Probabilities do not exist”
– Bruno de Finetti
7
Problems with Bayesianism
• Because probabilities don’t exist, they have to be
created
– but how?
• Bayes’ Theorem
• Subjectivity is explicit
– judgment of evidence
• Do probabilities really reflect the way we conceive
belief?
– is probability theory a good theory of evidence?
– what are the options?
8
One Option:
Dempster-Shafer Theory
•
•
•
•
Withholding belief distinct from disbelief
Seahawks or Steelers will win?
Set of possibilities: {sea, steel}
Probability theory:
– Weight of evidence attached to each exclusive possibility
– p(sea), p(steel)
• D-S theory:
– Weight of evidence attached to each subset
– m(Ø), m(sea), m(steel), m(sea U steel)
• Allows: m(sea U steel) = 1, all other m=0
– A compelling ignorance
9
Support and Plausibility
• Probability replaced by two belief measures:
– Each calculated from weights of evidence
– bel(sea) is the support for proposition ‘sea’
– pl(sea) is the plausibility of ‘sea’
– bel(sea) ≤ pl(sea)
– Upper and lower “probabilities”
• Complete ignorance
• SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete
ignorance on the matter of proposition ‘sea’
10
Complementary Cumulative Belief
Functions
ESD Sensor System On-Demand Failure Rate
1
0.9
0.8
Complementary
Cumulative
Plausibility
Belief Metric
0.7
Complementar
y Cumulative
Probability
0.6
0.5
Complementary
Cumulative
Support/Belief
0.4
0.3
0.2
0.1
0
1.0E-04
1.0E-03
1.0E-02
Failure Rate per Demand
1.0E-01
1.0E+00
11
Possibility Theory
• Genesis in fuzzy sets
• Possibility is an uncertainty measure that
mirrors the fuzzy set notion of imprecision
12
The Set of Tall Men
Membership to Set
1
0.8
Tall
Very Tall
0.6
m(h)
m'(h) = m2(h)
0.4
0.2
0
5' 8"
5' 9"
5' 10"
5' 11"
6' 0"
6' 1"
6' 2"
Height
13
Possibility Theory
• 2-tier belief: possibility and necessity
• nec(X) ≤ pos(X)
• Distinctive combinatorial logic
– nec(X^Y) = min[nec(X), nec(Y)]
– pos(XvY) = max[pos(X), pos(Y)]
• No conceptual connection to probability
– although probability/possibility can co-exist
14
Possibilistic Interpretation
of Intelligence Statements (Heuer)
Little chance
Better than even
Chances are slight
Highly likely
Possibility
Probability
15
Experience with
Nonprobabilistic Methods
• Not all good:
– Standardization of belief metrics?
– Treatment of dependences?
– Treatment of conflicting evidence?
– Computational demands?
– Interpretation of results?
– Incorporation into decision process?
• Plan B: Confront the problems with
probabilistic methods
16
Principled Basis for Probability
Formulation
• Analysts uncomfortable producing probabilities
– justified discomfort
• Alternative:
– Produce defensible basis for probability formulation
based on nonprobabilistic judgment
• Maximize expression of uncertainty subject to
judged constraints
• Borrow uncertainty metrics from:
– statistical mechanics
– information theory
• Entropy = -∑i pi.ln pi
– discrete probability distribution, pi
17
Application of InformationTheoretic Methods
•
Two USNRC programs:
– QUEST- SNL
• Quantitative uncertainty evaluation of source terms
– QUASAR – BNL
• Quantitative uncertainty analysis of severe accident releases
•
Both studies used the same form of input to the same deterministic models
– non-probabilistic input
• expert-generated input parameter uncertainty ranges
•
QUEST:
Bounding analysis
•
QUASAR:
Information Theory used to generate probability distributions
from bounds
•
Probabilistic analysis internal to methodology – no elicitation of
probability
18
Information Theory and the
Preservation of Uncertainty
Uncertainty Bands
QUASAR
I-131
QUEST
QUASAR
Cs-137
QUEST
1.00E-04
1.00E-03
1.00E-02
1.00E-01
1.00E+00
Release Fraction
19
Uncertainty Analysis as Resource to
Visual Analytics
•
VA Agenda
•
UA Insight
– Develop new methods and
technologies for capturing and
representing information quality and
uncertainty
– Probabilistic techniques
– Determine the applicability of
confidence assessment in the
identification, representation,
aggregation, and communication of
uncertainties in both the information
and the analytical methods used in
their assessment.
– Nonprobabilistic alternatives
– Develop methods and principles for
representing data quality, reliability,
and certainty measures throughout
the data transformation and analysis
process
• Elicitation methods
• Aggregation methods
• Information-theoretic approaches
• Dempster-Shafer
• Possibility theory
– Uncertainty propagation techniques
• Analytic
• Numerical
– Risk communication
• Risk representation
• Decision-analysis methods
20
Merit Criteria
for Uncertainty Analysis in Intel
•
•
•
•
Makes the analyst’s job easier
Represents strength of evidence intuitively
Can reflect dissonant evidence
Appropriately propagates uncertainty from analyst
to decision-maker
• Characterizes output uncertainty in a standardized
and interpretable way
• Computationally tractable
• Promotes insight
21