Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A Click PRACTICAL LOOK title AT to edit Master UNCERTAINTY MODELING style Click to edit Steve Master Unwin subtitle style Risk & Decision Sciences Group March 7, 2006 "The fundamental cause of trouble in the world today is that the stupid are cock-sure while the intelligent are full of doubt.“ Bertrand Russell 2 Illuminating the Path • Visual Analytics Agenda - Recommendations – Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty – Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. – Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process 3 Uncertainty Analysis as Resource to Visual Analytics • VA Agenda • UA Insight – Develop new methods and technologies for capturing and representing information quality and uncertainty – Probabilistic techniques – Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. – Nonprobabilistic alternatives – Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process • Elicitation methods • Aggregation methods • Information-theoretic approaches • Dempster-Shafer • Possibility theory – Uncertainty propagation techniques • Analytic • Numerical – Risk communication • Risk representation • Decision-analysis methods 4 MEASURING UNCERTAINTY CLASSICAL METHODS BAYESIAN METHODS NONPROBABILISTIC METHODS 5 Classical Statistics • Focus on Aleatory Uncertainty – random variation inherent in the system • Sampling produces confidence intervals • Need a sampling model – Generally unavailable for many real-world complex situations • Confidence intervals are not probability intervals – Propagation difficulties in even the simplest models 6 Bayesianism • de Finetti, Ramsey, Savage (1920s-50s) • Subjectivism – Epistemic Probabilities – Probability as a degree of belief • Classicists are coin tossers • Bayesians are believers – What is the basis for forming probability? • “ Probabilities do not exist” – Bruno de Finetti 7 Problems with Bayesianism • Because probabilities don’t exist, they have to be created – but how? • Bayes’ Theorem • Subjectivity is explicit – judgment of evidence • Do probabilities really reflect the way we conceive belief? – is probability theory a good theory of evidence? – what are the options? 8 One Option: Dempster-Shafer Theory • • • • Withholding belief distinct from disbelief Seahawks or Steelers will win? Set of possibilities: {sea, steel} Probability theory: – Weight of evidence attached to each exclusive possibility – p(sea), p(steel) • D-S theory: – Weight of evidence attached to each subset – m(Ø), m(sea), m(steel), m(sea U steel) • Allows: m(sea U steel) = 1, all other m=0 – A compelling ignorance 9 Support and Plausibility • Probability replaced by two belief measures: – Each calculated from weights of evidence – bel(sea) is the support for proposition ‘sea’ – pl(sea) is the plausibility of ‘sea’ – bel(sea) ≤ pl(sea) – Upper and lower “probabilities” • Complete ignorance • SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete ignorance on the matter of proposition ‘sea’ 10 Complementary Cumulative Belief Functions ESD Sensor System On-Demand Failure Rate 1 0.9 0.8 Complementary Cumulative Plausibility Belief Metric 0.7 Complementar y Cumulative Probability 0.6 0.5 Complementary Cumulative Support/Belief 0.4 0.3 0.2 0.1 0 1.0E-04 1.0E-03 1.0E-02 Failure Rate per Demand 1.0E-01 1.0E+00 11 Possibility Theory • Genesis in fuzzy sets • Possibility is an uncertainty measure that mirrors the fuzzy set notion of imprecision 12 The Set of Tall Men Membership to Set 1 0.8 Tall Very Tall 0.6 m(h) m'(h) = m2(h) 0.4 0.2 0 5' 8" 5' 9" 5' 10" 5' 11" 6' 0" 6' 1" 6' 2" Height 13 Possibility Theory • 2-tier belief: possibility and necessity • nec(X) ≤ pos(X) • Distinctive combinatorial logic – nec(X^Y) = min[nec(X), nec(Y)] – pos(XvY) = max[pos(X), pos(Y)] • No conceptual connection to probability – although probability/possibility can co-exist 14 Possibilistic Interpretation of Intelligence Statements (Heuer) Little chance Better than even Chances are slight Highly likely Possibility Probability 15 Experience with Nonprobabilistic Methods • Not all good: – Standardization of belief metrics? – Treatment of dependences? – Treatment of conflicting evidence? – Computational demands? – Interpretation of results? – Incorporation into decision process? • Plan B: Confront the problems with probabilistic methods 16 Principled Basis for Probability Formulation • Analysts uncomfortable producing probabilities – justified discomfort • Alternative: – Produce defensible basis for probability formulation based on nonprobabilistic judgment • Maximize expression of uncertainty subject to judged constraints • Borrow uncertainty metrics from: – statistical mechanics – information theory • Entropy = -∑i pi.ln pi – discrete probability distribution, pi 17 Application of InformationTheoretic Methods • Two USNRC programs: – QUEST- SNL • Quantitative uncertainty evaluation of source terms – QUASAR – BNL • Quantitative uncertainty analysis of severe accident releases • Both studies used the same form of input to the same deterministic models – non-probabilistic input • expert-generated input parameter uncertainty ranges • QUEST: Bounding analysis • QUASAR: Information Theory used to generate probability distributions from bounds • Probabilistic analysis internal to methodology – no elicitation of probability 18 Information Theory and the Preservation of Uncertainty Uncertainty Bands QUASAR I-131 QUEST QUASAR Cs-137 QUEST 1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00 Release Fraction 19 Uncertainty Analysis as Resource to Visual Analytics • VA Agenda • UA Insight – Develop new methods and technologies for capturing and representing information quality and uncertainty – Probabilistic techniques – Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. – Nonprobabilistic alternatives – Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process • Elicitation methods • Aggregation methods • Information-theoretic approaches • Dempster-Shafer • Possibility theory – Uncertainty propagation techniques • Analytic • Numerical – Risk communication • Risk representation • Decision-analysis methods 20 Merit Criteria for Uncertainty Analysis in Intel • • • • Makes the analyst’s job easier Represents strength of evidence intuitively Can reflect dissonant evidence Appropriately propagates uncertainty from analyst to decision-maker • Characterizes output uncertainty in a standardized and interpretable way • Computationally tractable • Promotes insight 21