Download TRAC-MTRY Briefing for MG Flynn

Unclassified
The Representation of Uncertainty for
Validation and Analysis of
Social Simulations
Debbie Duong, Augustine Consulting
David Makovoz and Hyam Singer, Impact Computing
TRADOC Analysis Center – Monterey
21 September 2010
Unclassified
What is the Basic Mission of Analysis?
• What does the commander want from analysis?
– The likelihood of outcomes of plausible courses of action
– How certain we are of those likelihoods
• What technologies can help us to find these outcomes?
– Simulations that compute results based on theoretical cause (correlation alone
isn’t general enough)
– Integrative Frameworks that use ontologies to compose multiple simulations to
recombine factors
– Probabilistic Ontologies in those frameworks to properly handle uncertainty
• What is a “Probabilistic Ontology” ?
– An “ontology” is a rich categorization of the objects of a domain, general and
specific, that includes the relations between objects. It is subject to inference,
and can enforce logical consistency
– A “probabilistic ontology” includes categorizations with a degree of uncertainty
8/16/2010
CG Modeling
2
Why it is difficult to find likelihoods of plausible
outcomes in Irregular Warfare
• Irregular Warfare(IW) is uncertain, and Integrative Frameworks are
needed to take uncertainty into account
– Integrative Frameworks can handle Epistemic Uncertainty, the uncertainty in
what social scientists know, by testing Courses of Action against different
combinations of theories by switching models in and out
– Integrative Frameworks can handle Intrinsic Uncertainty of the social world,
which is greater than the physical world because of “path dependencies” in
Complex Adaptive Systems, and the human use of symbols
• These basic forms of uncertainty lead to many sub-types that
probabilistic ontologies can combine, to make a single probabilistic
outcome space
–
–
–
–
–
–
–
8/16/2010
Uncertainty of correspondence
Uncertainty of properties
Uncertainty of subsumed relations
Social correlative studies
Dynamic descriptions
Confidence factors
Feasible Parameter Sets
CG Modeling
3
Uncertainty of Correspondence
• Example: “There is a 20% chance that what a child identifies as a
bug is the same as an entomologist would”
• Lack of correspondence between the categorization of data in
simulations comes from Epistemic Uncertainty
– Theoretical disagreement about the relevant categories and relations between
them
• Social data mostly comes in different, useful-for-a-purpose, points of
view and so it is hard to interoperate using logic
• How Probabilistic Ontologies can help
8/16/2010
– Inference may be used in detailed ontologies that break down to
elements and retranslate up, in a mediation ontology
- Example: “a child calls any animal with an exoskeleton a bug”
– Probabilistic relations may be used to cover what logic does not
- Example: “what an entomologist calls an insect, a child calls a “bug”
– In multiple runs of the simulation, change the match in the interface
from simulation to simulation probabilistically
CG Modeling
4
Uncertainty of Properties
• Example: “There is a 70% chance that a chair has four legs”
• Comes from the fact that human concepts are by nature fuzzy, and
ill-suited to crisp definitions that make computer concepts decidable
• Differences in points of view can be reflected in different beliefs in
the percent chance of a category having a property
• Uncertainty of properties allows something to have “membership”
values in many categories
• How Probabilistic Ontologies can help
8/16/2010
– Uncertainty of properties can be used to compute correspondence
using statistics of properties
- Example: Use the fact that John thinks 90% of chairs have four legs, and
Kate thinks 25% of them do, to match Kate’s idea of “chair” to John’s idea
of “chair” dependent on the number of legs
– In multiple runs of the simulation, change the match in the interface
from simulation to simulation probabilistically
CG Modeling
5
Uncertainty of Subsumed Relations
• Example: “There is a 90% chance that a murder weapon is a gun”
• Comes from the uncertainty of multi-resolutional models
– One model is at a higher (less specific) level of description, i.e. doesn’t need to
know the type of weapon, and the over is at a lower level of description, and i.e.
does need to know the type of weapon
– If they have the same point of view, then correspondence going from lower to
higher can be exact. However, correspondence from higher to lower must be
probabilistic
• How Probabilistic Ontologies can help
– Compute the conditional probability that a higher level translates to a particular
lower level category
– In multiple runs of the simulation, change the match in the interface from
simulation to simulation probabilistically
8/16/2010
CG Modeling
6
Social Correlative Studies
• Example: 10% of the time inflation occurs, unemployment is also
high
• Two branches of social literature are correlative studies and
theoretical studies
– Simulations take care of causal studies, because they have the ability to
represent cause in humans through motivation towards goals
– Even in causal simulations, statistics covers for what is not known, and falls
under the category of “correlative”
- For example, If I don’t know the causal mechanisms by which anomie is related to
unemployment, I take a random variate
• How Probabilistic Ontologies can help
– Correlative studies can help resolve conflicts between models and to turn
composed simulations into a single scenario
– This happens through finding a scalar correspondence to data, with the
correlative studies serving as the more trusted data than that created by the
simulation, to use to adjudicate between models
– A probabilistic match between the concepts of the social correlative studies and
the models is necessary to do this validation and adjudication
– If we know what data is correlated to what, we can assign surrogate data
8/16/2010
CG Modeling
7
Dynamic Descriptions
• There is over a 90% chance that a State with serious factionalization
and a partial democracy will become unstable within a year
• In order to compare a simulation’s data to social correlative data and
to another simulation’s data, not only should a “co-occurrence”
match be made, but a dynamic “sequential” match
– To validate a simulation against data, dynamic sequences of states should be
similar
– A Markov process describes dynamic sequences of states based on variable
values well, and can be a common framework for comparison of series data
• How Probabilistic Ontologies can help
– Probabilistic Ontologies can find the correspondence between data of
one simulation and data of another, so that Markov processes that
describe their dynamics can be compared.
– The Markov process itself may be expressed in a probabilistic ontology
8/16/2010
CG Modeling
8
Confidence Factors
• Example: John believes that the New York Times is wrong 20% of
the time while the National Enquirer is wrong 80% of the time
• In order to compute what we believe to be the correct state space of
output, we must also include the pedigree of data that we are using,
our confidence in the source of the data.
– Subject Matter experts have different levels of skills even within the
same school of thought
– Modelers and modeling techniques have different levels of
trustworthiness
– It may be that you would not want to trust data that did not correspond
to trusted data in the past
• How Probabilistic Ontologies can help
– Probability theory can fold confidence of source right in with the other
types of uncertainty, making a wider range of parameters to test for data
with less correspondence
8/16/2010
CG Modeling
9
Feasible Parameter Sets
• Example: Ninety five percent of the time that visibility is low, the
speed of the boat is low
• Analysis using multiple models is all about data farming feasible
sets of parameters that implement the study.
– We data farm for likelihoods of *plausible* Courses of Action
– This may include Strategic Data Farming, where motivation towards
goals are used to add parameters of moves and countermoves for
scripted simulations
• How Probabilistic Ontologies can help
– Uncertainty may tell the range, but ontology tells the rules that restrict
– The models should be treated like implementations of a functional
specification of a conceptual model, implementing an ontology
– Ontologies are rich enough to express the rules and relations of a
theory, and define what is open to scientific refute and what is mere
implementation details
– The ontology is a “cognitive wrapper” that automates how a human
would put in combinations of parameters
8/16/2010
CG Modeling
10
Summary
• Probability Theory can combine all the forms of uncertainty in the
parameters in the interface between models, to efficiently explore the
probabilistic state space
• Probabilistic Ontologies tell the rules of the study, and the models
merely implement these rules. Ontologies restrict the sets of inputs
to the models so that they are appropriate for exploring the space of
likelihood
• Probabilistic Ontologies can compose a federation of simulations
and help it to find a consensus, so that different theories may be
treated as another form of uncertainty through switching models in
and out
8/16/2010
CG Modeling
11