Download Expert Elicitation (Cooke`s method)

Risk Assessment and Risk Management
of Emerging and Re-emerging Infectious
Diseases:
Public Health / Risk Assessors Perspective
Susie ElSaadany,
Chief, Statistics and Risk Assessment
&
Adjunct Professor, Faculty of Medicine, University of Ottawa
Blood Safety Surveillance and Health Care Acquired Infections Division
Centre for Communicable Diseases and Infection Control
Infectious Diseases and Emergency Preparedness Branch
Public Health Agency of Canada
Mathematics in Emerging Infectious Disease Management, Cuernavaca, Mexico, January 10-14, 2011
Outline
•
Background
Blood Safety Programme at The Public Health
Agency of Canada
•
Challenges and Responsibility
•
Risk Assessment Tools
•
Expert Elicitation (as an Informative Process)
•
Examples
The Blood Safety Program
Healthcare Associated Infections
(HAI)
Parenteral
Transmission
Nosocomial
Transmission
(Bloodborne Pathogens)
SEPSIS
Nosocomial jurisdiction, if microbes cultured and/or identified. If no
microbes can be cultured or identified, then the case becomes Bloodborne
Pathogen’s jurisdiction, usually related to an unknown virus.
[The syringe issue related to a nosocomial transmission of bloodborne
pathogens (BBP) by reuse of a single syringe. This requires a BBP risk
assessment along with nosocomial and BBP risk mitigation strategies.]
Sources of Emerging Pathogens
Transmitted via Blood
Clotting
Factors
Platelets
Cryoprecipitate
Fresh/Frozen
Plasma
Emerging Blood
Pathogens
Cells, tissues,
organs, semen
Zoonotic, parasitic, viral, bacterial
and prion infections
Correctional
Facilities
Community
Prevalence
Xenotransplantation
Immigration and
population
movement
Climate change and
vector movement
Sources of Emerging Pathogens
Transmitted Parenterally via Healthcare
Blood,
Components &
Products
Cells, Tissues,
Organs &
Semen
Surgical
Implants of
Animal Tissues
Medical
Devices
Healthcare Associated
Infections (HAI)
(TPD)
Biologics
Emerging
Technologies
Surveillance initiated and/or under
development: high risk
(includes vaccines)
(BGTD)
Grandfathered-in
Biologics*
Surveillance not yet initiated:
high risk & significant gaps
* eg., heparins
TPD: Therapeutic Products Directorate, Health Canada
BGTD: Biologics and Genetic Therapies Directorate, Health Canada
Considered low risk
except for CJD/vCJD
and other prion
diseases
SRA Section Roles and
Responsibilities
•
Statistics and Risk Assessment Section (SRA) within Blood
Safety Surveillance and Health Care Acquired Infections Division:
–
Special focus on risk assessments for rare and
emerging diseases to meet modern health care needs for
better knowledge in the face of little to no scientific
information
–
Promoting the development of statistical techniques
surrounding issues of modeling uncertainty.
–
Facilitating iterative communication between regulators
(Health Canada, Biologics and Genetic Therapies Directorate)
and policy makers
Data Limitations for
Rare and Emerging Diseases
Scientific information required to model the public health risks for
rare and emerging diseases and events may be limited or
unavailable due to:
•
•
•
•
•
poor understanding of the disease / event
limited case numbers
lack of scientific study
lack of valid information in the bio-medical literature
obstacles in communicating what information is known
These limitations need to be managed against urgent needs
for risk assessment and regulatory policy development to
mitigate public health risks.
SRA Experience
•
Have encountered 3 types of “insufficient data”
1.
2.
3.
•
Data do not exist
Data cannot be published (proprietary)
Data exist, but are incomplete
Dealing with two kinds of uncertainty in modelling
public health threats:
–
–
heterogeneity or stochasticity
incomplete knowledge and/or systematic
measurement errors
SRA Experience & Tools
Bacterial infections
from ureteroscope
vCJD and
Islet cell Risk
vCJD and
Vaccine Risk
•
Viral Infections Due
to Improper Re-use
of Syringes
International and National Expert Advisory Group providing inputs
–
Expert Elicitation as a tool to gather quantitative information, but also
excellent qualitative tool for identification of knowledge gaps in science
•
Fuzzy Logic and Analysis for modeling of infectious diseases (uncertainty)
•
Linear deterministic models (variability)
•
Logical Data Analysis (data mining, artificial intelligence, pattern recognition)
Decision-Making Framework for Identifying, Processing and
Managing Health Risks
Collect Assumptions to Input in Model:
-Published literature
-Expert consensus
Identify the Issue
and Its Context
Monitor and
Evaluate Results
Assess Risks
and Benefits
Run Model Calculations
INVOLVE INTERESTED
AND
AFFECTED PARTIES
Implement
the Strategy
Identify and
Analyze Options
Select a
Strategy
Consideration of other
Parameters outside of
Model, by decision-makers
Risk Analysis Framework
1.
2.
Assessment of effectiveness of
measures taken
Review risk management and/or
assessment as necessary
Risk Management
A. Risk Evaluation
A brief description of the situation
Product or commodity involved
The values expected to be placed at
risk
(e.g. human health, economic
concerns)
Potential consequences
Consumer perception of the risks
The distribution of risks and benefits
D. Monitoring and review
1.
Risk
Communication
2.
3.
4.
C. Implementation of
management decision
B. Risk management and
option assessment
1.
2.
3.
Identification of available management options
Selection of preferred management option,
including consideration of an appropriate safety
standard
Final management decision
5.
6.
7.
Identification of TSE/ Blood, cells, tissues and
organs safety problem
Establishment of a risk profile
Ranking of the hazard for risk assessment and
risk management priority
Establishment of the risk assessment policy for
conduct of risk assessment
Commitment of resources
Commissioning of risk assessment
Consideration of risk assessment result
Risk perception
Value judgement
Precautionary principle
Benefits/ costs
Other technical factors
Regulatory or other control measures
Value judgements and
policy choices for the
risk assessment
process
Hazard identification
Hazard characterisation
Exposure assessment
Risk characterisation
Risk Assessment
Risk Assessment to Risk Management:
The Process Perspective
•
At the highest level, the logic underlying a risk
assessment is always the same:
Seriousness of Consequences
Low
of Event
High
Top
High
Probability
Medium
concern
Medium
Low
Low
concern
Risk Matrix as a tool for priority setting, Health Canada 2010
Risk Assessment Structure - Data Requirements
Issue
Identification
Exposure
Assessment
Hazard
Characterization
Risk
Characterization
Product
Raw
Materials
Factors
Affecting
DoseResponse
Methodology
Exclusions
Assumptions
Output of
Hazard
Characterization
Data Input for
Risk
Assessment
Equations
Processing
Risk
Assessment
Equations and
Outputs
Final Product
at
Point of Use
Uncertainty
and
Variability
Potential
Consumer
Exposure
Level of Confidence
in Model &
Limitations or
Contraints
Discussion
and
Recommendations
Examination
of Different
Risk Mitigation
Strategies
* Template developed by TSE Science and Policy Teams under the TSE Secretariat, HPFB
The concept of De minimis risk
Zooming in…
Bloodborne pathogens and health
care acquired infection
Emerging Infectious Disease
Current Activities
•
Emerging infections are a continuing threat to human health:
–
–
Some transmissible by blood
Others show evidence of possible transmissibility
•
•
•
•
•
•
•
•
•
•
HIV
HBV/HCV, Burden of illness studies
West Nile virus, SARS
Babesia species, Dengue virus, and vCJD
Xenotropic murine leukemia related virus
(XMRV) and blood
H5N1 and blood safety
H1N1 and pandemic influenzas in Relation to
PPE
Anti-microbial resistance (AMR)
Chronic Wasting Diseases (CWD)
Lyme disease
Expert Elicitation for Risk &
Disease Models
• N95 masks and infection of healthcare workers by SARS
• Dose-response of the Anthrax
• vCJD carried out in March 2009 (funded by PrioNet
Canada and PHAC)
• XMRV carried out in September 2010 (PHAC)
• CWD to be carried out in 2011 (funded by PrioNet
Canada)
Structured Expert Elicitation
Approach
Method Overview (1)
1. A group of problem domain experts (e) is selected. “E” is the
number of experts participating.
2. Experts meet in person to assess a set of ‘calibration items’,
whose true values are known. Each expert expressing his or her
views as elemental uncertainty distributions.
3. Expert responses are treated as statistical hypotheses and scored
for statistical likelihood that they are correct and display an
appropriate ability to gauge uncertainty in their own responses.
4. Calibration and Information scores are computed for each expert e.
Method Overview (2)
5. The two measures (calibration and information) are combined to
form a weight for each expert.
This is done using strictly proper scoring rules: experts receive
their maximal expected weight only by stating their true
degrees of belief over all the items
6. Experts are then elicited individually regarding their uncertainty
judgments in relation to questions of interest, and the
performance-based calibration scores obtained in Step 4 are
applied to the individual responses.
The result is a combined output, which displays the “best
answer” but more importantly, weighted poolings of the
group of experts’ uncertainty distributions around the best
answer
Method
Expert Elicitation and the Quantification of risk estimation
and uncertainty in relation to infection disease risk
•
Dr. Roger Cooke
(TU Delft, Resources for the Future)
–
–
•
“Classical model” Methodology for combining
expert opinion
Rational Consensus
Dr. Willy Aspinall
(Aspinall & Associates)
–
Facilitator and consultant
The Software
EXCALIBUR TU Delft
http://dutiosc.twi.tudelft.nl/~risk/
Expert Elicitation (Cooke’s method)
The five steps
1. Experts Selection
A group of problem domain experts (E) is selected
2. Calibration
These experts assess a set of (n) variables within their field
(‘seed items’), the true values of which are known
Each expert expressing his or her views as elemental
uncertainty distributions with quantitative support across
selected inter-quantile ranges …
E
Number of experts responding
e
One representative expert
{ X1 , ..., X n }
The set of seed/calibration variables to be assessed,
where n is the number of seed variables
One representative calibration question
i
{x1 , ..., xn }
The set of true values for X (recall: calibration
requires a set of carefully chosen assessment
questions with known values)
Ex.
x
5,i ,e
, x50,i ,e , x95,i ,e 
The set of an expert’s best estimate (median)
and 5-95% confidence bounds for each calibration
question i
Given
x
5,i ,e
, x50,i ,e , x95,i ,e  , the 5, 50, and 95 percentiles
split up the variable x’s range into 4 intervals
I1, i , e  [mi , x5, i , e ]
I 2, i , e  ( x5, i , e , x50, i , e ]
I 3, i , e  ( x50, i , e , x95, i , e ]
I 4, i , e  ( x95, i , e , M i ]
The intrinsic range [mi , M i ] of variable i is the smallest interval such that
mi  xi  M i and mi  xq,i, e  M i for all e  1, 2, ..., E , and q  5, 50, 95 .
He: Expert e is well-calibrated if:
Each of I1,i ,e ,..., I 4,i ,e is drawn independently according to p distribution,
p  ( p1 , p2 , p3 , p4 )  (.05, .45, .45, .05)
Experts are scored via two measures:
Calibration score: likelihood that expert
distributions over the set of seed items
correspond to the observed or measured
results based on a chi-squared test
Information score: measure of
“informativeness” compared to a given
background distribution, usually a uniform
or log-uniform distribution
Expert Elicitation (Cooke’s method)
2. Calibration (Cont’d)
Each expert gives his/her best guess within the intrinsic range [m, M]
the median (X50%), the 5% (X5%) and 95% (X95%) confidence bounds
5%
m X5%
45%
45%
X50%
5%
X95%
M
Expert Elicitation (Cooke’s method)
True value
for variable
2. Calibration (Cont’d)
1
m1
X1,5%
X1,95%
X1,50%
M1
n
mn
T1
s1 
n
Xn,5%
T2
s2 
n
Xn,50%
Xn,95%
T3
s3 
n
Mn
T4
s4 
n
Ti is the number of times the true values lie in the ith interval out of n
s3
s1
s2
s4
( s | True Probabilities)  s1 ln( )  s2 ln( )  s3 ln( )  s4 ln( )
.05
.45
.45
.05
Expert Elicitation (Cooke’s method)
2. Calibration (Cont’d)
( s | True Probabilities)  s1 ln(
s
s1
s
s
)  s2 ln( 2 )  s3 ln( 3 )  s4 ln( 4 )
.05
.45
.45
.05
H0: Expert is well Calibrated
 H0: (S1,S2,S3,S4)= (.05,.45,.45,.05)
Density of Chisquare
distribution with
3 degrees of
freedom
p-value
2n I(s|p)
Expert Elicitation (Cooke’s method)
3. Scoring
Based on two
measures of performance
Calibration Score (CS)

 Pr
2
 2n I ( s | p )


M m
ln 
X
.05
.45
.45
.05 
(
X

m
)
(
X

X
)
(
X

X
)
(
M

X
)
5
50
95
95
 1 n5 
 
M i  mi50


1

ln


Information Score (IS)
 ( X i ,5  mi ).05 ( X i ,50  X i ,5 ).45 ( X i ,95  X i ,50 ).45 ( M i  X i ,95 ).05 
n
i 1
3

X
Indicator (CS>α)
=
Score of the Expert
X50%
m
α is of X
value
maximizes theX
scores
95% M
5% that
of the decision maker
w

Expert Elicitation (Cooke’s method)
4. Experts Elicitation
Experts are then elicited individually regarding their uncertainty
judgments in relation to questions of interest (‘target items’), again within
their domain of expertise.
5. Aggregation
The weighted pooling of the group of experts’ uncertainty distributions
(hi,e) gives the so-called ‘Global decision maker’ (DM). There are several
forms that the DM can take, the weighted average is one of them
E
DM  i  
 w e h
i,e
e 1
E
 w e
e 1
Risk Management of TSE’s:
Structured Expert Elicitation
Following calibration,
the experts were asked to answer
22 target questions
Seven Questions out of the Twenty Two (1/2)
1. What is the dose in grams that would result in 50% of the exposed
population becoming infected from human consumption of BSE
cervical spinal cord or brain stem from near clinical or clinical cases?
(grams)
2. What is the current prevalence of vCJD infection in the Canadian
population? (1 in xxx)
3. What is the mean incubation period for a primary vCJD infected
human [by oral route]? (years)
4. What is the median length of time in months between the oral
infection of a human with BSE agent and the capability of one
unit of his/her blood to transmit vCJD? (months)
Seven Questions out of the Twenty Two (2/2)
5. What is the mean incubation period for a secondary vCJD
infected human [by transfusion]? (years)
6. What is the probability of transmission of vCJD by contaminated
neurosurgery instruments if reused in neurosurgery after one cycle of
standard sterilization procedures? (%)
7. How long is the incubation period for secondary transmission of vCJD
by vCJD contaminated neurosurgical instruments [brain to brain
usage]? (months)
What did the experts
say?
Target Question #1
What is the dose in grams that would result in 50% of the
exposed population becoming infected from human
consumption of BSE cervical spinal cord or brain stem
from near clinical or clinical cases? (grams)
Best expert judgment: about 1 gram.
Target Question #2
What is the current prevalence of vCJD infection in the
Canadian population? (1 in xxx)
Best expert judgement: the current prevalence of
vCJD in Canada is less than 1 in 500,000.
Twelve Risks and Pairwise Comparisons
Twelve Risks and Pairwise Comparisons
Platelet transfusion
FFP plasma transfusion
6
4
Whole blood transfusion
Dura Mater transplant
Packed red blood cells
Dental tissue graft
2
1
5
12
Corneal transplant
3
Hematopoietic stem cell transplant
10
Human derived urine fertility products
Bone marrow transplant
8
pdFVIII
pdFXI
9
7
11
Twelve Risks and Pairwise Comparisons
Platelet
transfusion
FFP plasma
transfusion
Whole blood
transfusion
Dura Mater
transplant
Packed red
blood cells
Dental
tissue graft
Platelet
transfusion
FFP plasma
transfusion
Whole blood
transfusion
Dura Mater
transplant
Packed red
blood cells
Dental
tissue graft
Twelve Risks and Pairwise Comparisons
Platelet
transfusion
FFP plasma
transfusion
Platelet
transfusion
>
Whole blood
transfusion
Dura Mater
transplant
Packed red
blood cells
Platelet transfusion > FFP plasma transfusion
<
=
Dental
tissue graft
>
>
> blood transfusion
>
<
FFP plasma transfusion
transfusion > Whole
<
FFP plasma
Whole blood
transfusion
>
>
<
Platelet transfusion <
> Whole blood transfusion
Circular triads
Dura Mater
transplant
>
Packed red
blood cells
>
>
Dental
tissue graft
Preliminary Analysis
• Test of Inconsistency
To test whether each expert gave his/her responses randomly
Chi-square test with 21 degrees of freedom
11 tests are significant with p-value<.01
• Measure of consistence
2 experts
2 experts
1
Preliminary Analysis
• Statistical Test of Agreement
To test whether the agreement between the group of experts is due to chance
Chi-square test with 90 degrees of freedom
Test is significant with p-value<.01
• Coefficient of Agreement
CoA=.3218
Probabilistic Inversion
•
Assign a random utility function (U1 , …, U12) to
each item
•
Scale the utilities to the interval [0,1]
•
From the aggregated preference matrices
number of times item i is prefered to item j
Pij 
E
P(U i  U j ) is estimated by Pij
Probabilistic Inversion
 ...
[0,1]n
{ui u j }
f (u1 , ..., un ) du1
dun  P(Ui  U j )  Pij
Given the joint density function f, we can find Pij
Given Pij ,we want to find the joint density function f
Space of all
Utilities
f
Space of Pairwise
Ordered Utilities
I( Ui >Uj ) and
Pij
Probabilistic Inversion
• Numerical algorithms
1. Iterative Proportional Fitting (IPF)
2. Parameter Fitting for Uncertain Models
(PARFUM)
• UNIBALANCE

TU Delft
http://dutiosc.twi.tudelft.nl/~risk/
Probabilistic Inversion
• Marginal distributions
• Items’ scores (the mean of the random
utility)
• Correlation
Results
Transmission Route
Score
St. dev.
1
Platelet transfusion
0.5599
0.2318
2
FFP plasma transfusion
0.6266
0.2501
3
Whole blood transfusion
0.7384
0.2075
4
Dura Mater transplant
0.9520
0.0370
5
Packed red blood cells
0.6002
0.2742
6
Dental tissue graft
0.2756
0.2484
7
Corneal transplant
0.6953
0.2263
8
Hematopoietic stem cell transplant
0.3197
0.1752
9
Human derived urine fertility products
0.3160
0.2034
10
Bone marrow transplant
0.4966
0.2195
11
pdFVIII
0.4353
0.2524
12
pdFXI
0.3992
0.2340
Results
Relative
Ranking
Transmission Route
Score
(normalized)
1.
Dura Mater transplant
1.0000
2.
Whole blood transfusion
0.6842
3.
Corneal transplant
0.6206
4.
FFP plasma transfusion
0.5190
5.
Packed red blood cells
0.4799
6.
Platelet transfusion
0.4203
7.
Bone marrow transplant
0.3268
8.
pdFVIII
0.2360
9.
pdFXI
0.1827
10.
Hematopoietic stem cell transplant
0.0652
11.
Human derived urine fertility products
0.0597
12.
Dental tissue graft
0.0000
Results
Results
BSE and vCJD EE Exercises in Ottawa
March 2008, 2009
• What is the size of the bovine to human species barrier
in the MM genotype for oral exposure to the classical
BSE agent?
2008
2009
More research is needed
• Opinion aggregation for different inter-dependent
schools
• Testing inconsistency of experts’ opinions
• Finding numerical algorithms for probabilistic
inversion method
• Fuzzy logic in EE
Submitted …
1. Comparative Expert Judgment Elicitation using the
Classical Model and EXCALIBUR under Conditions of
Uncertainty for Prion Disease Risks
2. Expert Elicitation for the Judgment of Prion Disease
Risk Uncertainties using the Classical Model and ,
EXCALIBUR and UNIBALANCE
By Michael G. Tyshenko, Susie ElSaadany, , Tamer Oraby,
Shalu Darshan, Willy Aspinall, Roger Cooke, and Daniel
Krewski
Conclusion
•
•
•
Best communication of complex methods to
stakeholders?
Validity of complex methods when high uncertainty
involved?
Best contribution in the name of precautionary
principle?
–
National – international collaboration in
development of specific tools
Involvement of all partners
–
–
Face to face meetings, scientific and policy
networks combined.
If you will begin with certainties, you
shall end in doubts, but if you will content
to begin with doubts, you shall end in
almost certainties.
- Francis Bacon
Funded by High Impact Grant
Project teams from:
•
Public Health Agency of
Canada, Statistics and Risk
Assessment Section
Dr. Susie ElSaadany
Angela Catford
Caroline Desjardins
•
McLaughlin Centre for Population
Health Risk Assessment,
University of Ottawa
Dr. Daniel Krewski
Dr. Michael Tyshenko
Dr. Shalu Darshan
Dr. Mustafa Al-Zoughool
Dr. Tamer Oraby