Download A hierarchical evaluation process to adopt new content areas for Tracking in Minnesota

A hierarchical evaluation process to adopt new content areas
for Environmental Public Health Tracking (EPHT) in Minnesota
Blair Sevcik MPH, Jeannette Sample MPH
Background
Phase 2 Evaluation Questions
For states with Environmental Public Health Tracking (EPHT) programs, allocating
limited resources for the selection and adoption of new content areas is not a new
issue. To address this issue, Minnesota EPHT has developed and refined a
hierarchical evaluation process to guide the process of adopting new statespecific content areas. Other exploration into the topic of identifying and
evaluating new content areas for indicator-based systems has been done.1
Using existing environmental health indicator models and evaluation frameworks,
staff first developed a list of selection criteria. Staff then prioritized the criteria into
a four phase hierarchical evaluation process that builds a scientific rationale for
including potential content areas into MN EPHT: Exploration, Feasibility,
Recommendation, and Implementation. In order to best utilize resources, the
Advisory Panel is informed and consulted after work for each phase is complete.
Work may halt if the rationale for pursuing a content area is judged not robust.
For each phase, MN EPHT identified an objective, key concepts, criteria and
related questions, tasks and deliverables, and involvement. Throughout the
evaluation process, staff consult the Technical Team for input on criteria and
rationale, as well as provide progress updates to the Advisory Panel. Using input
from Technical Team and Advisory Panel members, Project Management decides
whether the completed criteria support further exploration in the next phase.
Objectives
Methodology
Phase 1:
Phase 2:
Phase 3:
Phase 4:
Exploration
Feasibility
Recommendation
Implementation
DETAILED FEASIBILITY
•What is the data quality?
•Is there continuity? Are the data timely? Are the data comparable?
•Is aggregation possible at different geographic levels?
•What is the cost to MDH to obtain data?
Phase 3 Evaluation Questions
EMERGING ISSUES
Key Concepts
Under Minnesota law, an external stakeholder Advisory Panel is involved in
advising the Commissioner of Health on EPHT program priorities in Minnesota.
Ideas for Minnesota-specific content areas and measures could come from a
variety of sources, including staff, Advisory Panel members, the public, nongovernmental organizations, local public health officials, or legislators. Given
limited time and resources, MN EPHT staff determined that a documented
process to evaluate suggestions was needed to help guide the Advisory Panel
recommendations for Minnesota-specific content areas.
Figure 1: Flowchart representing the four-phase hierarchical
evaluation process for selecting new EPHT content areas
•Explore rationale
for including new
content area
•Determine if there
are resources to
evaluate new
content area
•What is the public
health
importance?
•Do available data
exist?
•Do resources
exist to gather
data?
•Examine ways
data can be
visually displayed
•Are data robust
and of high
quality?
•Can data be
manipulated in
useful ways for
tracking?
•Evaluate how
content area
perceived by
EPHT and
partners
•Present results to
Advisory Panel
•How would
content area
affect EPHT and
our partners?
•What is the final
summary
assessment of
the content area?
•Incorporate
content area into
EPHT structure
as a continuing
tracking measure
•Is the degree or level of exposure changing or perceived to be changing?
INFORMATION BUILDING
•Is this a hazard with unknown associations to health outcomes OR unknown level
of exposure in the population?
•Are there other programs at MDH that would be interested in this content area?
OUTSIDE INTEREST
•How are data to
be consistently
displayed and
discussed by
EPHT?
•Is there a high concern regarding the proportion of the population exposed to the
hazard?
•Is the exposure or disease a priority that has previously been identified by
environmental health organizations?
•Would this content area utilize existing datasets in a new way?
BALANCE
•Is there balance between hazard/exposure and disease content areas?
•Is there balance between age groups affected among content areas?
ECONOMIC IMPACT
•What is the economic impact in Minnesota of adding this content area?
Phase 4 Summary
In 2010, work began to pilot Environmental Tobacco Smoke (ETS) Exposure
through Minnesota’s evaluation process as a potential new content area. The
evaluation process was refined and eventually became the four-phase evaluation
process presented here. A flowchart was produced to guide the evaluation
process which outlines the key concepts and criteria that belonged to each of the
four phases (see Figure 1).
Criteria
Results
A template document was also produced that lists Evaluation Questions related to
the criteria for each phase, to determine the extent to which a new content area
addresses the key concepts in the evaluation process.
•Available
Resources
•Prevalence
•Causality
•Public Health
Impact
•Actionability
•Initial Feasibility
•Detailed
Feasibility
•Literature review
•Find data sources
•Contact data
steward
•List potential
measures
•Select data
sources
•Draft DUA if
needed
•Obtain data
•Pilot potential
measures
Once adopted, final documentation on the content area is created, to include a
“how-to-guide” about how measures are to be calculated, metadata, and
messaging for the data portal. A formal Data Use Agreement (DUA) must now be
signed by the data steward(s).
•Emerging Issues
•Information
Building
•Outside Interest
•Balance
•Economic Impact
Conclusions
In 2011, MN EPHT staff successfully completed the evaluation of ETS Exposure
as a new state-specific content area:
1) ETS Exposure was found to be prevalent, causally associated with adverse
health outcomes, actionable, important to public health, and feasible.
AVAILABLE RESOURCES
•Is there staff time/interest/expertise and financial/technical resources?
PREVALENCE
•Is there a high estimated proportion of the population that is exposed OR a high
estimated prevalence of disease or outcome?
CAUSALITY
•Is there evidence that exposure is a component cause of adverse health
outcomes OR that the disease has an environmental component cause?
Tasks/Deliverables
Phase 1 Evaluation Questions
•Focused interviews
and information
gathering
•Report on all
criteria with final
data and measures
•Summarize
strengths and
limitations
•Formalize DUA
•Create how-to
guide, metadata
and messaging
•Are there existing prevention or control programs at MDH or other Minnesota
organizations for the exposure or its adverse health outcomes?
•Can the level of exposure/disease be modified via policy, regulatory, or personal
actions?
•Is the exposure or disease tied to state or federal public health objectives?
•Can this content area be used to develop new program initiatives?
PUBLIC HEALTH IMPACT
Involvement
ACTIONABILITY
•Consult Tech
Team
•Inform and
consult Advisory
Panel
•Consult Tech
Team
•Inform and
consult Advisory
Panel
•Content area
•Consult Tech Team
•Advisory Panel votes representative
added to Tech
to recommend
Team
adoption
•Inform and consult
Executive Steering
Committee
•Is the population attributable risk (PAR) or public health impact of exposure
known or be estimated from available data? OR Is the severity of the disease
effect known and contributes to mortality or morbidity?
In addition to being of high quality, existing data would be utilized in a new way
with the adoption of this content area. Minnesota has chosen to incorporate ETS
Exposure as a continuing tracking measure. ETS Exposure is currently in Phase 4
(Implementation) and steps are being taken to incorporate this new content area
on Minnesota Public Health Data Access, a web-based data access portal.
Recommendations
This evaluation process successfully addressed public health priorities as well as
stakeholder input and provided clear rationale for adding the content area. Given
the limited time and resources of EPHT states, it is important to develop a strong
rationale prior to adopting new content areas. New content areas come with the
cost of implementation and annual maintenance on state portals, making it that
much more important that states should be selective in adding content areas and
use a process to explore the rationale.
Since EPHT programs are often challenged by resource needs, other states may
find Minnesota’s evaluation process, flowchart, and template document useful in
order to guide the use of resources on new content areas.
Acknowledgement
(INITIAL) FEASIBILITY
•Is there one or more data sources for exploration of “trackable” indicators?
•Does MDH have the legal authority to collect and use the data?
•Are private data classified and protected according to state and federal law?
2) The data for ETS Exposure measures were found to be: available, populationbased, representative of exposure, timely, and comparable.
1
Malecki KC, Resnick B, Burke TA. Effective Environmental Public Health Surveillance Programs: A Framework for Identifying
and Evaluating Data Resources and Indicators. Journal of Public Health Management and Practice. 2008;14(6): 543-551.
We gratefully acknowledge the contributions to this project of the MN EPHT
Technical Team and the Environmental Health Tracking and Biomonitoring (EHTB)
Advisory Panel. Minnesota is part of the National Environmental Public Health
Tracking Network. For more information about Minnesota’s EPHT program, visit
our website at www.health.state.mn.us/tracking, call us at 651-201-5900, or email
us at [email protected]