Download Policy Evaluation Primer

Policy Evaluation Primer
Action Research Cycle



Stage One: Problem Identification:
Acknowledge the need for change. Can be an
existing, even historical, problem, or an
emerging issue.
Stage Two: Evaluation: Develop and carry
out methods for evaluating the breadth and
depth of the issue and how it informs
practice.
Stage Three: Recommendations: Based
on the evaluation, provide specific
recommendations for change and/or
continued evaluation.



Stage Four: Application/Practice: Work
with the powers that be to take action and
institutionalize the recommendations.
Stage Five: Reflection: With changes in
place, reflect on ways in which new practices
affect the work environment. Concurrently,
reflect on what you, as an individual
researcher, and/or the team learned from the
process of the research.
Stage Six: Consideration of New
Questions: Acknowledge and dialogue about
new questions that have emerged from the
changes. Have the changes worked? Are
there any shortcomings?
Problem identification:


What is the problem or issue you wish to
examine?
Is the problem an ill-structured or
structured one?
• Initial situation clearly defined; one desirable
goal; clear dominant means to achieve that
goal
• Many ways to describe problem, complex goal
state, values matter, no one clear best goalstate, etc.
• Structuring ill-structured problems
complicated, but necessary
Problem identification:



Who is this a problem or issue for?
Who might be impacted by an eventual
resolution of the problem or issue? (e.g.
who are the stakeholders?)
What is your unit of analysis?


Problem Formulation
“Some ways to formulate complex
social problems surely are more useful
than others in a given situation, but there
is never a single, clearly correct problem
definition on which analysis can converge.
Nor is there any purely analytic way of
specifying which problems, once
formulated, ought to be higher than
others on the political agenda.” Lindblom and
Woodhouse P.22
How do we evaluate policies?

What’s the goal?
• To develop improved policies


Academic/’wonk’ process
Political process
• To critique policies we disagree with
• To find support for policies we agree
with
• To know what position to declare on a
specific policy

Who’s the audience?
• Are you responding to a request from a
politician/decision maker?
• Are you working to develop and refine
general policies (e.g. academics)?
• Are you working to promote specific
policies?

“If professional analysis deserves to
be seen as no more than an input to
political interaction and judgment,
never a substitute for it, then inquiry
and judgment by ordinary people
remain at the heart of the policymaking process.”
Lindblom and Woodhouse P.22
Evaluation requires analysis


To increase our understanding
To solve problems
Analysis




A search for understanding: why and
how?
Not description
We break a problem down into
component parts to simplify analysis
How does a car work?
• Take it apart and figure out how each
part works?
• Not enough– need to figure out the
interactions between the parts
• Same is true for policies
Evaluation in Complex vs. Simple
Problems
Traditional Scientific Method









Objective is to understand
Value-free puzzle solving exercise
Careful observations and measurements
Form falsifiable hypothesis
Repeated experiments to test hypothesis
If hypothesis is falsified, form new hypothesis
When hypothesis can’t be falsified, it becomes a
theory
Only expert opinions matter
Replicability, analytical rigor and peer review
determine quality
Ideology



Continued belief in theory even when
contradicted by empirical evidence
Refusal to test theories empirically
“The terms of the debate can be
grounded wholly in how the world
ought to be, not in how it is.”
Charles Murray
Public Policy Problems

System is in a constant state of rapid
change, and may be beyond our
comprehension.
• Complexity, non-linearity, chaos, evolution and
unexpected surprises

Not value-neutral: Multiple legitimate
perspectives
• Ethical values, not objective facts

Uncertain facts
• Lack of baseline data
• Different sets of facts
• Different interpretation of facts
Public Policy Problems



High stakes
Urgent decisions
Ill-structured problems
• Poorly defined problems
• Poorly defined solutions: usually many
possibilities

Difficulty of experimentation
•
•
•
•
sample size of one
ethical issues
constant change
scaling
Causality and complex systems


Feedback loops
Positive feedback loop
• Land prices increase, speculative demand
increases, increased demand increase land
prices…
• Greenspan says housing bubble exists, people
stop speculating, prices fall, people sell, prices
fall more…

Negative feedback loops
• Milk prices increase, farmers expand
production, increased supply drives price back
to equilibrium.
Ways of Stating Casual Relations
Religious Attendance and Divorce Rates

Religious attendance
____________ reduced divorce.
“causes”
“leads to”
“is related to”
“influences”
“is associated with”
“produces”
“results in”
“reduces the likelihood of…”
Post Normal Science




Facts are
uncertain
stakes are high
decisions are
urgent
Values matter
Transdisciplinary



In the university there are
disciplines, in the real world
there are problems
Communication and
understanding across
disciplines is essential
Communication with sectors
outside of academia is
essential
Identifying and Gathering Data


Define and structure your problem.
Put together outline of what you
need to know. Gaps are often first
information you need to collect.
Make list of data sources
• Best to rely on existing data

How much information do you need?
How much information?


Too much or too little information
Time and Cost
• “Most policy decisions are made using
no more than perfunctory analysis,
because decisions cannot wait until ‘all
the facts are in.’”
Lindblom and Woodhouse p.20

Optimal ignorance
Distinguishing “good” information
from “bad”


Source: peer reviewed, grey
literature, Internet, government
statistics, etc., wikipedia, etc.
Triangulation
Questions to ask:

Data collected independently?
• Examples





How were the data collected?
Why were the data collected?
When were the data collected?
Who collected the data?
Are only selected data being
reported?
Quantitative vs. Qualitative



Obsession with quantification and
‘academic purity’
Numbers and symbols
Interpretation of the facts
Literature review: Goals




To demonstrate a familiarity with a body
of knowledge and establish credibility.
To show a path of prior research and how
current analysis is linked to it.
To integrate and summarize what is known
in an area.
To learn from others and stimulate new
ideas.
Neuman W.L. 2000, p446
Interviews



Standardized questionnaires
‘Elite’ or specialized interviewing
Stakeholder interviews
Communication







Your communication must have a specific purpose. You
must fully understand this purpose, and know how to
articulate it to others;
You must communicate to the appropriate audience, in a
format suitable to that audience and to the situation;
Your content must be well organized, concise and to the
point;
You must capture and hold the attention of your audience;
Your graphic material must be suitable for the audience,
clear and easy to understand, and help to convey your
message;
You must be credible, and your information must be
credible; and
You must get critical evaluation of polished draft documents
or practice presentations