Download Scenario 2(a)- causal inference

Sharing information about
choosing and using evaluation options
Patricia Rogers – RMIT University
Canberra Evaluation Forum
20 June 2013 Canberra
Challenges
It is hard to choose the right
combination of evaluation methods
and proceses
It is hard to know
how to implement them well
Opportunities
Lots of material on evaluation methods
(some not maintained)
Many people willing to share their
experience and expertise
Documenting
Sharing
R&D
Events
Descriptions
Comments
Examples
Tools
Guides
Financial Supporters
Founding Partners
Contributors and collaborators include …

Co-creation by
practitioners,
researchers,
sector experts,
method experts

Reflection,
processes
for quality
and
authenticity

Rooted in practical experience
Advice from experienced
practitioners

Photo: David
Masters
Task oriented
taxonomy: Support to
navigate options
FRAMEWORK
Over 200
methods/options
related to 35 tasks
in 7 clusters

Methodological
pluralism
Simon Kneebone ‘Show Me The Change”
http://www.flickr.com/photos/smtc/sets/72157624112171834/
Spotlight on invisible
evaluation tasks and
less well known
methods
16

Respectful,
inclusive and
constructive
discussion
17
Scenario 1- describing a situation
• Community renewal project
• Have existing social indicator profile
• Want to learn how the residents see their
community – what are its strengths and
problems?
• Intended users of the evaluation – local
authorities for planning and advocacy
• What methods might you use – or suggest in a ToR?
• Under what circumstances might you use them?
• How could you oversee the quality of their use?
Information from individuals
• Deliberative Opinion •
Polls
•
• Email Questionnaires: •
• Face to Face
•
Questionnaires
•
• Global Assessment
•
Scales
•
• Goal Attainment
•
Scales (GAS)
•
• Internet
Questionnaires
•
• Interviews
•
• Key Informant
Logs and Diaries
Mobile Phone Logging
Peer/Expert Reviews
Photolanguage
Postcards
Projective Techniques
Questionnaires
Seasonal Calendars
Sketch Mapping
Stories (Anecdote)
Telephone
Questionnaires
Information from groups
• After Action
•
Review
•
• Brainstorming
• Card Visualization: •
• Concept Mapping
•
• Convergent
Interviewing
•
• Delphi Study
•
• Dotmocracy
•
• Fishbowl
•
Technique
Focus Groups
Future Search
Conference
Hierarchical Card
Sorting
Keypad technology
Mural
ORID
SWOT Analysis
World Cafe
Scenario 1- PhotoVoice
Participatory photography method
Particularly useful for giving a voice to
marginalized people
Community members :
• Take photos
• Select the most significant photos
• Tell stories about what they mean
• Identify key themes
What questions would you ask?
Scenario 2(a)- causal inference
• Pilot project of service delivery to individual
clients
• Intended users – department managers to
inform decision about future funding
• What methods might you use or suggest in a ToR?
• Under what circumstances might you use them?
• How could you ensure quality?
Scenario 2 (b) - causal inference
• Completed health intervention to reduce early
childhood obesity
• Series of evening information sessions
• Ten participating families
• Data (from service records) show children’s weight
was high at the beginning and lower afterwards
• Intended users – management of health service to
inform decision about future funding
• What methods might you use – or suggest in a ToR?
• Under what circumstances might you use them?
• How would you ensure quality?
Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers.
Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne
College of Advanced Education. Presented as a case study in Owen, J. (1999) Program
Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin
Weights of subsequent children
Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers.
Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne
College of Advanced Education. Presented as a case study in Owen, J. (1999) Program
Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin
Scenario 2 - Check timing is
consistent with program theory
• Use time series data to go beyond just
‘Before’ & ‘After’
• See if the change trajectory is consistent
with the theory that the program has
caused the observed results
Scenario 2 - General Elimination
Method
• Identify possible alternative
explanations
• Gather and analyze data to see if these
can be ruled out
Scenario 2 - methods for causal
inference
• Which methods were suggested by the
group?
• Which methods were not suggested?
• What new methods sound potentially
useful?
• What information would you need to be
able to use (or oversee someone using) a
new method?
Content
• Over 200 options
pages
• Over 500 resource
pages
• 16 approaches
(package of methods)
• 8 thematic pages –
sectors or crosscutting issues
Choose the right option
Use it well
Information on methods :
• Description
• Examples
• Advice for choosing the method
• Advice for using the method
• Short list of recommended
resources
• A full list of all resources, ordered
by user rating
• This information helps users
determine if the method suits
their needs and requirements
www.betterevaluation.org
How could you benefit from
BetterEvaluation?
Advice for
choosing
appropriate
methods
Advice on applying
methods
effectively
A framework for
thinking about
evaluation design
Discover new
methods
Learn from other
practitioners and
experts
Library of
resources, case
study, links and
people
How could you contribute to
BetterEvaluation?
Suggest
methods
Comment on
content
Improve the
description
Suggest
resources
Rate resources
Curate a
section
Host a
writeshop
Financially
support
BetterEval
Spread the
word
Partnering with
BetterEvaluation
An AusAID perspective
Context
• Pressures to demonstrate effectiveness of growing aid program
• Independent Review of Aid Effectiveness recommendations
‾ Strengthen our evaluation approach (evidence)
‾ More rigorous reporting on aid program effectiveness
‾ Standards for whole of government aid initiatives
• All projects (>$3m) to undertake an evaluation
• Challenges
‾
‾
‾
‾
lack of baselines/data,
range of programs and partners,
constraints of working in developing countries
diverse staff training needs
Why partner with BetterEvaluation?
Because it’s GREAT!
• Generous – free access
• Rigorous –promotes rigor and good quality
resource materials
• Easy – to navigate and interactive options
• Accessible & Active – housed in Australia but
often travelling!
• Timely – coincides with increasing demand for
monitoring and evaluation tools by aid program
and other managers
What do we get out of it?
• Helpful – practical advice & insights
• Encouraging – opportunity for feedback / discuss
ideas
• Leveraging Opportunities – access to new
networks & opportunity for us to influence
direction
• Promotion – opportunities to share our
evaluation products and learning
What’s next?
To GROW the Partnership
• Go places – BE is expanding relationships with
other organisations and agencies/greater reach
• Reach out Overseas – to our partners in AsiaPacific, development of local case study materials
in local languages & translations
• Work more collaboratively – stronger focus on
developing Canberra and Post skills & increasing
participation on the site
Questions/Comments?
Further Information
Comprehensive Aid Policy Framework
http://www.ausaid.gov.au/about/Documents/capf.pdf
2012 Annual Review of Aid Effectiveness
http://www.ausaid.gov.au/makediff/Pages/annual-review-aideffectiveness.aspx
2013 OECD DAC Peer Review of Australia
http://www.oecd.org/dac/peerreviews/OECD%20Australia%20FinalONLINE.pdf
ODE Website: www.ode.ausaid.gov.au