Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Sharing information about choosing and using evaluation options Patricia Rogers – RMIT University Canberra Evaluation Forum 20 June 2013 Canberra Challenges It is hard to choose the right combination of evaluation methods and proceses It is hard to know how to implement them well Opportunities Lots of material on evaluation methods (some not maintained) Many people willing to share their experience and expertise Documenting Sharing R&D Events Descriptions Comments Examples Tools Guides Financial Supporters Founding Partners Contributors and collaborators include … Co-creation by practitioners, researchers, sector experts, method experts Reflection, processes for quality and authenticity Rooted in practical experience Advice from experienced practitioners Photo: David Masters Task oriented taxonomy: Support to navigate options FRAMEWORK Over 200 methods/options related to 35 tasks in 7 clusters Methodological pluralism Simon Kneebone ‘Show Me The Change” http://www.flickr.com/photos/smtc/sets/72157624112171834/ Spotlight on invisible evaluation tasks and less well known methods 16 Respectful, inclusive and constructive discussion 17 Scenario 1- describing a situation • Community renewal project • Have existing social indicator profile • Want to learn how the residents see their community – what are its strengths and problems? • Intended users of the evaluation – local authorities for planning and advocacy • What methods might you use – or suggest in a ToR? • Under what circumstances might you use them? • How could you oversee the quality of their use? Information from individuals • Deliberative Opinion • Polls • • Email Questionnaires: • • Face to Face • Questionnaires • • Global Assessment • Scales • • Goal Attainment • Scales (GAS) • • Internet Questionnaires • • Interviews • • Key Informant Logs and Diaries Mobile Phone Logging Peer/Expert Reviews Photolanguage Postcards Projective Techniques Questionnaires Seasonal Calendars Sketch Mapping Stories (Anecdote) Telephone Questionnaires Information from groups • After Action • Review • • Brainstorming • Card Visualization: • • Concept Mapping • • Convergent Interviewing • • Delphi Study • • Dotmocracy • • Fishbowl • Technique Focus Groups Future Search Conference Hierarchical Card Sorting Keypad technology Mural ORID SWOT Analysis World Cafe Scenario 1- PhotoVoice Participatory photography method Particularly useful for giving a voice to marginalized people Community members : • Take photos • Select the most significant photos • Tell stories about what they mean • Identify key themes What questions would you ask? Scenario 2(a)- causal inference • Pilot project of service delivery to individual clients • Intended users – department managers to inform decision about future funding • What methods might you use or suggest in a ToR? • Under what circumstances might you use them? • How could you ensure quality? Scenario 2 (b) - causal inference • Completed health intervention to reduce early childhood obesity • Series of evening information sessions • Ten participating families • Data (from service records) show children’s weight was high at the beginning and lower afterwards • Intended users – management of health service to inform decision about future funding • What methods might you use – or suggest in a ToR? • Under what circumstances might you use them? • How would you ensure quality? Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin Weights of subsequent children Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin Scenario 2 - Check timing is consistent with program theory • Use time series data to go beyond just ‘Before’ & ‘After’ • See if the change trajectory is consistent with the theory that the program has caused the observed results Scenario 2 - General Elimination Method • Identify possible alternative explanations • Gather and analyze data to see if these can be ruled out Scenario 2 - methods for causal inference • Which methods were suggested by the group? • Which methods were not suggested? • What new methods sound potentially useful? • What information would you need to be able to use (or oversee someone using) a new method? Content • Over 200 options pages • Over 500 resource pages • 16 approaches (package of methods) • 8 thematic pages – sectors or crosscutting issues Choose the right option Use it well Information on methods : • Description • Examples • Advice for choosing the method • Advice for using the method • Short list of recommended resources • A full list of all resources, ordered by user rating • This information helps users determine if the method suits their needs and requirements www.betterevaluation.org How could you benefit from BetterEvaluation? Advice for choosing appropriate methods Advice on applying methods effectively A framework for thinking about evaluation design Discover new methods Learn from other practitioners and experts Library of resources, case study, links and people How could you contribute to BetterEvaluation? Suggest methods Comment on content Improve the description Suggest resources Rate resources Curate a section Host a writeshop Financially support BetterEval Spread the word Partnering with BetterEvaluation An AusAID perspective Context • Pressures to demonstrate effectiveness of growing aid program • Independent Review of Aid Effectiveness recommendations ‾ Strengthen our evaluation approach (evidence) ‾ More rigorous reporting on aid program effectiveness ‾ Standards for whole of government aid initiatives • All projects (>$3m) to undertake an evaluation • Challenges ‾ ‾ ‾ ‾ lack of baselines/data, range of programs and partners, constraints of working in developing countries diverse staff training needs Why partner with BetterEvaluation? Because it’s GREAT! • Generous – free access • Rigorous –promotes rigor and good quality resource materials • Easy – to navigate and interactive options • Accessible & Active – housed in Australia but often travelling! • Timely – coincides with increasing demand for monitoring and evaluation tools by aid program and other managers What do we get out of it? • Helpful – practical advice & insights • Encouraging – opportunity for feedback / discuss ideas • Leveraging Opportunities – access to new networks & opportunity for us to influence direction • Promotion – opportunities to share our evaluation products and learning What’s next? To GROW the Partnership • Go places – BE is expanding relationships with other organisations and agencies/greater reach • Reach out Overseas – to our partners in AsiaPacific, development of local case study materials in local languages & translations • Work more collaboratively – stronger focus on developing Canberra and Post skills & increasing participation on the site Questions/Comments? Further Information Comprehensive Aid Policy Framework http://www.ausaid.gov.au/about/Documents/capf.pdf 2012 Annual Review of Aid Effectiveness http://www.ausaid.gov.au/makediff/Pages/annual-review-aideffectiveness.aspx 2013 OECD DAC Peer Review of Australia http://www.oecd.org/dac/peerreviews/OECD%20Australia%20FinalONLINE.pdf ODE Website: www.ode.ausaid.gov.au