Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Management Response Action Plan Follow-up Process
Document related concepts
Transcript
Management Response Action Plan Follow-up Process * Canadian Evaluation Society 2009 Conference May 31 - June 3, 2009 William Blois, Environment Canada Mary Kay Lamarche, Human Resources and Skills Development Canada Judy Lifshitz, Public Health Agency of Canada Shannon Townsend, National Research Council Canada *The views expressed represent those of the presenters and not necessarily those of any federal department or agency. 1 “An evaluation was completed, and recommendations were made. Did anything change?” 2 Overview Contextualizing evaluation follow-up Exploring governance and accountability requirements Designing recommendations Developing follow-up guidelines and criteria Communicating requirements Establishing timelines Managing and tracking recommendations Reporting and secondary actions Dialogue 3 Contextualizing Evaluation Follow-up Federal Government Federal Accountability Act Treasury Board Policy on Evaluation Transfer Payment Policy Management Accountability Framework Departmental Performance Report Report on Plans and Priorities Program Performance Measurement Framework Other Levels of Government, NGO, etc. Public accountability requirement Funding requirement Board accountability Culture of continuous program improvement and improved program design Simply sound management practice 4 Contextualizing Evaluation Follow-up (con’t) Management Response Action Plans… Heighten accountability Inform priority setting and decision making Improve performance 5 Exploring Governance and Accountability Requirements Issues & Challenges Defining roles and responsibilities: Non-response: Who should be doing follow-up? Who writes the action plan? Who has final sign-off? What if a program doesn’t provide an action plan? Non-compliance: What if a program doesn’t provide a response to their action plan? Lessons Learned Establish clearly defined roles and responsibilities. Communicate defined roles and responsibilities as early as possible. 6 Designing Recommendations Lessons Learned (con’t) The recommendation should be written at an appropriate level to elicit a clear and actionable response. Characteristics of a strong recommendation: Brief and clear: Single sentence, clearly articulating what should be done based on the evidence of the evaluation. Ensures the elicitation of clear and actionable responses Non-prescriptive: Does not outline how the how the result is to be achieved. It may, however, provide suggestions that can be considered for achieving the desired end state. 7 Designing Recommendations (con’t) Lessons Learned (con’t) Target the recommendation: Establish, within the recommendation, the person accountable or responsible for acting upon it. Assign a risk rating or a priority: Where there may be multiple recommendations and limited resources, determine priority based upon risk. Use results-based language. 8 Examples of Recommendations Poor practice “In light of multiple findings that suggest a lower level of performance than expected, the program, given available resources and constraints, may wish to consider discontinuing to pursue activities in certain areas that are less relevant or, to reduce resources to streamline operations so that greater efficiencies are reached.” 9 9 9 Lengthy Complex and unclear Double barrelled Good practice “It is recommended that the program fund be targeted at the most critical horizontal federal priorities.” Brief 9 Clear 9 Actionable 9 9 Developing Management Response Action Plans 10 Common Criteria for Management Response Action Plans A plan may include the following: Recommendation statement Brief rationale Indication of acceptance or non-acceptance of the recommendation E.g., Accepted, accepted in part, rejected Explanation or justification required if the recommendation is not accepted Clearly defined action items to respond to the recommendation Timelines for implementation for each identified action item Responsibility for implementing the identified action items Expected indicators to demonstrate implementation of management action. 11 Sample Management Response Action Plan Template – Instructions1 Opinion of the Evaluation Report: Respondents are asked to provide a brief preamble offering an opinion on the evaluation report and their general view on its accuracy (Note that every effort should be taken to work with evaluators to correct errors or inaccuracies in the report before a response is given). Recommendation Drawn directly from the evaluation report. 1 Response and Planned Action(s) Responsibilities Timelines Measures of Achievement The program may choose to “accept” or “reject” in part or in full the recommendation. A response to the recommendation is expected whether it is accepted or not. Planned actions in relation to the response should be provided, if the recommendation is accepted. If the recommendation is accepted, then those responsible for the proposed actions should be identified. Responsibilities may be shared. If the recommendation is accepted, then timelines for the planned actions should be identified. Followup will occur with the program on progress vis-à-vis planned actions and will be reported to the DEC. If applicable, identify measures that demonstrate achievement of the results intended by the recommendation. Planning and Performance Management, National Research Council Canada 12 Issues & Challenges Evaluation recommendations are sometimes too broad, making it difficult to identify specific action items to address the issue. Responses or identified actions are often not concrete and actionable. Programs may be defensive when responding to the recommendations rather than focusing on ways to address the noted issue or problem. Highlight for programs that rejection of recommendations is an option. 13 Issues & Challenges (con’t.) Sometimes it is difficult for a program to implement actions because: It lacks resources (human, financial). It is dependent upon the actions of other areas, or other delivery agents or partners (e.g., horizontal evaluations). They are not considered a priority or there is a lack of buy-in. Not enough time elapses in order for implementation to occur (at time of follow-up). There is organizational change (e.g., mandate, structure). 14 Lessons Learned Easiest when responding to a strong recommendation. When developing management response action plans, systematically respond to the recommendations. When identifying action items, consider the feasibility of implementation in terms of time, resources, priorities, etc. Consider the results or impacts of actions to be taken and how these will be measured to determine success. Link management actions to ongoing initiatives within the organization to ensure that they are complementary and efficient. Systematically implement action items. 15 Sample Management Response Action Plan Template - Completed1 Opinion of the Evaluation Report: We have reviewed the evaluation and feel that it accurately reflects the history of the implementation of the initiative, and the results achieved to date. Recommendation The Institute should establish an accountability framework between itself and ABC organization to clarify the roles and responsibilities of the two under the existing agreement. 1 Response and Planned Action(s) Responsibilities Accepted. We will convene a meeting of both parties in order to clarify roles and responsibilities under the agreement. The framework will outline who is responsible for day-to-day management, facilities maintenance, communications, marketing, achievement of final results, etc. DG, Institute and Executive Director, ABC organization Timelines 1 November 2006 (signed, final framework) Measure(s) of Achievement • Established accountability framework that has been agreed to by both partners. Planning and Performance Management, National Research Council Canada 16 Communicating Requirements 17 Issues & Challenges Level of effort required by evaluation and the program. Identifying progress indicators for management actions. Follow up on areas where a recommendation was not made but identified as a weakness or opportunity. 18 Lessons Learned To be successful, evaluators should work with program managers to help them understand how to proceed. Meet with them to review recommendations and to assist in constructing responses and appropriate actions. Give them guidelines by identifying: Who should develop the responses and actions A timeline for submission of information Sample responses, actions and indicators 19 Lessons Learned (con’t) Communicate key messages as early in the process as possible: Outline who will use the follow-up information. Clarify the process and how a response should be laid out. Balance providing information with being concise and not overwhelming the program E.g., program managers, senior managers in the organization, oversight committee (e.g., board of directors), departmental evaluation committee, etc. Outline how it will be used: For decision-making For information For program improvement Reported publicly 20 Establishing Timelines 21 Issues & Challenges Defining the frequency under which evaluation recommendations are followed up on: Defined period (e.g., 6 or 12 month follow-up). Variable (e.g., according to the date of planned completion of the recommendation) Limiting the burden on the program and the evaluation team: Need to fit follow-up into regular program and evaluation activity. Need to make it relatively simple and easy to do. 22 Lessons Learned It is best to determine at what interval follow-up will occur: Too frequently, or too soon, can result in ‘buy-out’ rather than ‘buy-in’ A poorly timed follow-up may limit responsiveness and action Communicate ahead of time when you’ll be back to follow-up. Set up a ‘bring forward’ (BF) system. 23 Managing and Tracking Recommendations 24 Considerations Consider the use of a database. It may yield process efficiencies and provide a one-stop shop for: Recommendations Management actions Expected date of implementation Person responsible previous updates and analyses Status (open/closed) Date of previous update Date for subsequent follow-up A system can link to submitted information (stored separately) and previous analysis. This reduces the burden in accessing information and enhances the consistency of analysis. 25 Issues & Challenges (con’t) Accessibility of software may be key if responsibility for follow-up is shared among different evaluation leads. Alternatively, it needs to be an assigned function. Manner in which management actions are tracked depends to some extent on the process adopted for follow-up: Fixed-schedule (annual or bi-annual) follow-up can use simple tracking database (e.g., Access or Excel). Variable-schedule, based on unique implementation dates, may require a more sophisticated system (e.g., Team Mate Software). 26 Issues & Challenges (con’t) Automated Database Advantages Easier updates Simple review of all management actions Common reporting framework Ease of data aggregation across programs Possibility of automated reminders (evident only for variable schedule follow-up process) Disadvantages More costly (unless existing systems can be adopted/adapted) More time consuming to implement Training required Access may be limited (not all evaluators may have access to the database) 27 Lessons Learned Adapting existing tools can yield efficiencies related to cost and time (e.g., Environment Canada adopting Internal Audit tool). A comprehensive database linked to supporting documentation may enhance process efficiencies and consistency of analysis. 28 Reporting and Secondary Actions 29 Reporting Federally, follow-up results are reported to the Departmental Evaluation Committee. Others may report to boards, funders, program staff, the public, etc. Reporting can take a number of forms. It may vary according to prominence of the exercise and/or frequency of follow-up. Level of detail affects the degree to which results are actionable. At a minimum, it is important to report progress directly to responsible senior managers. 30 Non-Compliance At a federal level, no formal practices have been established to address non-compliance with management actions. At the discretion of the Deputy Head or senior management. Needs to be considered in the proper context (e.g., continued relevance of management action, competing priorities). Existence and implementation of a follow-up process may itself lead to higher compliance rates. For others, non-compliance may result in: Limited program improvement Loss of funding Loss of public confidence 31 Buy-in to Management Actions Evaluation buy-in and ownership of management actions may be the best guarantee that actions will be implemented. Early consultations ensure that addressed issues (and subsequent recommendations) are relevant to programs Higher-level, non-prescriptive recommendations focussed only on the desired goal allow programs to take ownership and determine how best to respond. Clear communications and frequent interaction with the program ensure understanding of the value of the evaluation and related recommendations. 32 Dialogue What is your ‘context’? What challenges does your organization face in getting buy-in on recommendations? Do programs respond with meaningful action plans? Where does the responsibility for ensuring action against recommendations fall? What role should evaluators play versus senior managers? What are the keys to success in ensuring recommendations are acted on? What can you share with others about effectively following up on recommendations? What processes do you use? What have been your lessons learned? 33
