Download Guide to Developing a Risk-Based Departmental

Provincial Government Programs
Cabinet Secretariat, Executive Council
Guide to Developing a Risk-Based
Departmental Evaluation Plan
December 2011
Guide to Developing a Risk-Based Departmental Evaluation Plan
TABLE OF CONTENTS
PURPOSE OF THIS GUIDE ........................................................................................... 2
INTRODUCTION ............................................................................................................. 3
OVERVIEW ..................................................................................................................... 4
Why do Risk-Based Planning? ..........................................................................................4
Transitional Considerations...............................................................................................5
When to Start the Planning Process ................................................................................5
GETTING STARTED ...................................................................................................... 8
WHAT GOES IN A PLAN? ............................................................................................. 9
Policy Requirements ............................................................................................................9
Five Step Process ...............................................................................................................10
GLOSSARY .................................................................................................................. 18
DOCUMENTS GLOSSARY .......................................................................................... 21
APPENDIX A: PRE-SCREENING ASSESSMENT TEMPLATE .................................. 26
APPENDIX B: EVALUABILITY ASSESSMENT TEMPLATE ...................................... 27
APPENDIX C: EVALUATION PRODUCT PROFILE ................................................... 28
APPENDIX D: DEPARTMENTAL EVALUATION PLAN .............................................. 29
APPENDIX E: INITIATIVE INVENTORY UPDATE....................................................... 30
APPENDIX F: GOVERNMENT PROGRAM RISK ASSESSMENT TOOL ................... 31
Cabinet Secretariat │ December 2011
1
Guide to Developing a Risk-Based Departmental Evaluation Plan
PURPOSE OF THIS GUIDE
This document guides departments in developing a risk-based Departmental
Evaluation Plan using a five step process and provides templates to assist with the
formatting of the Plan. The five steps are:
1. Compile List
of Departmental
Initiatives
2. Pre-screen
for Exclusion
/ Inclusion
3. Assess
Risk
4. Assess
Readiness for
Evaluation
5. Submit Plan Evaluation
Product Profiles
The Policy on Evaluation requires that all new and existing government initiatives be
assessed for evaluation priority and that those priorities are documented in a
Departmental Evaluation Plan.
New initiatives are assessed for evaluation priority
through the completion of an Accountability Framework1 as part of the Cabinet
submission process. Existing initiatives are assessed for evaluation priority using a
risk-based approach as defined in this document. A web Risk-Based Assessment tool
developed by the Economic and Statistics Branch (ESB) of the Department of Finance
to assist departments in ranking their initiatives from highest to lowest risk. A Guide to
using this tool is contained in Annex F.
Words or terms defined in the Glossary are bolded in their initial appearance in this
document.
1
Developing an Accountability Framework: a Resource and Reference Guide is available through the
Provincial Government Programs Office: 729-2590.
Cabinet Secretariat │ December 2011
2
Guide to Developing a Risk-Based Departmental Evaluation Plan
INTRODUCTION
The Policy on Evaluation requires Deputy Ministers/equivalents2 to develop a
Departmental Evaluation Plan for approval by the Minister and submission to the Clerk
of the Executive Council. On behalf of the Clerk, the Provincial Government Programs
Office will review departmental evaluation plans and provide comments to the Clerk on
the appropriateness of evaluation coverage including the risk-based approach used to
identify evaluation priorities. The Clerk may refer some or all plans to Treasury Board
or Cabinet for approval.
POLICY PROCESS



Plans are to be strategically focused and
founded on an assessment of risk that
Deputy Ministers are accountable for
developing and implementing the Plan.
Ministers approve the Plan.
Clerk may send the Plan to Treasury
Board or Cabinet for approval.
the department’s initiatives (programs
and services and major policies) may
no longer be relevant, effective or
efficient.
A department has the choice of developing an annual evaluation plan or a multi-year
plan. Multi-year plans are subject to annual review by the department and all plans are
to be monitored and updated as appropriate.
2
For purposes of this Guide, the term department refers to any organization identified in Annex A to the
Policy on Evaluation and the term Deputy Minister refers to any head of these organizations.
Cabinet Secretariat │ December 2011
3
Guide to Developing a Risk-Based Departmental Evaluation Plan
OVERVIEW
Why do Risk-Based Planning?
The ultimate goal of evaluation is improving client outcomes and ensuring government
initiatives provide value for money (i.e. demonstrating effectiveness, efficiency, and
economy). However not all initiatives need to be evaluated or can be evaluated in a
cost effective manner.
The purpose of a risk-based evaluation plan is to ensure
evaluation-related activities are strategically integrated into the department's existing
planning and performance measurement processes and reflect the broader need to
support Cabinet decision-making and budget deliberations.
Such planning ensures
resources are allocated to evaluation products (includes evaluations as well as
evaluation-related activities) that can provide the greatest value-added to a department
and Government generally.
As the Deputy Minister is accountable for the development and implementation of the
Plan, it should inform the Minister and Deputy Minister of the scope of the department's
evaluation activities and the limitations of that scope.
There should be sufficient
information presented in the Plan to help the Deputy Minister determine if the
evaluation/evaluation-related activities proposed are sufficient and support the priorities
of the Department and of Government.
The best evaluation plan is one that directly supports the information or decision-making
needs of the Minister, Deputy Minister, Cabinet and Treasury Board. It is more than a
list of evaluation-related activities. A good evaluation plan is a tool to:
 Engage the Minister and senior executives in identifying the priority evaluation
products required to provide them with information necessary to make informed
decisions;
 Communicate the importance of reliable performance data to support monitoring
and evaluation of departmental and government initiatives; and
 Identify and secure the resources for completion of the evaluation activities.
Cabinet Secretariat │ December 2011
4
Guide to Developing a Risk-Based Departmental Evaluation Plan
Transitional Considerations
The Policy on Evaluation recognizes that departments are not on a level playing field
when it comes to the complexity of their programs and their current evaluation capacity.
The Policy therefore accommodates the capacity discrepancy by allowing for a
transition period to full implementation by March 31, 2014. The composition of a plan is
flexible: plans may be annual or multi-year; there is no expectation that all initiatives be
evaluated, nor is there an established minimum number of initiatives to be evaluated
annually. Deputy Ministers must identify their evaluation priorities and identify what
evaluation products will be completed or commenced during the fiscal year, beginning
with 2012-13. Evaluation products may include evaluation-related training, terms of
reference (TOR) or Requests for Proposals (RFP) for a future program evaluation,
accountability frameworks, research and data analysis, evaluation reports and reviews
as part of a department’s Regulatory Improvement Plan. The identification of evaluation
priorities and the associated resources are the critical elements to be reflected in the
Departmental Evaluation Plan. Some departments may not be in a position to conduct
any evaluations during 2012-13 and may instead engage in capacity building activities
or develop performance measurement plans for priority programs.
As part of building capacity, departments are encouraged to pilot at least one internal
evaluation in 2012-13.
When to Start the Planning Process
To avoid duplication of effort, the planning process can be integrated with the
departmental strategic planning, budgeting and reporting processes.
In most cases,
the evaluation planning process should start in the Fall and be completed by early
Spring for the subsequent year's evaluation-related activities (See Table 1).
Initial
Departmental Evaluation Plans are to be forwarded to the Clerk no later than April 30,
2012. Departments should identify which evaluation-related activities can be completed
using existing resources and what assistance is needed to complete other product(s).
Ministerial approval is required prior to seeking incremental resources through the
Cabinet Secretariat │ December 2011
5
Guide to Developing a Risk-Based Departmental Evaluation Plan
budget process. If such approval is extended, a draft Departmental Evaluation Plan
must be included with the annual budget submission.3
Department’s choosing multi-year evaluation planning shall submit their Plan to the
Clerk at the end of April during the year in which the existing plan expires. All plans
shall be reviewed annually. Where substantive changes to a Plan are required, the
Clerk shall be notified as part of the Deputy Minister’s performance contract process.
3
Refer to the Department of Finance’s annual budget guidelines for process considerations.
Cabinet Secretariat │ December 2011
6
Guide to Developing a Risk-Based Departmental Evaluation Plan
Table 1 Central Agency Reporting Schedule
Documentation
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Transparency and
Accountability
Plans (every three
years)
Transparency and
Accountability
Annual Report
* Budget
Submissions
DM Contract
(submission and
reporting)
** HR
Accountability
Report
Evaluation Plan
(Department)
* If requesting additional resources to implement the proposed Departmental Evaluation
Plan, a draft of the plan must be attached to the request and submitted with the budget
briefing note.
Refer to the Department of Finance’s annual budget guidelines for
process considerations.
** Process and timing under review by the Public Service Secretariat
Cabinet Secretariat │ December 2011
7
Guide to Developing a Risk-Based Departmental Evaluation Plan
GETTING STARTED
As stated earlier, the best evaluation plan is one that directly supports the information or
decision-making needs of the
Minister,
Deputy
Minister,
Potential Information Sources
Budget Decisions
Auditor General Reports
Cabinet and Treasury Board.
For this reason it is advisable to
Budget Monitoring Reports
meet
Cabinet or Treasury Board
with
the
Minister
and
Deputy Minister to identify their
information needs, priorities, and
departmental challenges for the
upcoming year.
Client Surveys or Public
Consultations
Directions
Deputy Minister Performance
Contract
Departmental Plan (Strategic,
Business, Activity) and Annual
Report
Department Cabinet
Submission Plan
Fiscal Framework to identify
Performance Monitoring and
To complete an evaluation plan
sun setting funds
Evaluation Reports
there are multiple documents
Ministerial Mandate Letters
that can be helpful in attaining
information. Not all documents
Public Strategy Documents
Operational Plan
Prior Year’s Departmental
Evaluation Plan
Strategic Directions
are necessary as the required
Regulatory Improvement Plan
Speech From the Throne and
information may be accessible in
Work Plan
Budget Speeches
more than one document.
A
short description of documents and how they can be accessed is provided in the
glossary.
Cabinet Secretariat │ December 2011
8
Guide to Developing a Risk-Based Departmental Evaluation Plan
What Goes In a Plan?
Policy Requirements
In accordance with the Policy on Evaluation, the plan should include:
 Specific evaluations or evaluation–related activities (e.g. developing Accountability
Frameworks) as directed by Cabinet or




WHAT GOES IN A PLAN?
requested by the Clerk of the Executive
Evaluations directed by the
Clerk, Cabinet, Federal
Provincial Territorial or other
Funding Agreements
Evaluations required to
support horizontal initiatives
Evaluations required to inform
departmental decision making
Programs with sun setting
funds
Council following consultation with the
affected Deputy Minister (e.g. evaluations
outlined
in
the
government-wide
evaluation plan4).
 Evaluations
governmental
required
funding
by
inter-
agreements
or
legislative requirements;
 Specific evaluation activities required to
support the department’s responsibilities
in moving identified horizontal initiatives forward;
 Programs for which program funding is set to terminate unless it has already been
determined that an evaluation is not necessary; and
 Other direct spending of the department as appropriate to the information needs of
the Minister, Deputy Minister and Senior Management Team.
4
The Clerk will submit to the Premier by March 31, 2012, the initial government-wide evaluation plan.
Cabinet Secretariat │ December 2011
9
Guide to Developing a Risk-Based Departmental Evaluation Plan
Five Step Process
This Guide describes a five step process for developing a Department’s Evaluation
Plan.
1. Compile List of Departmental Initiatives
2. Pre-screening for Exclusion / Inclusion
3. Assess Risk
4. Assess Readiness
5 Submit Plan – Evaluation Product Profiles
Steps 2-4 have associated templates (Appendix A, Risk Assessment Report and
Appendix B) on which assessment decisions are recorded. Each of these completed
templates should be attached to the final Departmental Evaluation Plan.
1. Compile List of Departmental Initiatives
Developing a plan begins with identifying all departmental initiatives. The PGP Office
requests that departments review with their executive, at the beginning of a
department’s evaluation planning cycle (either annual or multiyear), their inventory of
programs and services to ensure initiatives are accurately reflected, that is
organizational initiatives are listed not organizational structures or functions. Inventories
requiring updating should be submitted to the PGP office using the template in
Appendix E5.
Each initiative listed in the inventory should include the associated
budget/cost. Some initiatives do not have a specified budget; in such cases, the cost of
resources to deliver that initiative should be estimated.
Initiatives are defined as a group of departmental activities designed to achieve specific
outcomes. They may be programs or services designed to benefit or influence external
clients, services to internal clients or activities required to monitor policy or program
delivery by third parties (e.g. HCS is responsible for provincial policy with RHAs
delivering the programs).
Where a department’s initiative is associated with a
5
Initiatives identified by departments in 2011 have been pre-populated into the risk assessment database
by the Economic and Statistics Branch of the Department of Finance. Please contact your assigned
Program Officer, Provincial Government Programs Office for information on how to access the web-based
tool.
Cabinet Secretariat │ December 2011
10
Guide to Developing a Risk-Based Departmental Evaluation Plan
Horizontal Initiative, the initiative should be itemized separately as per Template 6 of
Appendix E.
Corporate Shared Services such as printing services, payroll, purchasing, pensions and
group insurance administration, corporate financial services, various OCIO support
services, etc. should be included in the inventory of the department providing the
service. Departmental support functions/divisions such as Executive Support, Finance
Units, Strategic Human Resource Management Units, Policy and Planning Divisions,
and Communications Divisions should not be included.
2. Pre-Screen for Exclusion / Inclusion
From your inventory there may be initiatives that can be immediately excluded or
immediately prioritized for evaluation (included). The rationale for including or excluding
an initiative at the prescreening stage should be recorded in the standard template
found in Appendix A which will be attached to your Departmental Evaluation Plan
submission.
Exclusion
Evaluations can be time and resource intensive. While departments should have a
performance measurement system in place to regularly monitor and report on program
performance, evaluations should only be undertaken on a periodic basis and only if the
benefits of the evaluation are expected to exceed the costs of completing an evaluation.
Accordingly, initiatives that have been evaluated in the last two years can be excluded
from the next year’s departmental evaluation planning.
Other initiatives may become excluded after completing the Evaluablility Assessment
in Section 5.2.4 of this document.
Cabinet Secretariat │ December 2011
11
Guide to Developing a Risk-Based Departmental Evaluation Plan
Inclusion
For some initiatives, there may already be a mandatory requirement for an evaluation.
The source of the requirement may have been through Cabinet or Treasury Board
direction, initiatives identified under Accountability Frameworks (for new initiatives or as
part of an Horizontal Initiative), initiatives requiring funding renewal (as specified in
budget decisions or MCs), legislative requirements6, requests by Ministers or Deputy
Ministers, or where other funders are involved (e.g. the federal government) as a
requirement of the Contribution Agreement. Those initiatives with mandatory evaluation
requirements do not require a Risk-Based Assessment but do require an Evaluability
Assessment under Section 5.2.4 of this document.
Reviews or evaluations identified in a department’s Regulatory Improvement Plan
should be documented in the Departmental Evaluation Plan but do not require
assessment for risk or evaluability.
3. Assess Risk
All department initiatives remaining after the pre-screening process must be assessed
for risk. The probability that a program is no longer relevant, effective or efficient is the
overall risk that is being measured. The Economic and Statistics Branch (ESB) of the
Department of Finance has developed a web Risk-Based Assessment tool to assist
departments in ranking their initiatives from highest to lowest risk.7 The tool’s ranking is
based on the following seven risk variables with predefined weighted sub criteria. The
attached Risk Assessment Tool Guide will provide step by step instructions for
completing the on line tool.
Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a
significant percentage of a department’s resources may be evaluated periodically to
6
Legislative requirements refer to any legislation requiring periodic reviews of initiatives. It does not refer
to reviewing the legislation itself.
7 Please contact your Program Officer to determine how to access this web-based resource.
Cabinet Secretariat │ December 2011
12
Guide to Developing a Risk-Based Departmental Evaluation Plan
confirm value for money (effectiveness, efficiency and economy). 8 Helpful documents
may include: Budget Decisions, Estimates.
Gross Expenditure Variance: This risk factor considers the significance of over or
under spending trends.
Initiatives experiencing significant over or under spending
should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing
resources adequate to address the identified population’s need; is the initiative overresourced based on existing needs).
Helpful documents may include: Budget
Monitoring Reports, Report on the Program Expenditures and Revenues of the
Consolidated Revenue Fund.
Priorities: This risk factor considers how important the initiative is to achieving a
government-wide priority. Helpful documents may include: Prior year’s Departmental
Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy
Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic,
Business, or Activity), Annual Report, Operational Plan, and Work Plan.
Funding Renewal Commitments: This risk factor applies where evaluation may
support a request to continue funding that was approved for a fixed period or for which
there is no continuing provision in the fiscal framework.
Helpful documents may
include: Fiscal Framework, Budget Decisions.
Public Opinion: This risk factor considers client and other stakeholder feedback about
an initiative. There may be current sensitivities, interests or pressures from clients and
other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor
General, individuals, groups or MHAs). Helpful documents may include client surveys,
public consultations, Auditor General Reports, public strategy documents, departmental
correspondence such as letters to Ministers, etc.
8
ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to
assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant.
Departments may choose to include initiatives which use significant resources in their Plan even if
assigned a low risk assessment using this tool.
Cabinet Secretariat │ December 2011
13
Guide to Developing a Risk-Based Departmental Evaluation Plan
Relevance: This risk factor considers the results of a department’s environmental scan
and SWOT (strengths, weaknesses, opportunities, threats) analysis.
Departmental
strategic planning and/or horizontal strategy development may have identified changes
in the demographics and/or needs of the target population that requires a revised needs
assessment, process review and/or effectiveness evaluation (e.g. growing wait lists for
services, lower than expected take-up).
Helpful documents may include: Auditor
General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan
(Strategic, Business, or Activity) and Annual Report.
Performance Findings: This risk factor considers the findings from the performance
monitoring and or past evaluation reports.
Helpful documents may include:
Performance Monitoring and Evaluation Reports, Auditor General Reports.
Upon completing the web Risk-Based Assessment Tool for all initiatives, a final report
can be automatically generated ranking initiatives from 1 (highest risk) to 0 (lowest risk).
The final Risk Assessment Report is to be reviewed and approved by executive and
attached to the Departmental Evaluation Plan submission.
There is no identified “high risk” cut-off point. Initiatives will cluster differently along the
continuum across departments. For example some departments’ initiates may cluster
primarily in the orange to red zone while others may cluster closer towards the blue
spectrum. In either case departmental executive must choose a Risk Level Cut-Off
point (between 1-0). All departments will be expected to identify Evaluation Products
within the planning cycle. Those initiatives close to or above the cut off point must be
assessed for evaluability (i.e. which priority initiatives are able to have an evaluation
product addressed in this planning cycle).
Cabinet Secretariat │ December 2011
14
Guide to Developing a Risk-Based Departmental Evaluation Plan
4. Assess Readiness
The risk assessment process identifies which initiatives are potential priorities for
evaluation: the evaluability process identifies which priorities are feasible to evaluate in
the current planning cycle.
Initiatives to be assessed for evaluability include those
under Section 5.2.2 assessed as mandatory and those initiatives that have been
identified as priority due to their proximity to the Department’s Risk Cut-Off point
(Section 5.2.3).
All Evaluability Assessment outcomes should be recorded in the
standard template found in Appendix B. Each Evaluation or Evaluation Activity to be
completed in this planning cycle must have a corresponding Evaluation Product Profile
(Appendix C) completed to describe what is expected to be completed by when and at
what cost. The evaluability process is defined by three key questions as illustrated in
Chart 1.
Chart 1 Process Flow Chart: Priority Evaluation Initiatives
Cabinet Secretariat │ December 2011
15
Guide to Developing a Risk-Based Departmental Evaluation Plan
Question 1: Will the initiative likely be modified based on evaluation results?
Department leads should consult with their Minister and Deputy Minister to determine if
there is an ability to modify the initiative based on findings. If there are timing issues
which make evaluation in the current planning cycle not feasible, the initiative should be
kept in view and reassessed in future planning cycles.
Questions 2: Is the initiative ready to be evaluated?
Preconditions for evaluating the effectiveness and efficiency of initiatives include:
 a clearly articulated initiative
 clearly specified goals and outcomes
 a rationale linking the initiative’s activities to the desired outcomes and
 appropriate performance measurement data
If an initiative cannot be evaluated, the departmental evaluation plan should identify
what Evaluation Activity is required to position the initiative for evaluation at a future
date (e.g. developing a logic model, implementing a performance measurement plan,
gathering baseline data, and so forth). These Evaluation Activities must be detailed in
an Evaluation Product Profile, Appendix C.
Question 3: Are resources available to complete the evaluation?
Evaluation
activity
requirements.
planning
will
require
assessing
internal/external
resource
Where existing departmental resources are inadequate to cover all
priority evaluations, those that can be deferred should be kept in view for consideration
in subsequent evaluation cycles. Those that cannot be deferred and would require
incremental funding should be brought to the Deputy Minister’s attention. The Minister
and Deputy Minister must determine if existing resources can be reallocated or consider
the budget guidelines to determine if they will seek additional resources through the
budget process9.
If incremental resources are being requested, a draft of the
Departmental Evaluation Plan should be attached to the budget submission.
9
Departments need to consider the Department of Finance budget guidelines when determining if
additional resources should be sought through the budget process.
Cabinet Secretariat │ December 2011
16
Guide to Developing a Risk-Based Departmental Evaluation Plan
The Economic and Statistics Branch (ESB) is a resource available to all departments.
The ESB may be able to assist in gathering baseline data, developing data from
administrative or other sources, ensuring datasets are suitable for an evaluation and/or
developing or assessing potential measurement tools for an evaluation.
They can
assist in making sure the data and tools needed for an evaluation are available when
needed by considering requirements early in the process. If there will be incremental
cost to the ESB to provide the support, there may be a charge to the department which
should be identified in the budget submission along with any other incremental costs
associated with the evaluation.
5. Submit Plan – Evaluation Product Profiles
Once the Minister and Deputy Minister have been briefed and their input obtained, a
final Departmental Evaluation Plan should be prepared.
A standard template is
presented in Appendix D. For each proposed evaluation product, an overview should
be included reflecting: critical timelines (at a minimum expected start and end dates);
resource requirements including whether these would be internal or external; and `the
department lead overseeing the activity. A standard template to be completed for each
evaluation product profile is presented in Appendix C.
The final plan should be forwarded to the Minister for approval.
As per the Policy on
Evaluation, the approved plans must be forwarded to the Clerk of Executive Council by
April 30, 2012 for the initial plan, and in subsequent years as plans are scheduled for
renewal. The Clerk may submit the Departmental Evaluation Plan to Treasury Board or
Cabinet for further consideration.
The Clerk will be advised of the department’s progress towards implementation of the
plan as part of the Deputy Ministers’ performance reporting process.
Cabinet Secretariat │ December 2011
17
Guide to Developing a Risk-Based Departmental Evaluation Plan
GLOSSARY
Accountability
Framework:
The
Policy
on
Evaluation
requires
Deputy
Ministers/equivalents to ensure all Cabinet submissions for new policies and programs
identify the client, other benefits, and where feasible, establish specific, measurable,
achievable, realistic, and time bound (SMART) targets before decisions on
introducing/funding are made by Cabinet. It also requires them to “ensure Cabinet
submissions include an accountability framework documenting how performance will be
monitored and determining if and when a more comprehensive evaluation of the
program/policy will occur”. An Accountability Framework makes roles, responsibilities,
and expectations clear, supporting the availability of reliable and timely reports about
intended and actual results.
The Accountability Framework structure includes
performance-based planning, monitoring, performance-based reporting and feedback.
Accountability Frameworks are attached to applicable Cabinet papers submitted since
July 1, 2011.
Consolidated Revenue Fund: is the account into which public money (taxes and
revenues) is deposited, and from which funds are withdrawn in order to pay the costs of
public services. The Legislature authorizes the use of the money in the Consolidated
Revenue Fund.
Departmental Evaluation Plan: is a clear and concise summary that establishes the
department's evaluation activities / products to be undertaken over a specified period, in
accordance with the Policy on Evaluation.
Economy: minimizing the use of resources. Economy is achieved when the cost of
resources used approximates the minimum amount of resources needed to achieve
expected outcomes (i.e. attaining the right program inputs at the lowest possible costs).
Effective: the extent to which the observed outcomes are consistent with the intended
outcomes.
Cabinet Secretariat │ December 2011
18
Guide to Developing a Risk-Based Departmental Evaluation Plan
Efficient: the extent to which resources are used such that a greater level of output is
produced with the same level of input or a lower level of input is used to produce the
same level of output (i.e. attaining the most program outputs possible for each program
input). The level of input and output could be increases or decreases in quantity, quality
or both.
Evaluability Assessment: is used to determine if an initiative is able to be evaluated in
the current planning cycle. An initiative is evaluable if: it is likely be modified based on
evaluation results; the initiative can be clearly and logically articulated, it has
performance data or can be developed using evaluation methods and there are
resources available to complete the evaluation.
Evaluation: the systematic collection and analysis of information on the performance of
a policy, program or initiative to make judgments about relevance, progress or success
and cost-effectiveness and / or to inform future programming decisions about design
and implementation. Types of program evaluations include: Evaluability Assessment,
Needs Assessment, Monitoring / Review compliance with performance standards,
Implementation
/
Process
Evaluation
(Formative
Evaluation),
Impact/Outcome
Evaluation (Summative Evaluation), Program Review and Efficiency Assessment
(Cost/Benefit Analysis or Cost Effectiveness Analysis).
Evaluation Related Activity: any output of the departmental evaluation function which
may include, but is not limited to, the following: training, terms of reference (TOR) or
Requests for Proposals (RFP) for a future program evaluation, accountability
frameworks, research and data analysis, and reviews monitored under a department’s
Regulatory Improvement Plan.
Evaluation Product: Evaluations and evaluation-related activities.
Horizontal Initiative:
An initiative is considered horizontal if multiple provincial
ministers are accountable for the results of the initiative at this point in time.
An
initiative’s accountability structure may change over the course of its life cycle.
Cabinet Secretariat │ December 2011
19
Guide to Developing a Risk-Based Departmental Evaluation Plan
Committees of multi-departmental initiatives should conduct periodic reviews over the
initiative’s life cycle to determine the appropriate level of accountability.
Performance Measure: a quantitative or qualitative standard that characterizes and
defines initiative’s results including outputs and outcomes (e.g. the “success standard”
by which you know there has been a change in knowledge, skill, behavior, or attitude).
Performance Measurement: the process of selection, development and ongoing use
of performance measures for program management or decision-making.
Generally
performance measurement occurs in the form of quarterly or annual monitoring of
performance measures to determine if the desired change is occurring. Performance
measurement can identify the direction of change but does not attribute the cause of the
change to the initiative.
Performance: the actual results measured against defined standards (i.e. the extent to
which effectiveness, efficiency and economy are achieved by a program).
Policy:
A policy enables or restricts actions as a means of specifying outputs,
outcomes, or parameters. Policy consists of official guidelines or operating principles
that influence behaviour towards a stated outcome.
Program: a group of related purposeful activities that is intended to achieve one or
several related objectives. A program is often treated as a budget unit, that is designed
and managed to meet a specific public need either directly or indirectly (that is, supports
the operations of government).
Relevant: the extent to which a program responds to a demonstrable need of
Newfoundlanders and Labradorians and is appropriate for delivery by the provincial
government.
Risk-Based Approach: is a method for considering risk when planning the extent of
evaluation coverage of direct program spending / policy initiatives.
The risk being
measured is that an initiative may not be, or is no longer relevant, effective or efficient,
Cabinet Secretariat │ December 2011
20
Guide to Developing a Risk-Based Departmental Evaluation Plan
Value for Money: the extent to which a program demonstrates relevance and
performance (effectiveness, efficiency and economy). It is an important goal for public
officials and other stakeholders who are concerned with whether taxpayers and citizens
are receiving effective and efficient programs and services for their tax dollars.
DOCUMENTS GLOSSARY
Auditor General’s Reports:
As a legislative auditor, the Auditor General audits
financial statements and other accountability documents, evaluates management
practices and control systems, and determines compliance with legislative and other
authorities. The Auditor General reports at least annually to the House of Assembly on
significant matters which result from his/her examinations.
Budget Speeches: An itemized summary of estimated or intended expenditures for a
given period along with proposals for financing them. Annual budget speeches are
accessible on the House of the Assembly publications website. The Budget combines
the projected accrual revenues and expenses of the Consolidated Revenue Fund with
the budge of various Crown Corporations, Boards and Authorities which are controlled
by the Government of NL.
Budget Decisions: Subsequent to Cabinet decisions on department’s budget requests,
the Budget Division of the Department of Finance will issue documents recording the
decisions. From time to time, there may be a requirement for a department to submit
additional information or report back to Treasury Board or Cabinet on specific issues.
These documents are filed with Deputy Ministers and with Department Comptrollers.
Budget Monitoring Reports: On a monthly basis, Departmental Controllers submit to
the Budget Division of Finance, a report highlighting any variances between projected
expenditures to March 31st of each year as compared with the original budget.
Departmental Controllers will have copies on file.
Cabinet or Treasury Board Directions: This category includes minutes and other
correspondence issued by Cabinet or Treasury Board to a department directing a
Cabinet Secretariat │ December 2011
21
Guide to Developing a Risk-Based Departmental Evaluation Plan
specific action. These documents are generally in the form of a Minute of Council (MC);
Order in Council (OC) or a Treasury Board Minute (TBM). These documents are filed in
the minister’s office.
Client Surveys or Public Consultations: Many initiatives consult with citizens in the
development stage or report progress at incremental stages.
Feedback from
consultations is generally summarized and used to improve on program performance.
This feedback will be valuable in answering the risk-based assessment public opinion
question and may be relevant in answering question 1 of the evaluability assessment.
Oftentimes public feedback is included in public reports (e.g. “What We Heard”.
Program Managers will be able to identify where this information can be found).
Departmental Plan (Strategic, Business, Activity) and Annual Reports:
The
Transparency and Accountability Act requires all government entities to submit to
Cabinet a planning document. Depending on the entity’s categorization (category 1, 2,
or 3) these planning documents are referred to as Strategic, Business or Activity Plans
respectively. Departments are required to report annually to Cabinet on their plan’s
progress (i.e. Annual Reports. All plans and annual reports are public documents and
are posted on the department’s website).
Department Cabinet Submission Plan: To facilitate Cabinet scheduling, departments
are required to identify what issues they intend to bring before Cabinet, with a brief
description and an anticipated submission date. As the Policy on Evaluation requires all
Cabinet submissions requesting major new policies or new initiatives to have an
accountability framework, the Cabinet submission is a priority document when
developing an evaluation plan. The plans are highly confidential and are filed in the
department’s ministerial office. Cabinet Submission Plan information may be accessed
through department Policy Directors.
Deputy Minister’s Performance Contract: The Deputy Minister reports to the Minister
and the Clerk of Executive Council in ensuring that all over-arching policies of
government are communicated and coordinated, and that matters and sensitive issues
Cabinet Secretariat │ December 2011
22
Guide to Developing a Risk-Based Departmental Evaluation Plan
requiring strategic direction on the part of government are monitored to ensure their
expeditious resolution.
Each Deputy Minister has a performance contract which is
reviewed periodically with the Clerk in consultation with the Minister. These Contracts
are highly confidential and are filed in the Deputy Minister’s Office and with the Clerk of
Executive Council. Departmental policy divisions draft these contracts and can ensure
all pertinent information needs are met by the proposed plan.
Estimates of the Program Expenditure and Revenue Fund: is a supplementary
document to the Budget. It outlines the estimated expenditures and revenues of the
Consolidated Revenue Fund for the year and reflects the policies, programs and
priorities of government. This document is available on the Internet:
www.fin.gov.nl.ca/fin/public_accounts/index.html.
Fiscal Framework: While Estimates are prepared for a single fiscal year, the Budget
Division of the Department of Finance maintains a multi-year forecast of departmental
expenditures and revenues by budget categories (e.g. salary and operating envelopes,
major programs).
A quick review of this Fiscal Framework can identify budget
adjustments such as the removal of funds for one-time initiatives or those being piloted.
Ministerial Mandate Letters:
A letter oftentimes issued by the Premier to his/her
Ministers outlining the government priorities that fall within a ministry’s mandate. Such
letters would be filed in the Minister’s Office.
Operational Plan: The operational plan focuses on the key priorities of the Deputy
Minister which relate to the internal administration of the organization (i.e. monitoring
and evaluating organizational performance, organizational structure, development and
implementation of organizational policies and procedures). Operational plans are
encouraged by the Transparency and Accountability Office and, as such, department
Accountability Coordinators will be of assistance in accessing operational plans.
Performance Monitoring and Evaluation Reports: Many programs have undergone
monitoring and/or evaluation.
Understanding the monitoring and evaluation history,
Cabinet Secretariat │ December 2011
23
Guide to Developing a Risk-Based Departmental Evaluation Plan
including recommendations for program changes, will facilitate risk-based assessments.
For example, programs that have been evaluated in the last two years are excluded for
consideration under the current year’s risk-based assessment.
However, these
monitoring or evaluation reports may make recommendations regarding future
monitoring and evaluation needs. These reports come in many forms and may be
generated internally or externally.
Program Managers and Directors may be of
assistance in tracking such reports.
Public Strategy Documents: Strategic documents outline a plan of action designed to
achieve a particular goal.
Such documents are used by departments to clarify an
approach to an important broad-based issue (e.g. wellness, climate change, innovation,
and so forth).
These reports are helpful in identifying the various initiatives that are
covered under the strategy. Most departments post their strategy documents on their
website. Policy Directors can facilitate access to such documents if they are not readily
available.
Prior Year Departmental Evaluation Plans: Evaluation plans illustrate the decision
making process used to define priority initiatives for evaluation. An understanding of
these processes will facilitate identifying current priority initiatives. Evaluation Plans are
not public documents. They can be accessed through Policy Directors.
Regulatory Improvement Plan: All Departments are required under the Regulatory
Reform Initiative to submit an annual or multi-year plan to Service Newfoundland and
Labrador.
Quarterly progress reports by departments demonstrate qualitative and
quantitative improvements in regard to developing and implementing regulation.
Review-related activities monitored under this plan are considered to be an “evaluation
product” and should be credited as such in the Departmental Evaluation Plan.
Regulatory Improvement Plan initiatives do not require Risk or Evaluability
Assessments under the Policy on Evaluation.
All departments have a Regulatory
Reform Representative who can advise on, as applicable, the initiatives included within
the department’s Regulatory Improvement Plan.
Cabinet Secretariat │ December 2011
24
Guide to Developing a Risk-Based Departmental Evaluation Plan
Speech from the Throne: The Speech from the Throne occurs at the opening of a
session of the House of Assembly. The Speech is read by the Lieutenant Governor-inCouncil and prepared by the Premier's Office. Speeches are publicly accessible on the
House of the Assembly publications website.
Strategic Directions: Strategic direction means the articulation of a desired physical,
social or economic outcome that would normally require action or involvement of more
than one government entity. They are a summary of government’s commitments to the
people of the province. They are communicated by Government through Platform
Documents, the Speech from the Throne and Budget Speeches, press releases, and
policy documents. Each minister has his/her own strategic directions. Accountability
Coordinators play an essential role in the development of the Strategic Directions for
their respective ministers. The strategic directions of a department are contained in the
Department’s Plan (strategic, business, activity).
Work Plan: Work plan issues may emanate from the department plan, the operational
plan, or reflect priorities at a branch, division or position level. The work plan contains
the work direction of a specific branch, division or position. Divisional Directors will be
of assistance in accessing work plans.
Cabinet Secretariat │ December 2011
25
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX A: Pre-Screening Assessment Template
From your inventory there may be initiatives that can be immediately excluded or
immediately prioritized for evaluation (included). The rationale for including or excluding
an initiative at the prescreening stage should be recorded in template 1. This template
is to be attached to your Departmental Evaluation Plan submission.
Template 1 Pre-Screen Inventory
PRE-SCREENING OUTCOME
Initiative
Initiative W
Initiative X
Initiative Y
Initiative Z
Mandatory Inclusion
Commitment
MC2009-xxxx
Ministerial Priority
Exclusion
Evaluated in the last two years
 2010
Poverty Reduction Strategy
Evaluation
Cabinet Secretariat │ December 2011
26
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX B: Evaluability Assessment Template
The Template 2 below records evaluability decisions for all priority initiatives. Priority
initiatives include those identified through the Pre-Screening: Mandatory Inclusion
(Section 5.2.2) and those initiatives that have been identified as priority relative to the
Department’s Risk Cut-Off point.
Using Template 2, list the initiatives that have been assessed for “Evaluability” and
check the box identifying the assessment outcome. Where “Other” has been checked,
please add a footnote to provide an explanation for the choice.
Template 2 Evaluability Assessment Template
EVALUABILITY ASSESSMENT OUTCOMES
Initiative
W
X
Y
Z
*Evaluation
*Evaluation Activity To Be Completed
To Be
Completed In
This Planning
Cycle
In This Planning Cycle:
Not Ready
For
Evaluation
Insufficient
Capacity (e.g.
need staff
training, data
bases
developed etc)
Other
No Evaluation Or Evaluation Activity
To Be Completed In This Planning
Cycle:
Initiative
Insufficient Other
Unlikely To
Resources
Be Modified
Based On
Evaluation
Results




* An Evaluation Product Profile (Appendix C) is to be completed for each initiative that
indicates a check under the categories of Evaluation or Evaluation Activity.
Cabinet Secretariat │ December 2011
27
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX C: Evaluation Product Profile
The Evaluablity Assessment Outcomes (template 2) identify all Evaluations and
Evaluation Activities that are to be completed within this evaluation planning cycle. For
each planned Evaluation and Evaluation Activity, an Evaluation Product Profile
(template 3) is to be completed to describe what is to be completed, by when and at
what cost.
Template 3 Evaluation Product Profile
INITIATIVE TITLE:
Brief description of the initiative and the scope of the evaluation product to be
undertaken.
Proposed Timeframe
begin date:
end date:
Who is responsible to oversee
this work?
Resource Requirements
$ Within Base
*Budget Request
$ Reallocated
$ New
Internal
External
Economic Statistics Branch
Total
* Please refer to the Department of Finance’s annual Budget Guidelines to determine
the parameters under which new funding can be requested.
Cabinet Secretariat │ December 2011
28
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX D: Departmental Evaluation Plan
Template 4 Departmental Evaluation Plan
1. INTRODUCTION
1.1 Department’s Strategic Directions and Goals:
Provide a brief discussion of how evaluation is being used to support the department’s
mandate.
2. ASSESSMENT DOCUMENTS
2.1 Pre-Screening Outcomes
Attach the completed Appendix A Template 1
2.2 Risk Matrix Outcomes
Attach the web-based Risk Assessment Report
2.3 Evaluability Assessment Outcomes
Attach the completed Appendix B Template 2
3. YEAR IN REVIEW
3.1 Evaluation Products Conducted in Year XXXX-XXXX :
This section should identify evaluations and evaluation products that your department
has completed in the previous evaluation planning cycle. The Provincial Government
Programs Office has developed an evaluation inventory for each department for the
period 2009-2011. Please contact your PGP Officer for a copy.
3.2 Evaluation Activities to Be Carried Forward:
This section should identify evaluations and evaluation products that your department
had anticipated completing in the previous planning cycle but are required to carry
over into the next planning cycle.
4. CURRENT YEAR’S WORK PLAN
4.1 Proposed Evaluation Product Profiles:
Complete Template 3 in Appendix C for each initiative for which there is an
evaluation product to be addressed in the current year’s evaluation cycle.
4.2 Proposed Timeline for Evaluation Activities for Current Year:
Summarize the evaluation products to be delivered in this evaluation cycle through a
time line.
4.3 Resource Requirements:
Summarize the resource requirements identified in the individual product profiles
4.4 Regulatory Improvement Plans (RIP):
Evaluations/Product commitments associated with RIP should be noted in this section
and cross referenced to the RIP.
Cabinet Secretariat │ December 2011
29
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX E: Initiative Inventory Update
Departmental initiative inventories and their budgets should be reviewed, with
executive, at the beginning of each evaluation planning cycle to ensure initiatives are
updated and accurately reflected (i.e. new initiatives, initiatives removed, and ensure
organizational initiatives are listed not organizational structures or functions).
Templates 5 and 6 should be completed and submitted to the PGP Officer for
population into the web-database. The templates below provide examples.
Template 5 Departmental Initiative Inventory Update
Department
Health and
Community Services
Finance
* Estimated 2011
Budget
Initiative: Component
% of Gross
Dept Budget
NL Provincial Drug Program:
Access Program
Tax Expenditures: PIT Child Benefit
Where a department’s initiative is associated with a Horizontal Initiative, this association
should be identified in the inventory as illustrated in template 6.
Template 6 Horizontal Initiative Inventory Update
Horizontal
Initiative (HI)
HI Lead
Department
Provincial Waste
Management
Environment and
Conservation
Sub-Initiative (SI)
Policy
Implementation
Enforcement
SI Department
EC
MA
SNL
* Estimated
Budget for
Year 20XX
$ 40,000
$50M
$100,000
* If the initiative does not have its own Cost Centre in ORACLE GL, please provide an
order of magnitude estimate of the budget resources it consumes (can be rounded to
nearest $100,000 or 1/2 million)
Estimated 20xx Budget / Total Gross Current Account Expenditures per Estimates 20xx
Cabinet Secretariat │ December 2011
30
Guide to Developing a Risk-Based Departmental Evaluation Plan
APPENDIX F: Government Program Risk Assessment Tool
USING THE RISK-BASED ASSESSMENT TOOL
The Guide Developing a Risk-Based Departmental Evaluation Plan, Step 3, discusses
the process for completing a risk-based assessment. All department initiatives
remaining after the pre-screening process must be assessed for risk. The probability
that a program is no longer relevant, effective or efficient is the overall risk that is being
measured. Step 3 of the Guide references an online web Risk-Based Assessment tool
developed by the Economic and Statistics Branch (ESB) of the Department of Finance
to assist departments in ranking their initiatives from highest to lowest risk. The tool’s
ranking is based on seven risk variables with predefined weighted sub criteria. The
information contained in this appendix will aid employees in navigating through the web
Tool.
The tool can be accessed using the following address. Please open your internet
browser and copy and paste the following - http://inform.stats.gov.nl.ca/riskassessment.
A single user ID and password has been assigned to each Department but more than
one person can access the database at one time. Once logged in you will see the
following screen.
Cabinet Secretariat │ December 2011
31
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 1
Screen 1 provides three tab options. The “Add Risk Assessment” button is used each
time you go to complete the initial risk assessment. The “Revised Risk Assessment”
button is used to make changes to an already completed risk assessment. The “View
My Projects” button allows you to view and export to Excel or Word all risk assessments
completed by Project.
To begin a new risk assessment, click on Tab “Add Risk Assessment” and you will get
the following screen:
Cabinet Secretariat │ December 2011
32
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 2
A “Select Project” and “Program Names” appears on Screen 2. The “Select Project” tab
has been pre-populated using the Departmental abbreviation and 2012 and the
associated program names have been similarly pre-populated based on information
provided to the Provincial Government Programs Office (PGP). The PGP can add new
Projects (for example if a lead for a Horizontal Initiative decides to use this tool to assist
in developing an evaluation framework for their strategy) and or new program names.
In this illustration it presents the list of programs for CS2012 (Cabinet Secretariat)
Program Description as well as a tab for each of the seven risk factors. When printing
out reports it is the “Select Project” identifier that will be used.
All risk assessments
completed under your department’s Project name will be available for exporting and
printing.
Cabinet Secretariat │ December 2011
33
Guide to Developing a Risk-Based Departmental Evaluation Plan
To complete a risk assessment, scroll through the list of program names and select the
one you want to assess. If it does not exist, please contact the PGP. Once you select a
program and click NEXT, you will advance to the following screen.
Screen 3
Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a
significant percentage of a department’s resources may be evaluated periodically to
confirm value for money (effectiveness, efficiency and economy). 10 Helpful documents
may include: Budget Decisions, Estimates. The budget criterion is unique as there are
two screens to be completed: Budget Value and Gross Expenditure Budget.
The
screen with the highest risk (1-5) must be the only screen with the “top box” checked.
10
ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to
assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant.
Departments may choose to include initiatives which use significant resources in their Plan even if
assigned a low risk assessment using this tool.
Cabinet Secretariat │ December 2011
34
Guide to Developing a Risk-Based Departmental Evaluation Plan
In this example, the budget value was 1. Before hitting Next, click on gross
expenditure budget and complete the following screen.
Screen 4
In this example the Gross Expenditure Budget is a 3; while the Budget Value was a 1.
As the Gross Expenditure response is higher, we leave the check mark for Gross
Expenditure Budget and now hit Next (Had the Budget Value been 4 or 5, then we
would have checked the Budget Value box and hit Next). If this is lower and you have
to go back to the Budget Value screen, the previous check is removed and you will
need to reenter it.
Cabinet Secretariat │ December 2011
35
Guide to Developing a Risk-Based Departmental Evaluation Plan
Once you have complete the applicable Budget screen, hit Next and this will bring you
to the following screen:
Screen 5
Gross Expenditure Variance: This risk factor considers the significance of over or
under spending trends.
Initiatives experiencing significant over or under spending
should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing
resources adequate to address the identified population’s need). Is the initiative overresourced based on existing needs?
Helpful documents may include: Budget
Monitoring Reports, Report on the Program Expenditures and Revenues of the
Consolidated Revenue Fund.
When your selection has been entered, click Next to bring you to the next screen.
Cabinet Secretariat │ December 2011
36
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 6
Priorities: This risk factor considers how important the initiative is to achieving a
government-wide priority. Helpful documents may include: Prior year’s Departmental
Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy
Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic,
Business, Activity), Annual Report, Operational Plan, and Work Plan.
When your selection has been entered, click Next to bring you to the next screen.
Cabinet Secretariat │ December 2011
37
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 7
Funding Renewal Commitments: This risk factor applies where evaluation may
support a request to continue funding that was approved for a fixed period or for which
there is no continuing provision in the fiscal framework. Helpful documents may include:
Fiscal Framework, Budget Decisions.
When your selection has been entered, click Next to bring you to the next screen.
Cabinet Secretariat │ December 2011
38
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 8
Public Opinion: This risk factor considers client and other stakeholder feedback about
an initiative. There may be current sensitivities, interests or pressures from clients and
other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor
General, individuals, groups or MHAs). Helpful documents may include client surveys,
public consultations, Auditor General Reports, public strategy documents, departmental
correspondence such as letters to Ministers etc. If after such a review you remain
unaware of client or stakeholder feedback then click on risk level 5 as there is an
increased risk that there may be issues/concerns of which you are not aware.
When your selection has been entered, click Next to bring you to the next screen.
Cabinet Secretariat │ December 2011
39
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 9
Relevance: This risk factor considers the results of a department’s environmental scan
and SWOT (strengths, weaknesses, opportunities, threats) analysis.
Departmental
strategic planning and or horizontal strategy development may have identified changes
in the demographics and or needs of the target population that requires a revised needs
assessment, process review and/or an effectiveness evaluation (e.g. growing wait lists
for services, lower than expected take-up). Helpful documents may include: Auditor
General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan
(Strategic, Business, Activity) and Annual Report.
When your selection has been entered, click Next to bring you to the next screen.
Cabinet Secretariat │ December 2011
40
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 10
Performance Findings: This risk factor considers the findings from the performance
monitoring and or past evaluation reports.
Helpful documents may include:
Performance Monitoring and Evaluation Reports, Auditor General Reports.
Please
note that level 5 should be selected for initiatives where there are no performance
findings available. If there are no performance findings, there is increased risk that an
initiative is not performing as well as one may think.
Screen 10, performance findings, is the final of the seven risk criteria. Upon completing
Screen 10, click on the Review Selections tab.
Note: On this or any other criteria screen, you can click Back to go to an earlier screen
or if you want to go a specific criterion to review or change you can click on the
applicable top tab.
Cabinet Secretariat │ December 2011
41
Guide to Developing a Risk-Based Departmental Evaluation Plan
Risk Overview: Screen 11 provides an overview of the selections you have chosen for
all criteria for a particular initiative. This screen provides the options of editing your
selections or calculating your risk assessment level.
Screen 11
If you are satisfied with your risk choices, click Calculate Risk Assessment Level.
Please note you can change your calculations at a later date, by using the Revise Risk
Assessment on Screen1.
Once you select the Calculate Risk Assessment Level you will see the following screen.
Cabinet Secretariat │ December 2011
42
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 12
As each additional initiative is assessed, its summary is added to your Risk Assessment
Report in accordance with its risk level, thus providing a final report of all initiatives
ranked by risk. A Risk Assessment Report, illustrated below, can be generated anytime
by clicking the View My Risk Assessment Projects tab on Screen 1 (Home Page) and
converting to Excel or Word.
Cabinet Secretariat │ December 2011
43
Guide to Developing a Risk-Based Departmental Evaluation Plan
Risk levels are colour coded, as illustrated below, with blue representing lowest risk and
red representing highest risk. The black markers represent the calculated risk level of
each initiative listed in your Risk Assessment Report.
Screen 12 will require you to click the EXIT button. Doing so will take you back to
Screen 1 (Home Page) where you can begin a new risk assessment on another
initiative, revise any of your previous initiatives’ risk assessments, or view all your risk
assessments.
Cabinet Secretariat │ December 2011
44
Guide to Developing a Risk-Based Departmental Evaluation Plan
Screen 1
To view all your completed risk assessments and produce an Excel or Word report, click
the View My Projects button. The following is an example of a screen you will see.
Cabinet Secretariat │ December 2011
45
Guide to Developing a Risk-Based Departmental Evaluation Plan
If you do not want to print a particular program, you can mark the box to the left of the
applicable program name and hit Remove Selected Program. Once you are satisfied
with the report you have the option to export to Excel or Word by clicking the applicable
button.
Once you are finished running reports or completing risk assessments, you can exit by
closing your browser.
Cabinet Secretariat │ December 2011
46