Download Ex-post comparison of project and non-equivalent control

APPLICATION OF
GWM&E: AN EYE ON
IMPACT EVALUATION
BY
EDWIN OKEY IJEOMA, PhD.
UNIVERSITY OF PRETORIA
 INTRODUCTION
OUTLINE
 M&E ORIGIN & GLOBAL PERSPECTIVE
 MONITORING, EVALUATION & IMPACT
ASSESSMENT
 LINKING MONITORING & EVALUATION
 IMPACT EVALUATION/ASSESSMENT
 TYPES OF IMPACT EVALUATION
 GWM&E OUTLOOK
 GWM&E POLICY IMPERATIVE
 CONCLUDING REMARKS
INTRODUCTION
 What is M&E
 Why M&E
 What is the benefits
 What are its principles
 How does one apply M&E
 Whose responsibilities is to apply M&E
 What impacts does it make
--- To the people
---To government/stake holders
---To practioners
M&E IN
PERSPECTIVES
 Before 19th Century concepts of different countries &
regions
 Guidelines and policies existed within various
organisations /NGOs
 OECD Evaluation policies of the international financial
institutions
 Evaluation polices of the
---World Bank
---IMF
---European Union etc.
 U.N Coordinated efforts A/RES/59/250 2004. To
encourage meeting the MDGs in 2015
 UN forms the Un Evaluation Group
MONITORING, EVALUATION
AND IMPACT ASSESSMENT
 Monitoring: Various definitions (OECD)
---A continuous function
---Systematic collection of data
---Development interventions
---Indication of the extent of progress
---Achievement of objectives and progress in
allocation of funds.
 MONITORING: (IJEOMA 2007)
A continuous assessment of both the functioning of the
project activities in the context of implementation
schedules and the use of project inputs by the
targeted population in the context of expectations.
---To ensure that inputs & work schedules are
proceeding according to plan
---Provide records of input use, activities and results:
Early warning of deviation from initial goals and
excepted outcomes
EVALUATION
Evaluation: A broader concept than monitoring.
Evaluation is the “ the systematic and objective
assessment of an ongoing or completed project,
programme, or policy, including design,
implementation, and results: The aim is to
determine the relevance and fulfillment of
objectives, development efficiency, effectiveness,
impact and sustainability (OECD)
EVALUATION
Broader than Monitoring
 Performance: compares achievements
with expected output
 Quality: Adherence to accepted
standards of scientific work precision
 Relevance of Research: Investigates
research is relevance to objectives
 Impact: The effects on the output to the
ultimate uses. “People level impact”
TYPES OF
EVALUATION
 EX-ANTE EVALUATION
---Occurs before the event aimed at assessing the potential
impact of evaluation
 ONGOING EVALUATION
---Occurs during the event aimed at evaluating the
performance and quality in progress
 EX-POST EVALUATION
---Occurs immediately after the event aimed at the successful
completion and relevance of evaluation
 IMPACT EVALUATION
---Several months, years aimed at assessing if results have
been achieved
IMPACT
EVALUATION
‘IMPACT’ definitions are also over stretched .it means
different thing to different people
---The UN has impact definition
---OECD has impact definition
---World & IMF has impact definition
---SA GWM&E has impact definition
 Impact evaluation is the systematic identification of
the effects-positive or negative, intended or not-on
households, institutions and the environment
caused by a given development activity such as a
programme or project (World Bank 2006)
USES OF IMPACT
EVALUATION
 Measuring outcomes and impacts of an activity
 Distinguish the outcomes and impacts from other
external factors
 Clarification or justification of cost on an activity
 Well informed policy decisions on
---expansion
---modification
---elimination
 Drawing lessons on managing future
activity
 Comparing the effectiveness of
alternative interventions
 Strengthening Accountability for results
MERITS OF
IMPACT EVALUATION
 Provides estimates of the magnitude of
outcomes and impacts for different
democratic group, region or overtime
 Provide answers to some of the most
central development positions- To what
extent are the difference made
 Systematic analysis and rigor give
decision-makers added confidence
DEMERITS OF
IMPACT EVALUATION
 Some approaches are expensive and
time consuming
 Reduce utility when decision-makers
need information quickly
 Difficulties in identifying counter-factual
issues
MODELS OF
IMPACT EVALUATION

Randomized pre-test or post-test
evaluation
 Design
Subjects (families, schools, communities etc) are
randomly assigned to project and control
groups. Questionnaires or other data
collection instruments (anthropometric
measures, school performance rests, etc) are
applied to both groups before and after the
project intervention. Additional observations
may also be made during project
implementation
 Example
Water supply and sanitation or the provision of other
services such as housing, community infrastructure etc
where the demand exceeds supply and beneficiaries are
selected by lottery. Example: Bolivia Social Fund.
 Indicative cost and time
1-5 years depending on time which must elapse before
impacts can be observed. Cost on range from $50,000 $1 million depending on the size and complexity of the
program being studied
Quasi-experimental design: before and after comparisons of
project and control populations.
 Design
Where randomization is not possible, a control group is selected
with marches the characteristics of the project group as closely
as possible. Sometimes the types of communities from which
project participants selected for subsequent phases can be
used as the control for the first phase project group
 Examples
These models have been applied in World Bank low-cost housing
programs in El Salvador, Zambia, Senegal and the Philippines
 Indicative cost and time
Cost and timing similar to Model 1
Ex-post comparison of project and non-equivalent
control group
 Design
Data are collected on project beneficiaries and a nonequivalent control group is selected as in Model 2. Data
are only collected after the project as been implemented.
Multivariate analysis is often used to statistically control for
differences in the attributes of the two groups.
 Example
Assessing the impacts of micro-credit programs in
Bangladesh. Villages where micro-credit programs were
operating were compared with similar villages without
these credit programs
ASSESMMENT OF EX-POST
 Design
Some evaluations only study groups affected by the project
while others include matched control groups. Participatory
methods can be used to allow groups to identify changes
resulting from the project, who has benefited and has not, and
what were the project’s strengths and weakness.
Triangulation is used to compare the group information with
the opinions of key informants and information available from
secondary sources. Case studies on individuals or groups
may be produced to provide more in-depth understanding of
the processes of change.
 Example
Assessing community managed water supply projects in
Indonesia
 Indicative cost and time
$25,000 upwards (the Indonesia study cost $150,000). Some
studies are completed in 1-2 months; others take a year or
longer.
LINKING MONITORING WITH
EVALUATION
Evaluation
Monitoring
Recording
Information from
other sources
Information form
Monitoring
(data)
Analysis
Analysis
Recommendation
information
Reporting
Corrective Action at the
Operation Level
Storage
Affirmation or
Modification of
Objectives, Resources
& Processes
Rediagnosis &
Replanning
Ex-post Impact
Assessment
Feedback
Diffusion
Diagnosis
Monitoring
Evaluation
Recommendation
Ex-ante Evaluation
Impact Assessment or
Screening
Implementation
Ongoing evaluation &
monitoring
Planning

GWM&E
Definition of a M&E systemOUTLOOK
Formal managerial elements are the organisational culture,
capacity and other enabling conditions which will
determine whether the feedback from the M&E function
influence the organisation’s decision-making, learning and
service delivery.
The GWM&E Framework seeks to embed a management
system within public sector organisations which
articulates with other internal management systems (such
as planning, budgeting and reporting systems).
 Relationship between institutional M&E systems and
GWM&E system
 Create an overall picture of national, provincial and local
performance
System Overview

Aim of the GWM&E System
Provide an integrated, encompassing framework of M&E
principles, practices and standards to be used throughout
Government, and function as an apex-level information
system which draws from the component systems in the
framework to deliver useful M&E product for its use.

Overview
One of the ways Government is increasing effectiveness is by
concentrating on monitoring and evaluating. This is
because it is a pivotal competence that has positive
effects but up and downstreams.

SYSTEM DESCRIPTION
The GWM&E system is intended to facilitate a clear sequence of events based
on critical reflection and managerial action In response to analysis of the
relationship between the deployment of inputs, the generation of service
delivery outputs, their associates outcomes and impact
The GWM&E system intends to produce the following outputs:
 Improved quality of performance information and analysis at programme
level within departments and municipalities (inputs, outputs and outcomes).
 Improved monitoring and evaluation of outcomes and impact across the
whole of government through, e.g. Government Programme of Action bimonthly Report based on the national Indicator
 Sectoral and thematic evaluation reports
 Improved monitoring and evaluation of provincial outcomes and impact in
relation to Provincial Growth and Development Plans
 Projects to improve M&E performance in selected institutions across
government
 Capacity building initiatives to build capacity for M&E and foster a culture of
governance and decision-making which responds to M&E findings








GWM&E: Guiding
Principles for Implementation
The implementation should be clearly linked with priority public sector
reforms initiatives
As far as possible, the GWM&E framework should incorporate and
consolidate existing M&E initiatives in the three spheres, aligning them to
the overall aims of government
Roles and responsibilities of each stakeholders should be clearly defined
and related to their mandate.
The implementation plan should adopt a differentiated approach across
spheres and sectors
The administrative burden of compliance across government should be
minimised
Where M&E systems are supported by IT solutions, the emphasis will be
on systems integration and ease of data interchange.
Monitoring and the development and enforcement of statistical standards
are important pre-conditions for effective evaluation
Regular review of the implementation plan against milestones
CONCLUDING
REMARKS
Implementation of the GWM&E has a legal mandate-approved by
Parliament and Cabinets
 Institutional Mandate- To be driven by the Presidency
 Implementation Stakeholders- identified in Govern. departments and
Ministries

AREAS OF FURTHER RESEARCH/ CONCERN
HOW TO MAINSTREAM THE M&E POLICY IN THE PUBLIC SECTOR
WORK PLACE
 Clear definition of the responsibilities of the Implementation agencies
 The use of the M& E results for public policy-making
 Identification of early warning through the M&E systems against, fraud,
corruption and Mal-administration in government.
 How do you convince the politicians that most projects and programmes of
govt. may be impact-driven, i.e. the results may only be reflected in the
longer term
 Mass marketing of the need and Benefits of M&E system in Public sector
Mgt.