Download Methodologies of monitoring and evaluating capacity

UNFCCC Expert Workshop on Monitoring and Evaluating
Capacity-Building in Developing Countries
Rio de Janeiro, Brazil, 6-7 November, 2008
Methodologies of monitoring and
evaluating capacity development
Makoto Kato
Japan
[email protected]
1
Outline
• Defining Capacity, and Capacity Development
• Methodologies and Tools for Monitoring and
Evaluating Capacity Development
• Comparison? M&E at national level?
• Uniqueness of Capacity Development in
Climate Change area
2
Defining Capacity, and Capacity
Development
Capacity:
Developing countries’ ability to cope with
challenges by their own organization and
actions.
Capacity development:
Process in which such capacity is being
strengthened and sustained at individual,
institutional, and society levels as a whole.
Source: JICA 2006
3
Capacity at different levels
(Capacity Development must be comprehensive)
Consensus in Burden
sharing by different
stakeholders
New Waste Segregation
Rules
Society Level
Policy Goals toward
Waste Reduction
Institutional Level
Awareness raised about
Waste Reduction
Individual Level
Strengthened
Management of Waste
Control Authority
Administrators’
Capacity
Enhancement
Source: JICA 2006 4
Different Approaches to Capacity Development
(Capacity Development should be country-driven and sustainable)
Level of Capacity necessary to cope with challenges
External
Capital Transfer
Not
Sustainable
Existing Capacity
Approach A:
Transfer of capacity from external player
(Filling a Gap with external capital transfer)
External Input
(Incl. Knowledge
Sharing)
More
Sustainable
Existing Capacity
Approach B:
Developing capacity with external
facilitation(building capacity under the
ownership of host countries/communities)
Source: JICA 2006
5
Key Drivers of Capacity Development
•
•
•
•
•
Ownership
Enabling Environment
Incentives
Leadership
Knowledge
6
Methodologies and Tools for Monitoring and
Evaluating Capacity-Building (1)
Flow of M&E
Conceptual Figure of Project Cycle Management(PCM)
Plan
Evaluation
Identification
Evaluation
Formulation
Implementation /
Monitoring
Preparation
Implementation
7
Methodologies and Tools for Monitoring and
Evaluating Capacity-Building (2)
Where indicators are used:
Format of Project Design Matrix （PDM）
Narrative Summary
Indicators
Means of
Verification
Important
Assumption
-
-
-
Overall Goal
Vertical
Logical
Relationship
Project Objective
Outputs
Activities
Inputs
Pre-conditions
8
Methodologies and Tools for Monitoring and
Evaluating Capacity-Building (3)
How Overalls Goals/project objectives and Indicators are made?
Problem Tree
Effect
Cause
Core Problem
Methodologies and Tools for Monitoring and
Evaluating Capacity-Building (3)
How Overalls Goals/project objectives and Indicators are made?
Objective Tree
Ends
Core Objective
Feasibility
Means
Expected Situations
Methodologies and Tools for Monitoring and Evaluating
Capacity-Building
• Evaluation is heavily dependent on “Target Setting”
(based on capacity assessment)
Target in different levels: Overall Goal, Objectives of Efforts
(eg. Policy/Programme/Project objectives)
• Indicators are useful only if/when they are elaborated
with means of verification, important assumption and precondition (used in PDM)
• M&E are properly done by Project Owners (and
contracted external evaluators)
11
Can we compare?
Same type Projects in 2 different countries
Country A
Governance
Structure
Project X
Development
Priority
Country B
Governance
Structure
Development
Priority
Existing Capacity
Existing Capacity
Other attributes
(eg, Size)
Other attributes
(eg, Size)
Project Y
Comparison is only useful for drawing lessons from “Good Practices”
(Simple replication does not happen)
12
Can we compare?
Same type Projects in the same country
Country C
Governance
Structure
Development
Priority
Project Q
(Internal Factors)
• Willingness of Stakeholders
•Entry point of Efforts etc.
Existing Capacity
Other attributes
(eg, Size)
Project P
(Internal Factors)
• Willingness of Stakeholders
•Entry point of Efforts etc.
Easier to compare the two, since the external factors are the same or similar.
But still internal factors of projects should be taken into account.
13
Can we compare? M&E at National Level
Accuracy of Comparison (Evaluation)
• Project level>Programme level>Policy level>
Regional/International Network
(At higher level, quantified indicators may omit useful qualified
information)
Methods of M&E
• We use the same M&E methods(some variety), but indicators are
more tailor-made
Pre-condition of Defining Indicators
• Capacity Assessment (jointly done by D-ing+D-ed countries) is
crucial
• Assessment(BAU) Monitoring(Indicators) Evaluation
(Indicators) must be coherent, and reflect a specific context.
14
Uniqueness of Capacity Development in
Climate Change area
Narrative Summary
Indicators
Overall Goal
-
Means of
Important
Assumption
• ThisVerification
Area is already decided
by
2/CP7.
• Fill in one of the 15 scopes
e.g. (promotion of) CDM
-
Project Objective
Outputs
Activities
Inputs
Pre-conditions
Entry points of such efforts are different by host countries. Host countries
chose such entry point, jointly with our Agencies.
15
Conclusion from our practice
• A single methodology (with variation) for M&E is used.
• Indicators are used and functions in specific context (within
PDM), and simple aggregation of evaluation results is strictly
avoided.
• Indicators are useful to interpret lessons from Good Practices.
• Evaluation at larger level employs more qualitative/narrative
way.
• Entry points of efforts are different by countries, and it affects
selection of indicators.
• For climate change, we approach both from “Overall Goal” and
from the bottom side of PDM( and it still works).
16