Download 06a - Harmnozation approaches - ECG draft oct 21

Approaches to Harmonizing
Public and Private Sector Project Evaluation
Draft: October 21, 2013
Acronyms
CAS
CED
CPS
DAC
DE
DEM
E&S
EBRD
ECG
EIB
EP
ERMP
EROIC
ERR
ESMF
ESMS
EV
EvD
FRR
GOI
GPS
ICRR
IDB
IEG
IFC
IFI
ILO
IT
M&E
NPV
NSG
OECD-DAC
OP
PMGSY
QAE
ROAA
ROAE
ROIC
SEMS
SG
SME
TC
WACC
WB
WBG
Country Assistance Strategy
Central Evaluation Department
Country Partnership Strategy
Development Assistance Committee (of OECD)
Development Effectiveness
Development Effectiveness Matrix
Environmental and Social
European Bank for Reconstruction and Development
Evaluation Cooperation Group
European Investment Bank
Evaluation Principle
Environmental Risk Management Plan
Economic Return on Invested Capital
Economic Rate of Return
Environmental and Social Management Framework
Environmental and Social Management System
Evaluation Department (EIB)
Evaluation Department (EBRD)
Financial Rate of Return
Government of India
Good Practice Standards
Implementation Completion and Results Report (WBG)
Inter-American Development Bank
Independent Evaluation Group (WBG)
International Finance Corporation
International Financial Institution
International Labor Organization
Information Technology
Monitoring and Evaluation
Net Present Value
Non-Sovereign Guarantee
Organization for Economic Cooperation and Development –
Development Assistance Committee
Operational Practice
Pradhan mantra Gram Sadak Yojana (Prime Minister’s Rural
Road Program) (India)
Quality at Entry
Return on Average Assets
Return on Average Equity
Return on Invested Capital
Social and Environmental Management System
Sovereign Guarantee
Small- and Medium-Scale Enterprise
Technical Cooperation
Weighted Average Cost of Capital
World Bank
World Bank Group
Contents
1.
Introduction ............................................................................................................................. 1
2.
Similarities and Differences Between the Public GPS and Private GPS ................................... 3
3.
Option 1: A Common Reporting Framework .......................................................................... 9
Mapping the Existing GPS Criteria to OECD-DAC Criteria........................................................ 9
Private Sector Application: IFC Telecommunications Project .............................................. 15
Public Sector Application: World Bank Telecommunications Project .................................. 18
4.
Option 2: A Common Evaluation Methodology ................................................................... 21
The Basis of the Evaluation.................................................................................................... 23
Evaluation Criteria ................................................................................................................. 24
Evidentiary Requirements, Analytical Methods, and Benchmarks ....................................... 31
Private Sector Application: IFC Tier I Bank Financing Project ............................................... 42
Public Sector Application: World Bank Rural Roads Project................................................. 51
Annex 1: Glossary of Terms and Definitions................................................................................. 59
Annex 2: Recent Developments in ECG Members ....................................................................... 62
Tables
Table 1: Evaluation Criteria in the Public GPS ................................................................................ 4
Table 2: Evaluation Criteria in the Private GPS............................................................................... 5
Table 3: Mapping the Existing GPS Criteria to OECD-DAC Criteria ............................................... 12
Table 4: The Basis of the Evaluation ............................................................................................. 23
Table 5: Option 2 Criteria and Sub-Criteria .................................................................................. 28
Table 6: Evidentiary Requirements, Analytical Methods, and Benchmarks ................................ 33
Figures
Figure 1: Option 1 Criteria and Sub-Criteria ................................................................................. 15
Figure 2: Option 2 Criteria and Sub-Criteria ................................................................................. 27
This paper was prepared for the Evaluation Cooperation Group meeting in October 2013 by Kris
Hallberg (consultant and lead author) and Nicholas Burke (contributing consultant) under the
direction of Cheryl Gray (Director, Office of Evaluation and Oversight, Inter-American
Development Bank) and Marvin Taylor-Dormond (Director of Private Sector Evaluation,
Independent Evaluation Group, World Bank Group).
1.
Introduction
1.1
Members of the Evaluation Cooperation Group (ECG) have worked over many years to
harmonize their evaluation methodologies through the development and application of Good
Practice Standards (GPS) for evaluation. GPSs have been developed in several areas, including
for the evaluation of individual public sector and private sector operations. The latest versions
of the Private GPS and Public GPS were approved by ECG in 2011 and 2012 respectively.1
1.2
While the Public and Private GPSs have been instrumental in moving forward the evaluation
agendas and practices in many IFIs, the Boards of Directors of some IFIs have questioned the
reasons for the differences in public and private sector evaluation approaches and have asked
whether it might be possible to attain greater consistency among them. In addition,
comparisons are inevitably drawn between the aggregate evaluation results of public sector and
private sector operations when they are reported together in periodic Central Evaluation
Department (CED) synthesis reports, inviting the question of whether differences in aggregate
ratings are due to actual differences in performance or to differences in the way these
operations are evaluated. And in sector, thematic, and country evaluations, CEDs often find
themselves evaluating public and private sector operations using different criteria, making it
difficult to present comparable results.
1.3
For these reasons, several ECG members have been experimenting with more harmonized
approaches to evaluating public sector and private sector operations. The Inter-American
Development Bank (IDB) has created a development effectiveness framework for private sector
operations. The European Bank for Reconstruction and Development (EBRD) is preparing a
guidance note on evaluation criteria that can be applied to both public and private sector
projects. The European Investment Bank (EIB) is using a common set of evaluation criteria for
public and private sector projects in its thematic evaluations.
1.4
At the April 2013 ECG meeting, Nicholas Burke (Consultant) presented a paper describing areas
of commonality and divergence between the Public GPS and Private GPS.2 The paper found that,
for the most part, differences between the two GPSs reflect the unique features of public and
private sector interventions, the IFI’s corresponding institutional mandates, their operational
structures, their instruments of support, and the nature of their client relationships. In another
session at the same meeting, Alejandro Soriano (IDB) and Kris Hallberg (Consultant) applied the
Public GPS and Private GPS to projects with a financial intermediary that had received both
sovereign guaranteed and non-sovereign guaranteed loans from IDB. It was found that the
different criteria and evidentiary requirements in the two GPSs could potentially lead to
different assessments and ratings.
1.5
Using these efforts as a starting point, this paper proposes ways to reduce the divergence in
how public sector and private sector evaluations are prepared and reported. The proposed
1
Evaluation Cooperation Group (2011). Good Practice Standards for the Evaluation of Private Sector
Operations. Fourth Edition, November 8; and Evaluation Cooperation Group (2012). Good Practice
Standards for the Evaluation of Public Sector Operations. 2012 Revised Edition, February.
2
Burke, Nicholas (2013). The Private and Public Sector Good Practice Standards: A Background and
Comparison. Paper presented at the April 2013 ECG meeting.
-1-
approaches apply to the evaluation of individual projects, rather than higher-level evaluations at
the sector, thematic, or country level, although there may be applicability to these evaluation
products as well. As is the case in the Public GPS and Private GPS, the paper focuses on
investment operations as opposed to non-lending advisory services and technical assistance.
Applicability to policy-based lending has not yet been tested.
1.6
ECG members differ in their visions of what a common approach would look like. Some have
expressed a desire to develop a “common reporting framework”. This would involve mapping
the existing criteria in the Public and Private GPSs to common headings – for example, to the
OECD-DAC evaluation criteria – with “modules” or “branches” to the existing criteria in the two
GPSs. A common reporting framework, called Option 1 in this paper, would not require
significant changes to the criteria or definitions in the current GPSs.
1.7
Other ECG members would like to go further to define a “common methodology” -- including
criteria, guidelines for their application, ratings scales, and evidentiary requirements -- that
could be applied to both public and private sector operations. A common methodology, called
Option 2 in this paper, would involve changes to the standards in both GPSs, some of them
significant.3
1.8
Even under a common methodology, some differences in approach may be warranted to
accommodate the unique characteristics of public and private sector projects. In these cases,
branches to the different approaches are presented. In addition, in areas in which there may be
more than one approach to solve a problem, the paper presents the options and discusses the
advantages and disadvantages of each.
1.9
It should be noted that differences in evaluation results between public sector and private
sector operations may arise not just because of differences in evaluation methodologies, but
also because of differences in the rigor with which those methodologies are applied in practice.
Determining the extent to which the GPSs are being applied would require benchmarking a
representative sample of evaluations from all ECG members, which is outside the scope of this
paper. Instead, the paper focuses only on differences in the standards contained in the two
GPSs.
1.10
The next section of the paper discusses the key similarities and differences between the Public
GPS and Private GPS, and the relative strengths and weaknesses of each. Option 1 (a common
reporting framework) and Option 2 (a common evaluation methodology) are presented with a
discussion of the advantages and disadvantages of each. For illustration, both of these options
are applied to examples of private sector and public sector projects.
1.11
Annexes provide definitions of terms used (Annex 1) and recent efforts of ECG members to
harmonize evaluation criteria (Annex 2).
3
Other options could be considered, such as the so-called “Option 3” which proposes to address the
weaknesses and gaps in the existing GPSs and be implemented in combination with Option 1.
-2-
2.
Similarities and Differences Between the Public GPS and Private
GPS
2.1
The Public GPS and the Private GPS are already harmonized or mostly harmonized in a number
of areas. The main area of full harmonization is the independence of the CED. The Private GPS
contains a section on independence which is similar in content to the separate GPS on CED
independence.4 The Public GPS does not have its own section on independence, instead
referring to the Independence GPS.
2.2
Areas related to CED reporting, dissemination, and learning from evaluations are substantially
harmonized. These include standards on the CED’s delivery of synthesis reports to the Board,
CED reports on the quality of the IFI’s self evaluation system, the dissemination of evaluation
results to a wide range of audiences, and CED monitoring of IFI Management’s implementation
of CED recommendations. With respect to disclosure, the Public and Private GPSs have a similar
standard, namely that the CED’s disclosure policy is explicit and is consistent with the IFI’s
general disclosure policy. In practice, to protect the proprietary information of private sector
clients, few evaluations of private sector operations are disclosed, and if they are, it is in a
redacted or summarized form. In terms of evaluation processes and report preparation, the
Public GPS and the Private GPS are mostly similar.
2.3
In other areas, there are notable differences in standards between the two GPSs. In order of
importance for the comparability of evaluation results, the most significant differences are:
2.4
Evaluation criteria: The evaluation criteria currently used in the Public GPS and the Private GPS
are shown in Tables 1 and 2. There are differences in the terminology and definitions of criteria
as well as the methodology of the assessment. The Public GPS core criteria correspond directly
to the OECD-DAC evaluation criteria of Relevance, Effectiveness, Efficiency, and Sustainability.
The correspondence of the Private GPS criteria to OECD-DAC is less obvious, although there is at
least partial correspondence of Financial Performance, Economic Sustainability, Contribution to
IFI Mandate Objectives, and Environmental and Social Performance. Both GPSs contain criteria
to assess IFI performance. In the Public GPS, the performance of the client is assessed
separately in a single criterion (Borrower Performance). In the Private GPS, the performance of
the client is manifest in the financial performance and compliance of the project company, and
is therefore considered in the GPS criteria of Financial Performance and Environmental and
Social Performance.
4
Evaluation Cooperation Group (2010). Good Practice Standards on the Independence of Central
Evaluation Departments. June.
-3-
Table 1: Evaluation Criteria in the Public GPS
Criteria and Sub-Criteria
Relevance
Effectiveness
Efficiency
Definition
Consistency of the project’s objectives with beneficiary
needs, the country’s development or policy priorities and
strategy, and the IFI’s assistance strategy and corporate
goals (the relevance of objectives); and the adequacy and
coherence of the project’s components to achieve those
objectives (the relevance of design).
The extent to which the project achieved, or is expected
to achieve, its stated objectives, taking into account their
relative importance.
The extent to which the project converted its resources
economically into results.
Sustainability
The likelihood of continued long-term benefits, and the
resilience to risk of net benefit flows over time.
IFI Performance
The quality of services provided by the IFI during all
project phases, including the IFI’s performance in
ensuring project quality at entry, satisfactory
implementation, and future operation.
Borrower Performance
The adequacy of the Borrower’s assumption of ownership
and responsibility during all project phases.
Assessment Methodology
The assessment of the relevance of objectives also considers the
extent to which the project’s objectives were clearly stated and
focused on outcomes as opposed to outputs. The relevance of
design also considers whether the project’s financial instrument was
appropriate to achieve project objectives and country needs.
Tests the validity of the anticipated links between the project’s
activities, outputs, and intended outcomes (i.e., the results chain).
Outcomes are evaluated against a counterfactual.
Uses cost-benefit analysis (NPV or ERR) to the extent that data is
available and it is reasonable to place a monetary value on benefits.
In addition, in all cases, cost effectiveness analysis is conducted.
The assessment of Sustainability considers various aspects such as
technical, financial, economic, social, political, and environmental
sustainability.
Summary Rating: Aggregate Project Performance Indicator
-4-
The assessment of quality at entry covers the IFI’s role in ensuring
project quality and in ensuring that effective arrangements were
made for satisfactory implementation and future operation of the
project. The assessment of the quality of supervision is based on the
extent to which the IFI proactively identified and resolved problems
during project implementation. IFI Performance includes the
application of relevant environmental and social safeguards.
The assessment of Borrower performance considers the
effectiveness of both the government and the implementing agency
in ensuring quality project preparation and implementation,
compliance with covenants and agreements, establishing the basis
for sustainability, and fostering participation by the project’s
stakeholders.
Table 2: Evaluation Criteria in the Private GPS
Criteria and Sub-Criteria
Financial Performance and
Fulfillment of Project
Business Objectives
Definition
Financial performance: the project’s contribution
to the company’s financial results, or the
company’s financial results where the project is
indistinguishable from the company. Fulfillment of
business objectives: the extent to which the
project delivered on its process and business
objectives.
Economic Sustainability
The project and/or company’s contribution to
growth in the economy.
Contribution to IFI Mandate
Objectives
Depending on the IFI, mandate objectives may
include private sector development, the
development of efficient financial and capital
markets, or transition to a market economy.
Environmental and Social
Performance
The project company’s overall environmental and
social performance in the area of influence of the
project.
An optional supplementary sub-criterion of
Environmental and Social Performance is called
“Environmental and Social Impact”: the extent of
improvement (or deterioration) in the client’s
environmental and social performance since
project approval.
Assessment Methodology
Based on the FRR or ROIC, the achievement of returns projected at
appraisal, the achievement of process and business goals articulated at
approval, performance indicators in financial statements compared to
appraisal projections, and the company’s business prospects. For financial
intermediary operations, if possible, the assessment also considers the
performance of the sub-portfolio or fund. Financial performance is
assessed on a with-vs.-without project basis, or a before-vs.- after project
basis.
Considers the impact of the project on all key economic stakeholders –
including and beyond the company’s owners and financiers (e.g.,
employees, suppliers, competitors, neighbors). Economic Sustainability is
determined by the project’s ERR or EROIC, along with qualitative
stakeholder analysis. For financial intermediary operations, Economic
Sustainability includes the economic activities of sub-borrowers or the
economic viability of fund investees. Economic Sustainability is assessed
on a with-vs.-without or before-vs.-after basis.
Considers positive and negative contributions to, e.g., competition;
market expansion; private ownership and entrepreneurship; frameworks
for markets; transfer and dispersion of skills; demonstration effects;
standards for corporate governance and business conduct; development
of financial institutions and financial/capital markets; attracting FDI flows;
and development of physical infrastructure. A rating of “Neutral” is
permitted in cases where a project has no observable or attributable
impacts relevant to the IFI’s mandate objectives.
The assessment is based on the project company’s compliance primarily
with the IFI’s specified standards at approval and, secondarily, with those
prevailing at the time of the evaluation. For financial intermediary
operations, the assessment considers the financial intermediary’s or fund
manager’s Environmental and Social Management System (ESMS) and its
implementation. If required by the IFI’s specified standards, it considers
the environmental and social effects of sub-projects or fund investee
companies. A rating of “Not Applicable” is permitted where, by virtue of
the project’s expected lack of environmental and social impacts, the IFI
did not prescribe relevant safeguards.
Summary Rating: Project Outcome
-5-
Criteria and Sub-Criteria
IFI Investment Profitability
Definition
The profitability of each of the IFI’s investment(s) in
the project company.
IFI Work Quality/Bank
Handling
The quality of the IFI’s pre- and post-commitment
work.
IFI Additionality
The IFI’s value proposition (value added) in
providing support to the project.
-6-
Assessment Methodology
Based on either (i) the investment’s net profit contribution measured in
risk-adjusted, discounted cash flow terms, or (ii) the quality of the
investment’s gross profit contribution.
The quality of the IFI’s pre-commitment work considers all aspects of the
IFI’s work in screening, appraising, and structuring the project and the
IFI’s associated investment. The quality of post-commitment work
considers all aspects of the IFI’s portfolio responsibilities in monitoring
and supervising the project and the IFI’s associated investment.
Considers the IFI’s financial additionality in providing funding and/or
catalyzing other funding; and its non-financial additionality in improving
the project’s risk profile, design, or functioning. The rating is based on a
comparison with the counterfactual of how the project would have
proceeded without IFI support.
2.5
Basis of the evaluation: The Public GPS says that evaluations are primarily objectives-based.
This means that projects are evaluated against the statement of objectives found in the design
documents, or as reconstructed by the evaluator. In contrast, the Private GPS does not require
a project-specific statement of its development, policy, or transition objectives. However,
consistent with international banking and investment practices, project-specific business
objectives and market-based benchmarks are used to assess commercial performance.
2.6
Attribution: The Private GPS calls for with-versus-without-project comparisons, or beforeversus-after-project comparisons, for two criteria (Financial Performance and Fulfillment of
Business Objectives, and Economic Sustainability). This means that, if a before-versus-after
approach is taken, outcomes would not necessarily be attributable to the project.
Environmental and Social Performance is based primarily on compliance with safeguards rather
than on environmental and social performance compared to a counterfactual (an optional subcriterion, Environmental and Social Impact, does not call for comparison with a counterfactual).
IFI Additionality calls for comparison with a counterfactual of no IFI support. The Public GPS
calls for comparison with a counterfactual in the Effectiveness criterion, and is more specific on
how causality is to be established: either through an impact evaluation or through a theorybased approach. It does not say anything about using a with-versus-without project analysis
under the Efficiency criterion, although this is implied because the benefits used in the analysis
are those evidenced under Effectiveness.
2.7
Safeguards: Environmental and social safeguards receive extensive treatment in the Private
GPS, and the rating on this criterion is included in the overall project performance rating. In the
Public GPS, safeguards are given much less emphasis. They enter only as part of the IFI
Performance and Borrower Performance assessments, which are not included in the overall
project performance rating.
2.8
Definitions of performance indicators, rating scales, and threshold values: The Public GPS does
not define the rating scales (e.g., what “Moderately Satisfactory” means). With the exception of
the economic rate of return, performance indicators and targets are presumed to be projectspecific and thus there are no GPS-defined indicators or threshold values for a satisfactory
rating. In contrast, the Private GPS contains detailed standards on performance indicators,
rating scales, and threshold values for aspects related to business performance and the project’s
ERR/EROIC. Like the Public GPS, however, it does not have performance indicators or
benchmarks for development, policy, or transition results.5
5
Confusion sometimes arises because of differences in the use of the term “indicator” in the Public GPS
and Private GPS. The Public GPS refers to “performance indicators” or “outcome indicators” as variables
that can be expressed in quantitative terms (e.g., the project’s ERR; the number of project beneficiaries;
the public sector share of GDP). This is consistent with OECD-DAC definition of “performance indicator”
(“a variable that allows the verification of changes in the development intervention or shows results
relative to what was planned”). In the Private GPS, some of the “outcome indicators” are qualitative in
nature, consistent with the OECD-DAC definition of “indicator” (“a quantitative or qualitative factor or
variable that provides a simple and reliable means to measure achievement, to reflect the changes
connected to an intervention, or to help assess the performance of a development actor”). These
qualitative factors would be called “sub-criteria” or “performance attributes” or “performance
dimensions” in the Public GPS. In this paper, “performance indicator” means a measurable variable. The
qualitative factors to be considered are contained in the definitions of sub-criteria and the evidentiary
requirements for rating criteria and sub-criteria.
-7-
2.9
Evaluation methods for projects of different types: The Private GPS defines four main types of
projects, and tailors the evaluation methodology to the particular type of project. The project
categories are A – direct investments; B - corporate investment programs; C – financial
diversification or short-term funding; and D – investments in multiple sub-projects, either
through a financial institution (D1) or a fund (D2). The Public GPS distinguishes only two types
of projects – investment operations (including technical assistance loans) and policy-based
operations – and has few differences in evaluation methodology for the two types.
2.10
CED involvement in the self evaluation system: The Public GPS establishes an institutional
model in which the CED has little involvement with the self evaluation system. The CED
supports project evaluability and the quality of self evaluation reports (e.g., through IFI staff
training) but does not write guidelines for self evaluation. Instead, the CED and IFI
Management share responsibility for harmonizing self- and independent evaluation guidelines.
In the Private GPS, the CED is responsible for developing guidance for both self- and
independent evaluation, in consultation with IFI Management. As a result, the self- and
independent evaluation methodologies are automatically harmonized.
2.11
Selection of projects for evaluation: In the Private GPS, the CED chooses the set of projects for
which self evaluations will be prepared – either the entire population of projects ready for
evaluation, or a representative sample. All of the self evaluations are then subjected to
independent validations or more in-depth independent evaluations. In the Public GPS, self
evaluations are prepared for the entire population of completed projects, and the CED may
choose to independently evaluate either 100 percent or a representative sample. Since the
sampling methods are similar in the two GPSs, the end result would be similar selection and
coverage for independent evaluations.
2.12
Timing of evaluations: Both GPSs state the principle that evaluations should be timed so that
outcomes are observable. The Private GPS defines a specific condition for determining when
self evaluations should be conducted: when the project reaches “early operating maturity”.
Thus, there is specific guidance on when business outcomes are observable, but no equivalent
guidance on when development, policy, or transition outcomes are observable. The Public GPS
calls for self evaluations to be prepared within a fixed period of time determined by the IFI,
typically six to twelve months after project completion. This timeframe may or may not
correspond to the length of time needed to observe desired outcomes. However, independent
evaluations have no such established timeframe.
2.13
Both GPSs have their strengths and weaknesses. The Private GPS is stronger in terms of explicit
definitions of performance indicators, benchmarks, and ratings guidelines, at least for the
project’s commercial objectives; on the attention given to environmental and social safeguards;
on the criteria for deciding when to conduct and evaluation (although this defined on the basis
of commercial indicators rather than development, policy, or transition indicators); and on the
application of evaluation principles to different types of projects. The Public GPS is stronger in
its coverage of project evaluability; its attention to the clarity of project objectives and results
frameworks; and in its delineation of methodologies to establish the attribution of observed
outcomes to the project.
-8-
2.14
The sections below propose ways to achieve greater consistency in how public sector and
private sector operations are evaluated, and in how evaluation results are reported. The next
section focuses on a common reporting framework (called Option 1), and the subsequent
section on a common evaluation methodology (Option 2). A comprehensive, harmonized GPS
would also include sections on evaluability (evaluation-related guidelines for project design at
entry, including the definition of what the project aims to achieve, the market failures it intends
to address, beneficiaries and target groups, the logic connecting project activities with outputs
and outcomes, and performance indicators and targets); evaluation processes (the coverage
and selection of projects for evaluation, evaluation products, and the timing of evaluations); and
reporting, disclosure, and learning. These topics are not covered in this paper because there is
less divergence between the Public GPS and the Private GPS in these areas.
3.
Option 1: A Common Reporting Framework
3.1
This section proposes a common reporting framework by mapping the evaluation criteria in the
existing Public GPS and Private GPS into the standard OECD-DAC evaluation criteria: Relevance,
Effectiveness, Efficiency, Sustainability, and Impact. The aim is to be able to compare ratings
for public sector and private sector operations, both for the individual criteria and for
aggregations of criteria. The section concludes with an application of the Option 1 approach to
two projects, one private sector and one public sector.
3.2
The mapping approach under Option 1 has several advantages:

Since the mapping framework sits on top of the existing criteria, it would minimize the
need to revise the existing GPSs and could be implemented relatively easily by ECG
members. Nevertheless, there would be a need for some changes in both GPSs, since
some now-optional criteria would become mandatory.

Future GPSs in other areas (e.g., the evaluation of technical assistance) could similarly
be mapped to the OECD-DAC criteria.

The scheme could be adapted to hybrid projects with both public and private sector
characteristics by applying either the Public GPS or the Private GPS as appropriate for
each criterion under the common OECD-DAC framework.
The approach also has some shortcomings which are described in the introduction to Option 2.
Mapping the Existing GPS Criteria to OECD-DAC Criteria
3.3
Table 3 and Figure 1 show that most of the Public GPS criteria map relatively easily to OECD-DAC
criteria, which stands to reason because OECD-DAC criteria were the starting point for the Public
GPS. The Public GPS criteria of Relevance, Effectiveness, Efficiency, and Sustainability map to
the same OECD-DAC criteria.6 The optional criteria of Achievement of Corporate Goals and
6
The Public GPS does not include an “Impact” criterion. OECD-DAC’s definition of Impact is “positive and
negative, primary and secondary long-term effects produced by a development intervention, directly or
indirectly, intended or unintended”. Most of the elements of this definition (effects that are positive and
-9-
Unintended Outcomes could be mapped to the OECD-DAC criterion of Impact. An overall
project performance rating could be calculated from the ratings on Relevance, Effectiveness,
Efficiency, Sustainability, and Impact. IFI Performance and Borrower Performance would not
figure into the overall rating.
3.4
The mapping of Private GPS criteria into OECD-DAC criteria is less straightforward. Some of the
Private GPS criteria can be mapped directly, but for others, only a component of the criterion
maps to OECD-DAC. The mapping of the Private GPS criteria shown in Table 3 and Figure 1
reflects the following reasoning:

Relevance is captured by the Contribution to IFI Mandate Objectives.

Effectiveness considers the extent to which the client achieved the project’s business
and process objectives (which are measured mainly in terms of outputs) and forecasted
financial performance targets (which can be considered as outcomes). The assessment
considers both internal (to the client) and external factors influencing the achievement
of business objectives, and the extent to which the project/company’s operational
performance met expectations. This is captured by one of the component parts of the
Private GPS’s indicator, Financial Performance and Fulfillment of Project Business
Objectives – namely, the Fulfillment of Project Business Objectives.

Efficiency considers the efficiency of the project from the perspective of financial
stakeholders in the company. The efficiency with which the client company deploys
project funds to meet objectives is captured in the profitability of the operation, since
this is a measure of the extent to which the client utilizes project assets to maximize
revenues and minimize costs. Thus, the most appropriate Private GPS criterion to
measure the efficiency of the company/project is one component of Financial
Performance and Fulfillment of Project Business Objectives, namely Financial
Performance, as measured by the FRR or ROIC or other profitability indicators as
appropriate to the project type.

Impact considers the observed and expected effects of the project on a wider group of
stakeholders (beyond its financial stakeholders), including customers, suppliers,
employees, neighbors, competitors, and taxpayers. The applicable Private GPS criteria
are Economic Sustainability and the optional supplementary sub-criterion of
Environmental and Social Performance called Environmental and Social Impact. The
latter assesses the extent of improvement (or deterioration) in the client’s
environmental and social performance since project approval.
negative, primary and secondary, direct and indirect, and intended) are picked up in the Public GPS’s
Effectiveness criterion. The realization of long-term effects depends on the timing of the evaluation, so
these are not separated into a different criterion. Unintended outcomes are assessed under an optional
criterion but are not included in the overall project performance rating, due to the Public GPS’s reliance
on an objectives-based approach. Similarly, the Public GPS allows for the use of additional criteria to
assess the achievement of corporate goals (to the extent that they are not expressed in the project’s
statement of objectives), but their ratings are not included in the overall project performance rating.
- 10 -

Sustainability takes into account both the project/company’s continued commercial
viability and prospects for growth (since other project benefits rely on the continued
existence of the company in a competitive market), and the company’s compliance with
environmental and social performance standards. Thus, the Private GPS criteria that are
mapped into Sustainability are part of the Fulfillment of Business Objectives and
Financial Performance (namely the part that relates to the client company’s prospects
for sustainability and growth), and Environmental and Social Performance.
3.5
A summary rating for overall project performance for private sector operations could be
calculated from the ratings on Contribution to IFI Mandate Objectives (which maps to
Relevance); Fulfillment of Project Business Objectives (Effectiveness); Financial Performance
(Efficiency); Economic Sustainability and Environmental and Social Performance (Impact); part of
Fulfillment of Business Objectives and Financial Performance and Environmental and Social
Performance (Sustainability). IFI Additionality, IFI Investment Performance, and IFI Work
Quality/Bank Handling would not figure into the overall rating.
3.6
To illustrate how Option 1 would work in practice, below are two applications of the approach:
one to a private sector operation and one to a public sector operation, both in the
telecommunications sector. The projects are:
3.7

an IFC telecommunications project to finance the expansion and upgrade of a
company’s GSM cellular telephone network.

a World Bank telecommunications project to improve delivery of communications
services to the Government and people of Afghanistan.
For the private sector project, information is taken from the IFC’s XPSR (equivalent to the XASR)
and IEG’s Evaluation Note (XASR-A). For the public sector project, the sources are the WB’s
Implementation Completion and Results Report (ICRR, equivalent to the CR) and IEG’s ICRR
Review (CR validation). The projects are rated using the ratings systems in the Private GPS and
Public GPS respectively.
- 11 -
Table 3: Mapping the Existing GPS Criteria to OECD-DAC Criteria
(existing criteria shown in bold)
OECD-DAC Criteria
Relevance
OECD-DAC Definition1
The extent to which the objectives of a
development intervention are
consistent with beneficiaries’
requirements, country needs, global
priorities, and partners’ and donors’
policies. Note: Retrospectively, the
question of relevance often becomes a
question as to whether the objectives
of an intervention or its design are still
appropriate given changed
circumstances.
Effectiveness
The extent to which the development
intervention’s objectives were
achieved, or are expected to be
achieved, taking into account their
relative importance.
Efficiency
A measure of how economically
resources/inputs (funds, expertise,
time, etc.) are converted to results.
Public GPS Criteria
Relevance considers both the relevance of
objectives and the relevance of design.
Relevance of objectives: assessed against
beneficiary needs, the country’s development
or policy priorities and strategy, and the IFI’s
assistance strategy and corporate goals.
Projects dealing with global public goods also
assess relevance against global priorities.
Relevance of design: assesses the extent to
which project design adopted the appropriate
solutions to the identified problems. It is an
assessment of the internal logic of the
operation (the results chain) and the validity of
underlying assumptions.
Effectiveness: the extent to which the project
achieved (or is expected to achieve) its stated
objectives, taking into account their relative
importance. The assessment uses appropriate
methods to determine the contribution of the
project to intended outcomes in a causal
manner.
Efficiency: The extent to which the project has
converted its resources economically into
results.
The assessment uses both cost-benefit analysis
(if feasible and practical) and costeffectiveness analysis (in all cases).
- 12 -
Private GPS Criteria
Contribution to IFI Mandate Objectives: the
project’s contribution to the IFI’s mandate
objectives, be they to stimulate development
of the private sector, develop efficient
financial/capital markets, or transition to a
market economy. Includes both positive and
negative contributions.
Fulfillment of Project Business Objectives
(part of the Financial Performance and
Fulfillment of Project Business Objectives
criterion): the extent to which the project
delivered on the process and business
objectives stated at approval and achieved
operational performance targets.
Financial Performance (part of the Financial
Performance and Fulfillment of Project
Business Objectives criterion): the
incremental effect of the project on the
company, including all financial stakeholders in
the project and/or company. The incremental
effect of the project is assessed on a with-vs.without project basis, or a before-vs.-after
OECD-DAC Criteria
Impact
OECD-DAC Definition1
Positive and negative, primary and
secondary long-term effects produced
by a development intervention, directly
or indirectly, intended or unintended.
Public GPS Criteria
Cost-benefit: the extent to which the benefits
of the project (achieved or expected to be
achieved) exceed project costs. A form of a
cost-benefit calculation is an ERR.
Cost-effectiveness: the extent to which the
project achieved its benefits at least cost,
compared to alternative ways of achieving the
same results.
Achievement of Corporate Goals: currently an
optional criterion in the Public GPS to assess
the project’s contributions to broad corporate
goals such as poverty reduction, social
cohesion, rural development, institutional
development, etc. that are not included in the
project’s statement of objectives.
Unintended Outcomes: currently an optional
criterion in the Public GPS to cover positive or
negative impacts of the project that were not
included in the project’s statement of
objectives.
Sustainability
The continuation of benefits from a
Sustainability: the risk that changes may occur
- 13 -
Private GPS Criteria
project basis.
Impact considers both the project’s Economic
Sustainability (effects on wider stakeholders)
and its Environmental and Social Impact.
Economic Sustainability: The project and/or
company’s contribution to growth in the
economy, assessed on a with-vs.-without or
before-vs.-after basis. The effect of the project
on all key economic stakeholders – including
and beyond the company’s owners and
financiers. Quantitative methods include ERR
and EROIC.
Environmental and Social Impact: The extent
of environmental and social change,
considering both the ex ante and ex post
conditions of the project compared with the
IFI’s requirements at approval and, therefore,
the extent of progress or regress in the
project’s environmental and social
performance. This is an optional
supplementary criterion under Environmental
and Social Performance, which is otherwise
concerned with compliance with safeguards.
Sustainability considers both the
OECD-DAC Criteria
OECD-DAC Definition1
development intervention after major
development assistance has been
completed. The probability of longterm benefits. The resilience to risk of
the net benefit flows over time.
Public GPS Criteria
that are detrimental to the continued benefits
associated with the achievement or expected
achievement of the project’s objectives, and
the impact on that stream of benefits if some
or all of these changes were to materialize.
Both the probability and the likely impact of
various threats to outcomes are assessed.
Overall Project
Performance
Not prescribed in OECD-DAC.
Currently calculated from the ratings on
Relevance, Effectiveness, Efficiency, and
Sustainability.
Could be calculated from the ratings on
Relevance; Effectiveness; Efficiency;
Sustainability; and Achievement of Corporate
Goals and Unintended Outcomes (Impact).
Private GPS Criteria
project/company’s continued viability and its
compliance with environmental and social
safeguards. The former is reflected in a
component of the Fulfillment of Business
Objectives and Financial Performance
criterion, namely the client company’s
prospects for sustainability and growth. Thus,
the relevant indicators are part of Fulfillment
of Business Objectives and Financial
Performance and Environmental and Social
Performance.
Currently calculated from the ratings on
Financial Performance and the Fulfillment of
Business Objectives, Economic Sustainability,
Environmental and Social Performance, and
Contribution to IFI Mandate Objectives.
Could be calculated from the ratings on
Contribution to IFI Mandate Objectives
(Relevance); Fulfillment of Project Business
Objectives (Effectiveness); Financial
Performance (Efficiency); the forward-looking
part of Fulfillment of Project Business
Objectives and Financial Performance, and
Environmental and Social Performance
(Sustainability); and Economic Sustainability
and Environmental and Social Impact
(Impact).
1
Organization for Economic Cooperation and Development (2002), Glossary of Key Terms in Evaluation and Results Based Management. Paris. The definitions
of the five criteria are slightly different from those shown in another source on the DAC website, DAC Criteria for Evaluating Development Assistance, which is
based on OECD (1991), The DAC Principles for the Evaluation of Development Assistance; OECD (1986), Glossary of Terms Used in Evaluation, in ‘Methods and
Procedures in Aid Evaluation’; and OECD (2000), Glossary of Evaluation and Results Based Management (RBM) Terms. The definitions in the 2002 Glossary of
Key Terms in Evaluation and Results Based Management are used in this paper because the source publication is more recent and because it is also used for
definitions of other evaluation terms.
- 14 -
Figure 1: Option 1 Criteria and Sub-Criteria
Private GPS Criteria
Public GPS Criteria
OECD-DAC Criteria
Relevance
Relevance
Contribution to IFI Mandate
Objectives
Effectiveness
Effectiveness
Fulfillment of Project
Business Objectives
Efficiency
Efficiency
Financial Performance
Economic Sustainability
Achievement of Corporate
Goals (opt.)
Impact
Environmental and Social Impact
(opt. suppl. criterion)
Unintended Outcomes (opt.)
Sustainability
(the forward-looking part of) Fulfillment
of Project Business Objectives and
Financial Performance
Sustainability
Environmental and Social Performance
Overall Project Performance
Rating
IFI Additionality
IFI Performance
IFI Investment Performance
Borrower Performance
IFI Work Quality/Bank Handling
Private Sector Application: IFC Telecommunications Project
1. PROJECT INFORMATION
The Project consisted of financing the expansion and upgrade of a Company’s GSM cellular telephone
network, at an estimated cost of USD 1.95 billion. It included: (i) the funding of the Company's network
expansion with associated working capital requirement; and (ii) the refinancing of USD 1.1 billion
equivalent of short-term debt. The Company had previously been awarded one of two cellular licenses in
the country, aimed at increasing competition and private sector participation in mobile telephony.
2. RELEVANCE
Rating under Private GPS
Contribution to IFI Mandate Objectives:
Satisfactory
Rating under OECD-DAC Framework
Relevance: Satisfactory
The IFI’s global telecommunications sector strategy was to: (i) foster new private investment and
competition in order to improve access to basic communication services; (ii) increase the availability of
advanced technologies; and (iii) support the development of downstream industries. The Project was
expected to fit well with these objectives. Moreover, the results reinforce this alignment and justify the
Satisfactory rating as follows:
i) Objective: To increase the public's access to affordable mobile services and quality of service through
increased private investment and competition in the country. Result: The Project enabled the Company
to invest in its 3G network in turn contributing to the overall enhancement of technology used by mobile
- 15 -
subscribers in the country. The percentage of 3G network users reached 48% of total mobile subscribers
by 2011, which is significantly higher than the EU average of 30%.
ii) Objective: To provide the recently established Company much needed long term capital, to complete
its investment program and allow it to emerge as a legitimate competitor against the two large incumbent
mobile operators. Result: It enabled the Company to resist the strong competition from the dominant
operators and increase its market share, albeit by less than hoped for, from 15% to 20%.
iii) Objective: To support broader economic growth through increased availability of advanced
technologies and introduction of new value added services such as mobile banking. Result: The Company
established partnerships with more than 75 major retailers and banks, it developed new service
applications, and won the ‘Best Product Award’ for its Credit-Card-In-SIM application.
3. EFFECTIVENESS
Rating under Private GPS
Fulfillment of Business Objectives: Partly
Unsatisfactory
Rating under OECD-DAC Framework
Effectiveness: Partly Unsatisfactory
In respect of the Project’s business objectives: (i) The Company invested heavily in its 3G network. It
exceeded forecast annual capital expenditure in order to accelerate its roll-out of services, investing on
average USD 310 million per annum since 2009. The funding also helped the Company to establish a
research & development department, which invested more than USD 65 million over five years and has
provided innovative solutions to its business partners. (ii) The Company successfully refinanced USD 1.1
billion of short-term debt, replacing it with long-term finance more suited to the tenor of its network
assets.
At the time of board approval, it was estimated that the Company would achieve a market share of 23%
by 2010. Despite the higher than forecasted increase in mobile penetration and subscribers in the
country, its market share had only reached 20% by the end of 2011, due mainly to intense competition
from the incumbent operators. Although outputs of the project were achieved, one of the principal
outcome objectives (market share) was not. Therefore, effectiveness is rated Partly Unsatisfactory.
4. EFFICIENCY
Rating under Private GPS
Financial Performance: Unsatisfactory
Rating under OECD-DAC Framework
Efficiency: Unsatisfactory
The financial performance of the Company, measured in FRR terms, was Unsatisfactory. Lower than
expected market share resulted in lower revenue growth, impeding the financial returns for the Company.
Recorded revenues of USD 1,251 million in 2011, in real terms, were 16% lower than forecasts. The
Company’s EBITDA margin was expected to increase gradually from 17% in 2007 to 34% in 2011, getting
closer to the margin level of the dominant operator. However, in order to increase its market share,
marketing and promotional expenses were much higher than planned on a per-subscriber basis. This had
an adverse impact on the EBITDA margin, which averaged only 13% from 2007 to 2011.
A combination of lower revenues, tighter margins, and the accelerated capital program, resulted in
negative free cash flow for the Company. To reduce interest costs, USD 2.3 billion of shareholder loans
were converted into equity. From the perspective of financial stakeholders in the Company, returns to
shareholders have therefore been negative, whilst the financial performance has increased the risk for
senior lenders. These results are consistent with an Unsatisfactory rating.
- 16 -
5. SUSTAINABILITY
Rating under Private GPS
Future Business Prospects: Satisfactory
Env. and Social Performance: Partly
Unsatisfactory
Rating under OECD-DAC Framework
Sustainability: Partly Unsatisfactory
The Company’s management is now focusing more on profitable and sustainable growth. It is expanding
data services and aiming to increase its post-paid subscriber base (higher Average Revenue Per User ARPU). The percentage of post-paid customers increased from 37% in Q1 2010 to 44% in Q1 2012, and
the Company has now achieved the highest post-paid customer base among mobile operators in the
country. Furthermore, the share of data revenues in total revenues increased from 4.9% in 2009, to
13.4% in 2011. The Company continues to receive strong support from its shareholders, and with the
balance sheet now restructured and de-leveraged, has a much more robust basis for future operations.
Finally, the regulatory environment has also improved, with implementation of pro-market competition
rules such as network sharing, number portability, asymmetric termination tariffs, and unsubsidized onnet tariffs. Business prospects going forward are therefore rated Satisfactory.
The Project, originally appraised in 2004, was classified as Category B for its E&S risk. The Company
started implementation of a Safety Health and Environment (SHE) Management System. Although, it
committed to obtain ISO 9001 and ISO 14001 certificates by the end of 2007 and OHSAS 18001
certification by the end of 2008, these are still outstanding. During a 2010 site visit, the Company
reported that there was an ongoing updating process for adaptation of SHE, which will be applicable to
the mobile sector. However, by 2012, this adaptation process had not been finalized. The lack of a fully
developed and formalized SHE management system and centralized EHS monitoring mechanism and
organization results in a Partly Unsatisfactory rating for Performance Standard (PS) 1. The project
performance in regard to PS2, PS3, and PS4 has been Satisfactory at appraisal and evaluation, but most of
the Company’s mobile base stations have been installed without the required permissions and are not in
compliance with PS5 and regulatory requirements. Overall, Environmental and Social Performance is
rated Partly Unsatisfactory.
6. IMPACT
Rating under Private GPS
Environmental and Social Impact: No Change
Economic Sustainability: Satisfactory
Rating under OECD-DAC Framework
Impact: Satisfactory
Deficiencies in the Company’s SHE Management System were identified at approval and requirements for
these to be addressed were incorporated into the Project agreements. Since the Company has yet to
meet these requirements in full and deficiencies still exist, the rating of environmental and social impact is
No Change.
The Project's ERR is estimated at 17%. Quantified net benefits to society in NPV terms are approximately
USD 721 million, comprising largely of tax and fee payments to the Government (roughly equivalent to
46% of the Company’s total revenues). Although the Project has not performed well for the Company’s
financial stakeholders, it has had a number of Satisfactory impacts on the market, on related businesses,
and for customers, as follows:

Consumers: The Company increased its subscribers from 7.5 million to 12.8 million, and has
offered more affordable services than its competitors. Consumers also benefited from the high
quality of service offered by the Company as it increased its geographical/population coverage.
- 17 -

Market: As the third-biggest player, the Company has had some positive impact on the
competitiveness of the market. Prices have decreased dramatically, with ARPU across the
country falling from approximately USD$ 19 in 2004 to USD$ 11 in 2011. At the same time, the
quality of mobile services increased as evidenced by lower unsuccessful call and blocked call
rates, and increased customer relations activities.

Employment: By 2012, the Company had 2,790 employees compared to 1,833 employees at
approval. It established eight outsourced call centers in frontier regions of the country, boosting
local employment in these areas. Direct and outsourced call center employees are provided with
special training. Through its franchised dealers, the Company has increased indirect employment
from 514 to 3,656.
7. OVERALL PROJECT PERFORMANCE
Project Development Outcome, based on
Private GPS:
Mostly Unsuccessful
Overall Project Performance Rating,
based on OECD-DAC Framework:
Mostly Unsuccessful
Public Sector Application: World Bank Telecommunications Project
1. PROJECT INFORMATION
The project aimed to improve delivery of communications services to the Government and to the people
of Afghanistan, via three components:
(i) A USD 14 million component to support the supply, installation, and commissioning of a
communications network as well as improved and extended coverage to Ministries and other key
Government organizations in Kabul as well as Provincial Capitals and selected other sites, and
operations and maintenance and training of Afghan Tel staff.
(ii) A USD 7 million component to provide consulting services, training, technical advisory services
and equipment to (a) separate MOC’s policy, regulation, operational and ownership functions,
(b) strengthen the MOC’s policy and project management capacity, (c) further establish and “bed
down” independent regulatory capacity, (d) establish the monitoring equipment required to
properly administer the MOC’s spectrum plan, and (e) establish Afghan Tel as an efficient State
owned corporation.
(iii) A USD 0.5 million component to support consulting services and equipment for the
separation of the MOC’s policy and operational functions in the postal sector, and to help
develop the sector.
- 18 -
2. RELEVANCE
Rating under Public GPS
Relevance: Substantial
Rating under OECD-DAC Framework
Relevance: Substantial
Relevance of objectives: ICT remains a priority in the Afghanistan National Development Strategy (ANDS)
and an important priority according to the most recent Bank Interim Strategy Note. This includes broad
based work to improve infrastructure services in most areas, including telecommunications, to address
overall business constraints. The objectives related to postal services are not mentioned as priorities of
the Strategy Note, but they clearly require improvement in Afghanistan. For example, the average
number of letter-post items posted per inhabitant in 2007 was only 0.24, significantly below the average
for other neighboring countries with similar income per capita level such as Bangladesh (0.79 in 2007),
Nepal (2.56 in 2008), and Pakistan (2.03 in 2008).
Relevance of design: Project design was based on solid background analysis during a one-year
engagement with the Ministry and other donors in the telecommunications sector. However, there were
weaknesses in the results framework, which did not allow proper monitoring of outcomes. Although
design was kept relatively simple, in line with the conflict-affected situation in Afghanistan, the addition of
the postal sector was inappropriate. It added to complexity while at the same time investing inadequate
resources to achieve the goal of improving services.
3. EFFECTIVENESS
Rating under Public GPS
Effectiveness: Substantial
Rating under OECD-DAC Framework
Effectiveness: Substantial
The delivery of communications services to the government has been significantly improved and all
targets regarding the Government communication network (GCN) were met. The GCN is fully functional
(voice, data, and video) and connects all 34 provincial capitals and 40 line ministries/agencies in Kabul.
Telephone connectivity of Government has reached over 70 thousand lines — ten times over the original
target of 6,550 lines.
Regarding the delivery of communications services to the people of Afghanistan: Teledensity has grown
from 0.2% in 2002 to about 35% by 2009 and total number of subscriptions is about 11 million. Service
providers increased from two in 2002 to seven plus seven operational Internet service providers (ISPs).
Mobile signal coverage has reached over 70 percent of the population. Tariffs have fallen to about 10 US
cents a minute from US$2 per minute in 2002. Investments in the sector are above US$1 billion and
surpassed the target of US$950 million. These achievements cannot be attributed solely to the project,
and it is hard to determine the relative contribution of the project in the context of rapid ICT sector
growth worldwide, along with complementary support from other Bank projects, other donors and
technical assistance providers.
More modest results were achieved regarding postal services. Although new services such as express
mail have been introduced, the Postal Commission has been instituted and the Postal Law has been
drafted, indicators of postal performance remain significantly low.
- 19 -
4. EFFICIENCY
Rating under Public GPS
Efficiency: Modest
Rating under OECD-DAC Framework
Efficiency: Modest
As the project was prepared as an emergency credit, no economic and financial analysis was conducted
during preparation. Nor were economic or financial calculations made at evaluation, nor was costeffectiveness analysis carried out. At least an approximate estimate of the economic benefits could have
been made based on the opportunity revenues generated from the government network if the lines/or
the services were provided to private customers rather than to the government. Furthermore, financial
calculation should have been made based on the revenues from the government network paid to Afghan
Tel by the government, especially given the significant increase in cost of the GCN component (US$17
million compared with US$ 14 million at appraisal).
Research suggests that every 10 percentage-point increase in mobile telephony penetration in developing
countries leads to an acceleration of GDP growth by 0.75 percentage point. These calculations, however,
should be considered with caution as such extrapolations and estimates may well suffer from
econometric estimation problems. Some indirect benefits to which the project made a partial
contribution are: the increase in affordability of telephone services (tariffs dropped from US$2 per minute
of talk-time to about 10 US cents per minute over the period of the project), the social impact due to the
growth of the telecommunications sector (creation of 60,000 jobs directly or indirectly) and the favorable
fiscal impact. However, these achievements cannot be attributed solely to the project.
5. IMPACT
Rating under Public GPS
Achievement of Corporate Goals: Not Rated
Unintended Outcomes: Not Rated
Rating under OECD-DAC Framework
Impact: Not Rated
The project’s broader impacts on corporate goals such as poverty reduction, social cohesion, rural
development and institutional development were not assessed. There were no unintended outcomes.
6. SUSTAINABILITY
Rating under Public GPS
Sustainability: Moderately Likely
Rating under OECD-DAC Framework
Sustainability: Moderately Likely
The government infrastructure will be maintained, and the reforms undertaken and the degree of
competition achieved in the sector are unlikely to be reversed. However, the volatile and deteriorating
security situation in Afghanistan poses moderate risks to development outcome.
- 20 -
7. OVERALL PROJECT PERFORMANCE
Overall Project Performance Rating, based on
Public GPS:
Moderately Satisfactory
Overall Project Performance Rating,
based on OECD-DAC Framework:
Moderately Satisfactory
The operation substantially achieved most of its telecommunications objectives, and relevance of
objectives and design was also substantial. Achievements in the postal segment, however, were modest,
although this was a small part of the project. Limited evidence was presented to demonstrate efficiency
and attribution of benefits to the project. Outcome is rated as moderately satisfactory.
4.
3.8
Option 2: A Common Evaluation Methodology
The Option 1 approach goes part of the way toward the harmonization objective, but there are
still some differences between the public sector and private sector criteria and methodologies
that create inconsistencies and make comparisons difficult.

For private sector operations, the achievement of development, policy, or transition
objectives that are specific to the project is not considered in the Effectiveness rating.
Instead, the Effectiveness rating is based only on the achievement of the project’s
business and process objectives. In contrast, for public sector operations, Effectiveness
assesses the achievement of the project’s specific development, policy, or transition
objectives.

Attribution of outcomes to the project differs between the two GPSs. For the
Effectiveness criterion, the Public GPS requires the comparison of observed outcomes to
a counterfactual, either using an impact evaluation or a theory-based method. In
contrast, the Private GPS generally uses before-and-after comparisons to describe the
results of the project, and does not have an explicit methodology for establishing a
counterfactual.

The project’s Economic Rate of Return (ERR) is an indicator of Impact for private sector
operations but an indicator of Efficiency for public sector operations.

For private sector operations, the Efficiency criterion reflects only the financial rate of
return to the client, rather than the broader economic benefits and costs of the project
that are measured for public sector operations.

Compliance with environmental and social safeguards is the main factor in Sustainability
for private sector operations, but for public sector operations it enters into IFI and
Borrower Performance.

The mapping of existing criteria to the OECD-DAC criterion of Impact is straightforward
for the Private GPS criteria but not for the Public GPS. The latter incorporates most of
the elements of the OECD-DAC Impact criterion into the Effectiveness criterion. The
other elements of the OECD-DAC Impact criterion, reflected in Achievement of
Corporate Goals and Unintended Outcomes, are optional.
- 21 -

Contribution to IFI Mandate Objectives is mapped to Relevance for private sector
operations. Since this criterion includes both planned and actual contributions, it is not
equivalent to the Relevance criterion for public sector operations, which only assesses
the ex-ante relevance of project objectives to IFI mandate objectives.

IFI Additionality is assessed below the line for private sector operations, but would be
assessed above the line (under Relevance) for public sector operations.
4.1
Option 2 aims to go further than the mapping of existing GPS criteria into a common reporting
framework, to the definition of an evaluation methodology that could be applied to both public
sector and private sector operations. Option 2 attempts to solve the problems described above,
as well as to draw on the strengths and address the weaknesses of both GPSs.
4.2
The proposed methodology attempts to:
4.3

be relevant to the institutional structures and mandates of the wide variety of IFIs in
ECG.

be applicable to both public sector and private sector operations, allowing but
minimizing the use of modules specific to private sector or public sector operations.

allow comparisons of ratings of individual criteria across public sector and private sector
operations (e.g., Relevance to Relevance, Sustainability to Sustainability) as well as
comparisons of the overall project performance rating.

minimize overlaps among evaluation criteria.

sharpen the focus of evaluations on the project’s specific intended results, rather than
requiring projects to be evaluated against a common set of outcomes that may not all
be relevant to the project.

separate the results that are easily attributable to the project without comparison with
a counterfactual, from those that may be influenced by external factors and thus
require comparison with a counterfactual.

clarify how attribution should be taken into account in the assignment of ratings.
Each section below begins with the highlights of the proposed standards and how they differ
from the current GPSs, and suggests other options that might be considered. Both the text and
tables are formatted in landscape to allow for easier reading.
- 22 -
The Basis of the Evaluation
4.4
Highlights: The evaluation approach begins with an understanding of the basis of the evaluation (Table 4). Both public and private sector
evaluations are based on the specific, intended results of the project, either as drawn from project design and/or approval documents, or as
reconstructed by the evaluator. This is the current standard in the Public GPS, but for private sector operations it internalizes the project’s
private sector development, financial sector development, transition, or regional economic impact objectives into a set of intended results that
is specific to the project. Similarly, intentions to reach particular populations, or to reduce or mitigate market failures and externalities, would
be brought into the project’s intended outcomes.
4.5
Other options: Maintain the approach of the Private GPS, which evaluates all projects against common objectives under the Contribution to IFI
Mandate Objectives criterion (e.g., Private Sector Development in the case of IFC and Transition Impact in the case of EBRD). However, this
would be harder to do for public sector operations since the IFI mandate objectives for public sector operations are often quite broad (e.g.,
poverty reduction in the case of WB).
Table 4: The Basis of the Evaluation
Evaluation Principles and Practices
Projects are evaluated primarily against the outcomes that the
project intended to achieve, as contained in the project’s design
and/or approval documents.
Projects that are intended to reach specific populations, or to
address market failures or externalities, also are evaluated
against these intended outcomes.
If intended project results are unclear, or if results are expressed
solely in terms of outputs, the evaluator retrospectively
constructs an outcome-oriented set of intended results based
on the expected benefits and beneficiaries of the project,
project components, performance indicators, and/or other
elements of project design.
If intended results were revised during implementation, the
project is assessed against both the original and the revised
intended results.
Application to:
Public
Private
Sector
Sector
X
X
X
X
X
X
X
X
Comments
Same as Public GPS OP 12.1. Private sector operations are
evaluated against both their intended development, policy, or
transition outcomes, and their intended business outcomes.
A clarifying addition to both the Public and Private GPS.
Similar to Public GPS 12.3 – 12.5. In some IFIs, private sector
operations are less likely than public sector operations to have a
formal statement of objectives. It will therefore be necessary for
the evaluator to construct set of intended results as well as the
results chain linking them with project activities and outputs.
Same as Public GPS EP 13.
- 23 -
Evaluation Principles and Practices
Evaluations also may assess the contribution of the project to
broad corporate goals to the extent they are not expressed in
the project’s intended results.
Evaluations also may assess the project’s contribution to
unintended outcomes, both positive and negative.
Application to:
Public
Private
Sector
Sector
X
X
X
X
Comments
This goes beyond the objectives-based methodology of the Public
GPS to include the project’s contribution to non-project-specific
corporate goals. Depending on the IFI, these may include poverty
reduction, rural poverty reduction, social cohesion, institutional
development, EU policy, environmental sustainability, inclusive
growth, etc.
Goes beyond the Public GPS, which calls for unintended outcomes
to be discussed and documented but not rated. Thus, an element
of OECD-DAC “Impact” is added to the evaluation of public sector
operations.
Evaluation Criteria
4.6
Highlights: Harmonized evaluation criteria apply to both public sector and private sector operations, but some of the criteria have sub-criteria
that are applicable only to public sector or private sector operations. The structure is shown in Figure 2 and Table 5.

The Relevance criterion assesses the relevance of intended project results against country needs and priorities, IFI country strategies,
and IFI corporate goals (Strategic Alignment) as well as the ex-ante justification for IFI involvement (Rationale for IFI Involvement). For
operations that could potentially compete with the private sector (which may be the case for both public and private sector operations),
the Relevance assessment assesses the rationale for public sector intervention. Where applicable, it assesses the evidence behind
claims of market failures that justify the intervention, and the rationale for targeting specific populations. The Rationale for IFI
Involvement also discusses the intended value added of the IFI vis-à-vis other donors and/or private investors or lenders.

The Results criterion comprises four sub-criteria: Operational Performance, Contribution to Intended Outcomes, Contribution to
Corporate Goals, and Unintended Outcomes. For the Public GPS, this adds some of the elements of OECD-DAC Impact criterion and
some elements of the Private GPS’s Economic and Social Performance criterion to the existing Effectiveness criterion.

The Operational Performance sub-criterion assesses results that are under the control of the project (outputs and the fulfillment of
project business objectives) and thus easily attributable to it. The other Results sub-criteria (Contribution to Intended Outcomes,
- 24 -
Contribution to Corporate Goals, and Unintended Outcomes) are likely to be influenced by external factors and thus require comparison
with a counterfactual. This is clarified by using the term “contribution” rather than “achievement”.

For private sector operations, the project’s development, policy, or transition impacts, as well as its regional economic impacts, are
covered under Contribution to Intended Outcomes.

Contribution to Corporate Goals is a criterion that accommodates the current practice of many IFIs of assessing the project’s results in
achieving broad corporate objectives – e.g., poverty reduction, social inclusion, shared prosperity, institutional development, pro-poor
innovation – to the extent these are not contained in the project’s intended results.

The rating for Results is based on the ratings for Operational Performance, Contribution to Intended Outcomes, Contribution to
Corporate Goals, and Unintended Results. The rating on Contribution to Intended Outcomes criterion would normally be given greater
weight than the ratings on the other three criteria.

The Efficient Use of Resources criterion assesses the profitability of the project to all stakeholders: the project’s financial stakeholders
(for private sector operations), society at large (for both public and private sector operations), and the IFI (for private sector operations).
It also separates Economic Efficiency from Implementation Efficiency. This is because implementation delays may affect the timing of
both benefits and costs, and may not be captured in ERRs.

Sustainability covers the sustainability of outcomes achieved or expected to be achieved as well as the project’s environmental and
social sustainability. The latter is assessed by the project’s compliance with environmental and social safeguards (“do no harm”). This is
an addition to the Public GPS, which places less emphasis on safeguards compliance than does the Private GPS. For private sector
operations, Sustainability also includes an assessment of the likelihood of continued viability of the project’s company (or, for Group D
operations, the continued viability of the financial institution/fund and sub-borrowers/fund investees). For both types of operations,
negative environmental and social effects of the project, or positive effects that go beyond the “do no harm” resulting from compliance
with safeguards, are covered under Unintended Outcomes.

For both public and private sector operations, IFI Performance covers Quality at Entry (QAE) and the Quality of Supervision. QAE
includes an assessment of the logic of the operation – a divergence from the Public GPS, which covers this aspect under the Relevance of
Design. This would eliminate the substantial overlap between the Relevance of Design and QAE that currently plagues the Public GPS.
- 25 -

4.7
Client Performance assesses the non-financial performance of the project/company for private sector operations, and the performance
of the government and implementing agency for public sector operations.
Other Options:

Under the Results criterion, for private sector operations, include a sub-criterion for Regional Economic Impact. This would mimic the
current Private GPS criterion of Economic Sustainability, which assesses the project’s impact on stakeholders external to the company
(e.g., employees, suppliers, competitors, and neighbors). The reason for not doing this is that these results should be included from the
beginning in the project’s results chain, and assessed during evaluation along with other intended outcomes.

Rather than putting Financial Performance of the Project/Company under Efficient Use of Resources, include it under Results. The
argument is that private sector operations are aimed mainly at promoting commercially-viable enterprises, a prerequisite for achieving
development, policy, and transition impacts. However, the reason for putting Financial Performance under the Efficient Use of
Resources is because it is more consistent with the concept of efficiency, i.e., the profitable use of both private and public resources. It
also preserves the concept of Results as being the “end” of the intervention, as opposed to the “means” of a viable commercial
enterprise.

Rather than including IFI Profitability under Efficient Use of Resources, put it below the line as part of IFI Performance. The argument is
that the internal efficiency of the IFI has no bearing on the project's performance from a development point of view. The reason for not
doing so is that the Efficient Use of Resources conceptually captures the efficiency of resource allocation by all of the project’s
stakeholders, including the IFI.
- 26 -
Figure 2: Option 2 Criteria and Sub-Criteria
Private Sector Sub-Criteria
Harmonized Criteria
Public Sector Sub-Criteria
Strategic Alignment
Rationale for IFI Involvement
Relevance
Strategic Alignment
Rationale for IFI Involvement
Results
Achievement of Outputs
Fulfillment of Project Business Objectives
Operational Performance
Achievement of Outputs
Contribution to Intended Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Financial Performance of the Project/Company
Economic Efficiency
Implementation Efficiency
IFI investment profitability
Efficient Use of Resources
Outcome Sustainability
Commercial Sustainability
Compliance with Safeguards
Sustainability
Economic Efficiency
Implementation Efficiency
Outcome Sustainability
Compliance with Safeguards
Overall Project Performance Rating
Quality at Entry
Quality of Supervision
Non-Financial Performance of the Company
IFI Performance
Client Performance
- 27 -
Quality at Entry
Quality of Supervision
Government Performance
Implementing Agency Performance
Table 5: Option 2 Criteria and Sub-Criteria
Criteria and Sub-Criteria
Relevance
Strategic Alignment
Rationale for IFI Involvement
Results
Operational Performance
Achievement of Outputs
Fulfillment of Project Business
Objectives
Contribution to Intended Outcomes
Definition
Application to:
Public
Private
Sector
Sector
The consistency of the project’s intended
results with beneficiary needs; the country’s
development, policy, or transition priorities
and strategy; and the IFI’s assistance strategy
and corporate goals.
X
X
X
X
X
X
The justification for public sector involvement
and the IFI’s involvement in the project.
Achievement of outputs and intermediate
outcomes that are under the control of the
project.
The extent to which the project achieved its
targeted outputs.
X
The extent to which the project delivered on
its process and business objectives.
The extent to which the project contributed,
or is expected to contribute, to its intended
development, policy, or transition results,
taking into account their relative importance.
Also considers the project’s reach to intended
beneficiaries and target groups, and the
- 28 -
X
X
X
X
Comments
Similar to Public GPS OP 17.1 and the
Relevance of Objectives. In contrast to
the Public GPS, the Relevance criterion
does not include the Relevance of
Design, which is covered instead under
Quality at Entry.
Similar to Private GPS criterion, IFI
Additionality (OP 22.1), but focuses on
what was expected at project entry
rather than what was achieved at the
time of evaluation.
A new criterion. In practice, some IFIs
separate the discussion of outputs and
outcomes in the Effectiveness
assessment.
Similar to Private GPS OP 16.1 (Financial
Performance and Fulfillment of Project
Business Objectives).
Corresponds to Effectiveness in the
Public GPS (OPs 18.1 and 19.1). For
private sector operations, this subcriterion also includes regional
economic impacts that are now covered
under Economic sustainability (Private
Criteria and Sub-Criteria
Contribution to Corporate Goals
Unintended Outcomes
Efficient Use of Resources
Financial Performance of the
Project/Company
Economic Efficiency
Implementation Efficiency
Definition
Application to:
Public
Private
Sector
Sector
project’s success in reducing or compensating
for market failures identified at entry.
The project’s contribution (or expected
contribution) to broad corporate goals that
are not included in the project-specific
intended results.
X
X
X
X
Other results caused by the project, positive
or negative, that are not covered in the above
sub-criteria.
The profitability of the project to all financial
stakeholders in the project and/or company.
The profitability of the project from a social
perspective. The extent to which the costs
involved in achieving results were reasonable
in comparison with both the project’s
benefits and with recognized norms.
Other aspects of project design and/or
- 29 -
X
X
X
X
X
Comments
GPS OP 17.1). Adds target groups and
market failures.
Includes broad corporate goals that are
not covered in the Contribution to
Intended Outcomes (e.g., poverty
reduction, gender equality, social
cohesion, inclusive growth). For private
sector operations, includes IFI Mandate
Objectives (Private GPS OP 18.1) and
the optional sub-criterion,
Environmental and Social Impact.
Differs from the treatment of
unintended (unanticipated) outcomes in
the Public GPS (OPs 14.1 – 14.3)
because in the Public GPS they do not
enter into the overall project
performance rating. Covers one aspect
of the OECD-DAC Impact criterion.
Includes positive or negative
environmental impacts caused by the
project.
Similar to Private GPS OP 16.1. May be
applicable to public sector operations
that have a commercial orientation.
Similar to Public GPS OP 20.1 and
Private GPS OP 17.1.
In the Public GPS, implementation
Criteria and Sub-Criteria
Definition
Application to:
Public
Private
Sector
Sector
implementation that either added to or
reduced costs (e.g., implementation delays).
IFI Investment Profitability
Sustainability
Outcome Sustainability
The extent to which the IFI deployed its own
resources efficiently to generate a net profit
from its investment.
X
The resilience to risk of results achieved or
expected to be achieved.
X
Commercial Sustainability
Compliance with Safeguards
IFI Performance
Quality at Entry
Prospects for the continued viability of the
company, financial institution, and/or subborrowers/fund investees.
X
X
The project’s compliance with applicable
environmental and social safeguards in the
area of influence of the project.
The IFI’s role at project entry in ensuring
quality design and implementation
arrangements.
- 30 -
X
X
X
X
Comments
delays are considered in the assessment
of Efficiency (OP 20.5). Here,
implementation efficiency is explicitly
separated from Economic Efficiency.
Same as IFI Investment Profitability in
the Private GPS (OP 20.1).
Draws on Public GPS OP 21.1 but
extends the outcomes considered to
include regional economic impact
(mainly for private sector operations),
Contribution to Corporate Goals, and
Achievement of Unintended Outcomes.
The forward-looking part of the
Financial Performance and Fulfillment
of Project Business Objectives criterion.
See Private GPS EP 16 (B) and OP 16.2
(5). May be applicable to public sector
projects with a commercial orientation.
Similar to the Private GPS criterion,
Environmental and Social Performance
(OP 19.1). Adds an explicit sub-criterion
on safeguards to Public GPS. Ensures
that consideration is given to the actual
implementation of safeguards, not just
the adoption of a safeguards system.
Similar to Public GPS OPs 22.1 – 22.7.
The quality of the results framework
appears here rather than under the
Criteria and Sub-Criteria
Quality of Supervision
Client Performance
Non-Financial Performance of the
Company
Government and Implementing
Agency Performance
Definition
Application to:
Public
Private
Sector
Sector
The extent to which the IFI proactively
identified and resolved problems during
implementation and adequately executed its
portfolio responsibilities
The company’s execution of its
responsibilities for achieving process and
business objectives and its compliance with
relevant government regulations.
The performance of the government and of
the implementing agency during all project
phases in meeting their responsibilities as
owners of the project.
X
X
X
X
Comments
Relevance of Design. Includes the part
of the IFI Work Quality/Bank Handling
criterion in Private GPS that deals with
pre-commitment work (OPs 21.1- 21.2).
Similar to Public GPS OPs 22.8 – 22.10.
Includes the part of the IFI Work
Quality/Bank Handling criterion in
Private GPS that deals with monitoring
and supervision (OPs 21.1 - 21.3).
Part of Private GPS OP 16.1 (Financial
Performance and Fulfillment of Project
Business Objectives).
See Public GPS OPs 23.1 – 23.6
(Borrower Performance, covering
Government Performance and
Implementing Agency Performance).
Evidentiary Requirements, Analytical Methods, and Benchmarks
4.8
Option 2’s evidentiary requirements, analytical methods, and benchmarks for a positive rating are shown in Table 6.
4.9
Highlights:

Compared to both the Public GPS and Private GPS, the proposed evaluation methodology has more detail on evidentiary requirements
and analytical methods to assess the achievement of development, policy, and transition objectives. It maintains the Private GPS’s
evidentiary requirements for project, company, and IFI financial performance.
- 31 -
4.10

To establish results attributable to the project, three of the four sub-criteria under Results require a comparison of observed outcomes
with a counterfactual. The fourth sub-criterion – Operational Performance – is focused on the achievement of outputs and intermediate
outcomes that are under the control of the project (for example, the physical implementation of investments in physical assets, the
creation of credit lines) and thus does not require counterfactual analysis.

The ratings guidance for the Results sub-criteria goes beyond the usual definitions that do not mention causality (e.g., in WB evaluation
guidelines: “a positive rating requires that objectives be substantially achieved”) to bring attribution explicitly into the ratings. Thus, for
example, a positive rating on Contribution to Intended Outcomes requires that there is strong evidence that the project caused the
targeted improvement in outcome indicators compared to a without-project counterfactual. If outcome indicators improved but it
cannot be shown that the improvement was due to the project, the Contribution to Intended Outcomes criterion would receive a less
than satisfactory rating. Symmetrically, if outcome indicators deteriorated but it can be demonstrated that they would have
deteriorated more in the absence of the project (meaning the project made a positive contribution compared to the counterfactual), the
Contribution to Intended Outcomes criterion would receive a positive rating. All three attribution-related sub-criteria – Contribution to
Intended Outcomes, Contribution to Corporate Goals, and Unintended Outcomes – allow for a “Not Rated” rating on an exceptional
basis. If “Not Rated” is assigned, an explanation is provided (e.g., not applicable; insufficient evidence).

A summary rating for each of the criteria (Relevance, Results, Efficient Use of Resources, Sustainability, IFI Performance, and Client
Performance) is based on the ratings for the corresponding sub-criteria, with weights determined by the evaluator. An overall project
performance rating would be based on the summary ratings for Relevance, Results, Efficient Use of Resources, and Sustainability, with
each receiving equal weight.
Other options:

Do not allow the option of a “Not Rated” rating for Contribution to Intended Outcomes. The argument is that insufficient evidence
should not be a reason for not rating, since this provides a perverse incentive to not provide any evidence if the project results are bad.
The reasons for allowing a “Not Rated” rating are that projects with poor M&E systems should not be presumed to have produced
negative results, and that there are other ways of holding IFI Management accountable for failing to provide sufficient evidence. At the
project level, the quality of the M&E system enters into IFI Performance. At the portfolio level, the CED’s reports to the Board on
portfolio performance and on the quality of the self evaluation system can include data on the proportion of projects that were not
rated under each criterion, thus holding IFI Management accountable for providing insufficient evidence.
- 32 -
Table 6: Evidentiary Requirements, Analytical Methods, and Benchmarks
Criteria/Sub-Criteria
Relevance
Strategic Alignment
Evidentiary Requirements and Analytical Methods
A comparison of the project’s intended results with the country’s development,
policy, or transition priorities and with IFI country and sector assistance
strategies and corporate goals as expressed in Poverty Reduction Strategy
Papers, country strategies, sector strategies, and operational guidelines. For
IFIs that do not prepare country strategies, consistency with other analyses of
country conditions and needs. Also considers the clarity and realism of the
project’s intended results.
Benchmarks
Substantial clarity and realism of project
objectives and consistency with needs,
policies, and priorities are required for a
positive rating.
Private sector operations: Same as above. Also considers the implementation
of the IFI’s screening mechanisms at the pre-commitment stage.
Rationale for IFI Involvement
Strategic Alignment is assessed against priorities and conditions at the time of
project entry.
For operations that could potentially compete with the private sector (which
may be the case for both public and private sector operations), the assessment
requires evidence of the market failures that justify the intervention. Where
applicable, it also discusses the rationale for targeting specific populations. If
the rationale for intervention is based on social goals (such as redistribution),
these are explained.
Public sector operations: Also considers the comparative advantage of the IFI
vis-à-vis other providers (national and sub-national governments, other donors,
NGOs).
Private sector operations: Assesses the rationale for IFI intervention as opposed
to the client’s own resources or other external sources of support. Considers (i)
financial additionality (whether the client would have been able to obtain
sufficient financing/insurance from private sources on appropriate terms; the
- 33 -
A clear, evidence-based justification for
IFI involvement and anticipated value
added is required for a positive rating.
Criteria/Sub-Criteria
Evidentiary Requirements and Analytical Methods
IFI’s catalytic role in mobilizing funds from other investors and lenders; the IFI’s
role in reducing risks and providing comfort); and (ii) non-financial additionality
(the IFI’s role in bringing about a fair allocation of risks and responsibilities; the
IFI’s role in ensuring quality of project design and in supporting client capacity).
Benchmarks
The Rationale for IFI Involvement is assessed from the perspective of what was
expected to occur at entry (at the appraisal or pre-commitment stage). Thus,
the timing of the assessment is parallel to the timing of the Strategic Alignment
sub-criterion.
Results
Operational Performance
Achievement of Outputs
Fulfillment of Project
Business Objectives
Contribution to Intended
Outcomes
Compares achieved values with both initial targets and revised targets, if the
latter were formally revised during project implementation. In general,
counterfactual analysis is not needed because by definition the outputs are
under the control of the project and not influenced by external factors. For
private sector operations, outputs would include the physical investments
realized.
Evidence on process objectives might include, e.g., the implementation of an
investment plan, the establishment of a strong management team, or the
introduction of an accounting system. In most cases, the project’s process and
business outcomes will be attributable to the project, so there is no need for
comparison to a without-project counterfactual. Achievements are compared
to appraisal projections or performance targets.
Relevant evidence would include baseline and at-evaluation values of
performance indicators defined under the project as well as other quantitative
and qualitative information relevant to the intended outcomes, including the
project’s reach to target groups and mitigation of market failures.
Outcomes are assessed against the project’s intended results as contained in
appraisal and/or approval documents or as reconstructed by the evaluator. If
the intended results and/or targets were revised during implementation, the
assessment considers the project’s achievements against both the original and
revised intended results and/or targets.
- 34 -
For a positive rating, there must be
evidence that the project substantially
achieved its targeted outputs.
For a positive rating, there must be
substantial achievement of the project’s
process and business objectives
compared to appraisal projections or
performance targets.
A positive rating on Contribution of
Intended Outcomes requires that there is
strong evidence that the project
contributed (or is likely to contribute)
substantially to intended outcomes.
When the desired outcome is achieved
but there is evidence that the results are
primarily due to other factors, the rating
is adjusted downward, accordingly.
Criteria/Sub-Criteria
Evidentiary Requirements and Analytical Methods
Outcomes are compared against a without-project counterfactual. If feasible
and practical, this is done using an impact evaluation; if not, plausible causality
is established using a theory-based method: presenting evidence on the
achievement of all levels in the results chain and testing the validity of
assumptions.
For private sector operations, the analytical method includes stakeholder
analysis, i.e., the extent to which the project had its intended impact on
employees, suppliers, competitors, and neighbors. For projects through
financial intermediaries, the stakeholders include sub-borrowers and fund
investees. The assessment also considers the extent to which financial
intermediary projects reached their target groups. If the IFI-supported subborrowers cannot be identified, a before-and-after comparison of the financial
intermediary’s portfolio is made to determine whether the intermediary
increased its exposure to the target group, and then a theory-based method is
used to established plausible causality to the project (for example, evidence
that the intermediary had improved its marketing, screening, and credit
procedures as a result of the project with the intention of increasing its reach to
SMEs).
Contribution to Corporate
Goals
Benchmarks
The rating reflects the project’s
incremental contribution to observed
outcomes, regardless of whether the
observed outcomes moved in the “right”
or “wrong” direction. For example,
 If outcome indicators met or
exceeded targets, but there is
evidence that the change was due
mainly to external factors, a less than
satisfactory rating is warranted.
 If outcome indicators deteriorated,
but there is evidence that the decline
would have been worse in the
absence of the project, a positive
rating is warranted.
Assesses the incremental contribution of the project to broad corporate goals
that are not part of the project’s intended results – for example, poverty
reduction, rural poverty reduction, shared prosperity, gender equality,
institutional development, social cohesion, EU policy, etc.
On an exceptional basis,“Not Rated” is a
possible rating when evidence is missing
or weak.
A substantial and plausible contribution
of the project to the achievement of
corporate goals must be shown to merit a
positive rating.
The assessment covers actual achievements in these areas, as opposed to
expected achievements (the latter is covered under Relevance). It uses a
theory-based approach to establish plausible association between the project
and corporate goals. It also discusses other factors that could have affected the
achievement of those goals. The assessment may be supported by evidence
from other evaluations and research.
As with the Contribution to Intended
Outcomes, the rating reflects the
project’s incremental contribution to
observed outcomes, regardless of
whether the observed outcomes moved
in the “right” or “wrong” direction.
On an exceptional basis, “Not Rated” is a
possible rating when evidence is missing
- 35 -
Criteria/Sub-Criteria
Unintended Outcomes
Evidentiary Requirements and Analytical Methods
Assesses positive or negative results of the project that were not included in the
project’s statement of objectives.
Uses a theory-based approach to establish plausible causality between the
project and unintended outcomes. To be included, unintended outcomes must
be truly unanticipated, attributable to the project, quantifiable, of significant
magnitude, and at least as well evidenced as the project’s other outcomes.
Where there are unintended benefits, an assessment should be made of why
these were not "internalized" through project restructuring by modifying the
project’s intended results.
Benchmarks
or weak.
A substantial and plausible contribution
of the project to the achievement of
unanticipated outcomes must be shown
to merit a positive rating.
Positive impacts that are attributable to
the project merit a positive rating;
negative impacts that are attributable to
the project merit a negative rating.
The rating reflects the project’s
incremental contribution to observed
outcomes, regardless of whether the
observed outcomes moved in the “right”
or “wrong” direction.
“Not Rated” is a possible rating when
there were no unintended outcomes or
when evidence is missing or weak.
Efficient Use of Resources
Financial Performance of the
Project/Company
Assesses the incremental effect of the project on the financial performance of
the company (or for Group D projects, the financial intermediary or fund).
Financial performance is measured by the FRR or ROIC for the project; a
comparison of appraisal financial projections; and other performance indicators
from the company’s financial statements. For financial intermediary projects,
the required information includes the financial performance of the financial
intermediary or fund, and the financial performance of the sub-portfolio or fund
portfolio.
In evaluating financial performance, observed changes in performance are
compared with a without-project counterfactual. The choice of method is
appropriate to the project type.
- 36 -
For a positive rating, the project’s FRR or
ROIC is greater than or equal to the
company’s Weighted Average Cost of
Capital (WACC), or actual performance
meets or exceeds appraisal projections,
and/or other financial performance
indicators are in line with appraisal
projections. For intermediation projects,
there is adequate evidence that the
marginal performance of the subportfolio had a positive effect on the
financial intermediary’s profitability, or
that the projected or realized net return
Criteria/Sub-Criteria
Economic Efficiency
Evidentiary Requirements and Analytical Methods
Assesses the extent to which the costs involved in achieving project objectives
were reasonable in comparison with the project’s benefits, and the extent to
which the project was implemented at least cost compared to alternative ways
of achieving the same results.
Cost-benefit analysis is conducted to the extent that data is available and it is
reasonable to place a monetary value on project benefits. The costs and
benefits of the project include both private and social costs and benefits, and
extend to all affected stakeholders. Cost-effectiveness analysis compares the
unit costs of the project, or component costs, with those of similar projects.
Requires information on traditional measures of efficiency, e.g., FRR, ERR, NPV,
unit rate norms (cost per unit of input or cost per unit of output), and service
standards, as well as information on the cost of projects with similar objectives,
scope, and design.
For private sector operations, the calculation of the ERR begins with the
financial benefit to the project company (the FRR or ROIC) and adjusts for taxes
paid and subsidies received. It also includes any marginal employment effects
and incremental impacts on other stakeholders. Care should be taken to not
count the gross increase in employment or compensation without considering
what they would have been in the absence of the project. The assumptions
behind the calculations should be fully explained.
The analysis shows the incremental impact of the project, i.e., the costs and
benefits compared to the without-project counterfactual.
The project’s Economic Efficiency should not be confused with the achievement
of improved efficiency of the sector or program being supported. The latter is
an outcome and would be included in the assessment the Contribution to
Intended Outcomes.
- 37 -
Benchmarks
on equity or net IRR to the fund’s
investors is greater than or equal to the
fund’s WACC.
For a positive rating on Economic
Efficiency, the Private GPS’s benchmark is
that the ERR or EROIC should be greater
than or equal to 1.2 times the project
company’s WACC. The Public GPS’s
benchmark is that the ERR should be
greater than or equal to the opportunity
cost of capital. In addition, project costs
should have been equal to or less than
the costs of alternative ways of achieving
the same objectives.
Substantial cost overruns would usually
lead to a negative rating on Economic
Efficiency, but if were unrealistically
estimated at appraisal, they should
instead enter into the rating for Quality
at Entry.
Criteria/Sub-Criteria
Implementation Efficiency
IFI Investment Profitability
Evidentiary Requirements and Analytical Methods
Measures other aspects of efficiency not included in Economic Efficiency, such
as aspects of design and implementation that either contributed to or reduced
efficiency. Implementation delays are a typical implementation inefficiency.
The timeline of implementation is compared with the projected timeline at
entry (the appraisal or pre-commitment stage), and reasons for differences are
discussed.
Measures the profitability of the IFI’s investment(s) in the project.
The calculation of the profitability of the project to the IFI is based on either the
investment’s net profit contribution or the quality of the investment’s gross
profit contribution. The net profit contribution method is preferred, if cost
accounting data is available. Gross profit contribution is applied in a largely
qualitative manner as a proxy for likely investment performance.
Sustainability
Outcome Sustainability
The assessment is based on (i) the likelihood that some changes may occur that
are detrimental to the continuation of the project’s results or expected results;
and (ii) the impact on the operation’s results of some or all of these changes
materializing.
Benchmarks
Significant delays or other
implementation inefficiencies would
suggest a negative rating for
Implementation Efficiency.
For a positive rating, the net profit
contribution is sufficient relative to the
IFI’s target return on capital or overall
profitability objectives. Detail by type of
operation is contained in Private GPS OPs
20.2 – 20.5.
A positive rating requires strong evidence
that the expected value of risks is
moderate to low.
The risks may include technical, financial, economic, social, political,
environmental, government ownership/commitment, other stakeholder
ownership, institutional support, governance, and exposure to natural disasters.
Commercial Sustainability
Uses the evaluator’s judgment of the uncertainties faced by the operation’s
results (intended outcomes, unintended outcomes, contribution to corporate
goals) over its expected remaining useful life, taking account of any risk
mitigation measures already in place at the time of evaluation.
The forward-looking commercial viability of the company, financial
intermediary, and/or sub-borrowers/fund investees. Considers the company’s
(or financial institution’s or sub-borrowers’/fund investees’) adaptability and
prospects for sustainability and growth.
- 38 -
An expectation of continued commercial
viability in projected market conditions is
required for a positive rating.
Criteria/Sub-Criteria
Compliance with Safeguards
IFI Performance
Quality at Entry
Evidentiary Requirements and Analytical Methods
Based on projected future financial performance and the performance of the
company, financial institution, and/or sub-borrowers/fund investees relative to
the market or sector peers.
The Client’s compliance with applicable safeguard policies, if any, including
implementation of the mitigation plan. Based on the degree of compliance with
the IFI’s standards in effect at project entry, and the standards prevailing at the
time of the evaluation. For public sector operations, based on the
implementing agency’s management of its environmental and social impacts.
For private sector operations, based on the project company’s management of
its environmental and social impacts (and for financial intermediary operations,
the environmental and social performance of sub-projects or fund investee
companies).
Measures the extent to which the IFI identified, facilitated preparation of, and
appraised the operation such that it was most likely to achieve its planned
outcomes and was consistent with the IFI’s fiduciary role.
For public sector operations, this includes the quality of the IFI’s analytical work,
the quality of the results framework; the quality of design of the monitoring and
evaluation system; stakeholder and institutional analysis; risk assessment and
plans for managing risks; the quality of implementation arrangements; the
appraisal of environmental and social risks, and the inclusion of safeguards to
mitigate them; and the appropriateness of the investment instrument selected.
For private sector operations, this includes the quality of the IFI’s assessment of
the operation as being relevant to the IFI’s corporate, country, and sector
strategies; the quality of the results framework and the design of the
monitoring and evaluation system; the assessment of sponsors, company,
management, country conditions, market dynamics, project concept,
configuration and cost; the appraisal of the financial plan, source of project
funds, and assumptions used in the project’s financial projections; the
assessment of project and political risks, and steps taken to mitigate them; the
appraisal of environmental and social risks, and the inclusion of safeguards to
mitigate them; and the appropriateness of the investment instrument selected.
- 39 -
Benchmarks
For a positive rating, the project’s
implementing agency or company (or
financial institution, fund, sub-borrowers,
and investees) must be in material
compliance with applicable safeguards.
For a positive rating, the IFI should have
materially met its operational standards
in these areas, and there were no
significant shortcomings in project results
due to the IFI’s performance at project
entry.
Criteria/Sub-Criteria
Quality of Supervision
Evidentiary Requirements and Analytical Methods
For public sector operations, considers the IFI’s focus on development, policy, or
transition outcomes throughout project implementation; the supervision of
applicable fiduciary and safeguards aspects; the quality of supervision inputs
and processes; the implementation of the monitoring and evaluation system;
the candor and quality of performance reporting; and IFI’s role in ensuring
adequate transition arrangements for regular operation of supported activities
after project closing.
Benchmarks
For a positive rating, the IFI should have
materially met its operational standards
in these areas, and there were no
significant shortcomings in project results
due to the IFI’s supervision performance.
For private sector operations, includes the completeness of supervision reports
in documenting project status and risk; the monitoring of the client company’s
compliance with the terms of the investment; the monitoring of the client
company’s environmental and social performance, and adherence to relevant
government regulations and IFI requirements; the adequacy and timeliness of
the IFI’s response to emerging problems or opportunities; and the effectiveness
of hand-over procedures should there be changes in IFI staff monitoring
responsibilities.
Client Performance
Non-Financial Performance
of the Company
Government and
Implementing Agency
Performance
Non-financial performance covers compliance with relevant government
regulations and IFI requirements. For financial intermediary operations,
includes compliance by the financial intermediary and sub-borrowers/fund
investees.
Assesses the extent to which the borrower (including the government and
implementing agency or agencies) ensured quality of preparation and
implementation, and complied with covenants and agreements, towards the
achievement of intended outcomes. Includes such aspects as government
ownership and commitment; the enabling environment for the project;
adequacy of consultations with stakeholders; readiness for implementation;
timely resolution of implementation problems; fiduciary management;
compliance with environmental and social safeguards; adequacy of monitoring
and evaluation arrangements; relationships with other donors and
stakeholders; and adequacy of arrangements for the transition to normal
- 40 -
For a positive rating, the company or
intermediary is in material compliance
with relevant government regulations
and IFI requirements, and subborrowers/fund investees are in material
compliance with relevant government
regulations and IFI requirements.
For a positive rating, there were at most
moderate shortcomings in the
performance of the government and
implementing agency or agencies.
Criteria/Sub-Criteria
Evidentiary Requirements and Analytical Methods
operations after project closing.
Benchmarks
The evaluator should take account of the operational, sector, and country
context in weighing the relative importance of each aspect of government and
implementing agency performance as they affected outcomes.
The next section illustrates the Option 2 methodology by applying it to two projects, one private sector and one public sector:

an IFC investment operation with a mid-tier bank in an emerging market, designed to introduce a new financial instrument and expand
the bank’s SME and mortgage financing, particularly in under-served areas of the country. Evaluative evidence was taken from IFC’s
Expanded Project Supervision Report and IEG’s Evaluation Note.

a World Bank rural roads project in India designed to achieve broader and more sustainable access to markets and social services in
participating districts of the country. Evaluative evidence was taken from the WB’s Implementation Completion and Results Report and
IEG’s ICRR Review.
All ratings are on a four-point scale: Satisfactory, Partly Satisfactory, Partly Unsatisfactory, and Unsatisfactory (for Relevance, Results, Efficient
Use of Resources, Overall Project Performance, IFI Performance, and Client Performance) and Likely, Somewhat Likely, Somewhat Unlikely, and
Unlikely (for Sustainability).
- 41 -
Private Sector Application: IFC Tier I Bank Financing Project
1. PROJECT INFORMATION
Project Name
Country
Borrower
Approval Date
Commitment Date
IFC Financing
Co-financing
“Tier 1 Bank Financing Project”, or “The Project”
“The Country”
“The Bank”
5 July 2007
27 July 2007
Up to US$100 million subordinated Hybrid Tier I debt investment
None
Context
Following macroeconomic stabilization in 2002 and a subsequent fall in interest rates, the loan portfolios
of the Country’s banks had grown rapidly, becoming a significant part of their balance sheets for the first
time in over 30 years. Despite the recent growth, access to finance remained a key constraint for a
number of market segments. Small- and medium-scale enterprises (SMEs) continued to face inadequate
access to long term credit, the central and eastern regions of the Country were underserved, and nascent
mortgage financing had not received attention by the Country’s banks.
On the funding side, most banks faced constraints in raising long-term funding as the Country’s capital
markets lacked depth and sophistication, and capital resources were largely dependent on equity funding.
A recent decision of the Country’s banking regulatory authority to increase the minimum capital adequacy
ratio to 12 percent for banks expanding their branch networks, combined with rapid growth in riskweighted assets, made it imperative for many banks to raise capital.
Hybrid Tier I capital was used widely since the 1990s by financial institutions in Europe and the USA, and
in the years before the Project had become an attractive and useful instrument for banks in emerging
markets. In an effort to make this capital instrument available to the Country’s banks, the regulatory
authority introduced enabling legislation and regulations for Hybrid Tier I capital (called “Tier I” capital in
the Country) in November 2006. However, local financial institutions and banks, unfamiliar with this
instrument, had yet to take advantage of this regulation to provide themselves with much-needed capital
and financial flexibility.
Project Development Objectives
According to the Board Report (pp. i, ii, 2, 4, and 5), the project intended to:

open the market for Hybrid Tier I financing in the Country by (i) introducing the Hybrid Tier I
financing instrument; (ii) demonstrating the feasibility of the Hybrid Tier I financing instrument
and the effectiveness of its enabling regulations; and (iii) building market appetite for both Tier II
capital and Hybrid Tier I capital by potential syndication of IFC’s existing Tier II capital investment
in the Bank and eventually selling part of its Hybrid Tier I investment in the Project.

increase the Bank’s SME financing in the Country, particularly in the underserved central and
eastern regions of the Country.

expand the availability of residential mortgage finance throughout the Country, thus increasing
the level of home ownership.
- 42 -
Key performance indicators and targets were the following:
Intended Outcome
Catalyze the introduction of the Hybrid Tier I
financial instrument; clarify the terms and intent of
new regulations
Demonstrate the feasibility of the Hybrid Tier I
instrument and the effectiveness of its enabling
regulations
Increase the Bank’s financing of SMEs
Expand the Bank’s residential mortgage financing
Key Performance Indicators and Targets
Successful issuance, with regulatory approval, of
the Hybrid Tier I debt instrument
Three financial institutions in the Country adopt
similar Hybrid Tier I financing by 2009
Growth of the Bank’s total residential mortgage
portfolio by at least 20% per year through 2012
(baseline: US$200 million in December 2006)
Growth of the Bank’s residential mortgage portfolio
by at least 20% per year through 2012 (baseline:
US$200 million in December 2006)
Increase in mortgage portfolio to 10% of total Bank
loan portfolio by 2012 (baseline: 5.8% in December
2006)
Expand the Bank’s SME and mortgage financing in
underserved regions of the Country
Increase in the Bank’s market share of the
Country’s mortgage market to 5% by 2012
(baseline: 1.5% in June 2007).
Increase in branches outside major urban centers
by at least 100% by 2012 (baseline: 215). Most of
the projected total of more than 500 branches
would be in locations outside major urban areas.
Project Business Objectives
In addition to expanding the Bank’s SME and mortgage portfolios and the branch network, the Project
aimed to strengthen the Bank’s capital structure, allowing the Bank to raise up to US$1.25 billion from
other sources to increase its financial intermediation.
Project Description
The Bank was a leading mid-tier bank in the Country. It was 84 percent owned by a privately-owned
financial services holding company that was in turn a 50/50 joint venture between a national business
group with interests in steel, shipping, power, shipping containers, and financial services (42.1 percent
ownership), and a large European bank (42.1 percent). Other shareholders owned 15.8 percent of the
Bank. Since partnering with the European bank in 2005, the Bank significantly modified its strategy to
become a full-service bank with a strong focus on SME and consumer finance segments to complement its
historically strong corporate, private banking, and trade finance divisions. The Bank was a long-standing
client of IFC in the Country.
At entry, the indicative terms and conditions of the IFC investment were the following:
Currency and amount
Type
Maturity
Up to US$100 million
Hybrid Tier I subordinated debt instrument eligible as Tier I capital
under national banking regulations
Perpetual legal maturity with repayment at the Bank’s option on the
10th anniversary of disbursement
- 43 -
Alternative payment mechanism/
conversion
Interest rate
Front-end fee
Arrangement fee
Commitment fee
Subject to regulatory approval, payment of interest or principal to
be made in the form of common shares of the Bank.
Floating at 6 month LIBOR + 3.50% per annum with a step-up in
spread on the 10th anniversary of disbursement
1.00%
0.30%
0.50%
Hybrid Tier I instruments rank lower than Tier II subordinated debt but higher than common or preferred
equity, and contain some equity-like features. The note is payable or pre-payable by the borrower with
only regulatory approval, which in turn is based on capital adequacy requirements. As common practice,
there is a significant increase in the interest rate after ten years, which serves as a strong incentive for the
borrower to repay at that time.
In accordance with national banking regulations, the Bank could withhold payment of interest on an
interest payment date, and such interest would be non-cumulative. The alternative payment mechanism
was to provide for option of payment in shares if case interest is not payable, subject to approval by the
regulator and relevant corporate approvals. If the Bank did not repay the principal on the tenth
anniversary of disbursement, IFC would have the option of requiring payment in shares of the Bank,
subject to national regulatory and corporate approvals.
2. RELEVANCE: SATISFACTORY
Strategic Alignment: Partly Satisfactory
The Project’s intention to increase long-term financing of SMEs was likely to have been relevant to the
development needs of the financial sector, although the Board Report does not provide complete
evidence. The Board Report reports that credit penetration in the Country was only 39% of GDP, lower
than emerging market peers, and that credit penetration in the SME segment was even lower. It also
notes that the Country had the largest under-banked population in the region. However, the Board
Report did not provide sufficient data to conclude that SME financing as a proportion of the SME sector’s
GDP was less than in comparator countries. With respect to mortgage finance, the Board Report reports
that the Country’s ratio of mortgage loans to GDP was only 4 percent in 2005, compared to 38 percent in
the EU and 63 percent in the USA, and was low compared to some other emerging markets.
The Board Report argues convincingly that there was a need to introduce a new and innovative capital
instrument – Hybrid Tier I financing – to allow banks to support continued loan growth. The Hybrid Tier I
product would allow banks to move toward better and more diversified capital funding which in turn
would enhance the structure of their balance sheets and offer more competitive loan pricing to their
borrowers.
Prior to the Project, the Bank defined its SME portfolio as loan sizes less than US$1 million. The Board
Report is not clear whether this definition would be used under the Project, or whether loan size was a
good proxy for the size of borrowers. The XPSR provides data on the Bank’s SME portfolio using the IFC
definition for the Country (loans in excess of US$2 million).
The project’s intended outcomes were consistent with the World Bank Group’s Country Assistance
Strategy of November 2003, which supported among other things the establishment of a sound banking
sector. IFC’s country strategy aimed inter alia at expanding its client base to less developed regions of the
Country and addressing regional inequalities. IFC’s Financial Markets Strategy aimed at strengthening
financial systems and increasing access to finance, widening the financial product base as well as
- 44 -
deepening capital markets and broadening the investor base. The goal of increasing access to finance
included delivery of financial products to underserved sectors, in particular SMEs and low-income
households, while addressing regional inequalities. In line with this strategy, IFC’s SME financing was
expected to remain an important activity, along with increasing emphasis on housing finance.
Rationale for IFI Involvement: Satisfactory
Although the enabling legislation governing Hybrid Tier I capital was adopted in November 2006, local
financial institutions and banks, unfamiliar with this instrument, had not yet taken advantage of this
instrument to access capital and enhance financial flexibility. By introducing this product and setting a
market precedent, IFC was likely to provide greater comfort to international investors for such
subordinated debt issues.
The Board Report does not discuss the comparative advantage of IFC, vis-à-vis other sources of finance, in
providing funding for SME and mortgage lending. The XPSR argues that, without the IFC investment, it
would have taken the Bank longer to increase its SME portfolio and housing finance loans with respect to
market position and regional presence.
3. RESULTS: PARTLY UNSATISFACTORY
Operational Performance: Partly Unsatisfactory
Achievement of Outputs: Not Rated
The Project’s intended outcomes were expressed in terms of the volume of SME and mortgage financing
provided by the Bank. In some other SME and housing finance projects, these would be considered
“outputs” that would lead to broader development outcomes such as increased SME competitiveness and
increased home ownership and housing quality. In this project, the expansion of SME and mortgage
financing are considered to be intended outcomes, so they are not assessed in this section.
Fulfillment of Project Business Objectives: Partly Unsatisfactory
Strengthen the Bank’s capital structure: The Bank’s ratio of equity to total assets averaged only about 6.8
percent before the Project was approved in 2007, improving to 9.0 percent during 2007-2010 and to 10.9
percent in 2011-2012 after a capital increase resulting from a merger with another bank. The Bank
therefore needed and continues to need Tier I and/or Tier II capital to meet the required regulatory
capital adequacy ratio of at least 12 percent.
Contribution to Intended Outcomes: Partly Unsatisfactory
Catalyze the introduction of the Hybrid Tier I financial instrument; clarify the terms and intent of new
regulations. The new instrument was introduced under the Project. Prior to the Project, IFC worked with
the financial sector regulatory authority to clarify several aspects of the regulations and to obtain
approval for this first issue.
Demonstrate the feasibility of the Hybrid Tier I instrument and the effectiveness of its enabling
regulations (target: three financial institutions in the Country adopt similar Hybrid Tier I financing by
2009). The Hybrid Tier I product was not replicated in the market as a new capital product. Right after
the IFC deal was closed, the Lehman bankruptcy caused a significant increase in the cost of capital. It
- 45 -
became very expensive to issue such subordinated notes, as investors asked for large spreads. The lack of
investor interest in the new product appears to have been mainly due to factors external to the project.
Increase the Bank’s financing of SMEs. SME financing did not increase to the target level. According to
the Reach Data provided by the Bank to IFC, the SME loan portfolio (using the IFC definition for the
Country of up to US$2 million loan size) grew by 31 percent between 2007 and 2011 (target: 20 percent
per year between 2007 and 2012, or 107 percent increase by 2011). The SME loan share of total loans
actually fell from 67 percent in 2007 to 38 percent in 2011 due to the Bank’s diversification of the loan
portfolio among mortgages, retail loans, micro loans, and corporate loans.
Expand the Bank’s residential mortgage financing. The expansion in the Bank’s mortgage financing
exceeded targets. The Bank’s mortgage loan portfolio grew by 400 percent between 2007 and 2011
(target: 20 percent per year between 2007 and 2012, or 107 percent increase by 2011), and the share of
mortgages to total loans increased significantly from 6.5 percent in 2007 to 14.1 percent in 2011 (target:
10 percent by 2012). However, the Bank’s market share actually declined, from 1.5 percent in 2007 to
below 1 percent at the end of 2011 (target: 5 percent). The XPSR explains the lack of growth of mortgage
financing as the result of the 2008 sub-prime crisis which halted mortgage lending globally for at least
three years. However, it is not clear why, in this environment, the Bank was able to increase its mortgage
lending by 400% while at the same time losing market share.
Expand the Bank’s SME and mortgage financing in underserved regions of the Country (target: increase of
branches outside major urban centers by at least 100% by 2012). Although the Bank expanded the total
number of branches from a baseline of 215 to 509 by the end of 2012 (2012 target: 500), its presence in
less urban (underserved) areas did not increase. In 2011, the Bank merged with another bank, and the
branches of the latter were mainly in urban areas. The XPSR notes that there has been no change in the
Bank’s strategy of branch expansion to less urban areas, and that once the initial operational adjustments
to the merger are made, the Bank will implement this strategy.
Since the failure to achieve the targets for SME financing, mortgage financing, and expansion to
underserved areas was under the control of the Bank and thus attributable to the Project, the
Contribution to Intended Outcomes is rated Partly Unsatisfactory.
Contribution to Corporate Goals: Not Rated
None reported in the XPSR.
Unintended Outcomes: Not Rated
None reported in the XPSR.
4. EFFICIENT USE OF RESOURCES: PARTLY SATISFACTORY
Financial Performance of the Project, Company, or Intermediary: Partly Unsatisfactory
The XPSR does not provide information on the financial performance of the Project – i.e., the
performance of the Bank’s additional SME and mortgage portfolio that resulted from the IFC investment.
(In 2007, the Bank’s SME portfolio accounted for 59% of total loans, and the housing finance portfolio
accounted for 9.3%.)
The XPSR did not estimate any return on invested capital (ROIC) for the Project, even though the Board
Report included the ROIC and the EROIC as indicators to be tracked or estimated annually by IFC. The
- 46 -
expected ROIC of 14.5% was projected to exceed the WACC by 5% and the expected EROIC of 15.3% was
projected to exceed the WACC by 5.8%, both over the life of the project. In the absence of a ROIC and
EROIC, the XPSR and Evaluative Note provide information on financial performance for the Bank as a
whole. The Evaluative Note argues that, based on IEG’s evaluation experience with banking projects,
projects that meet or exceed the ROAA “satisfactory” benchmark of 1.5% or better during the evaluation
period will generally have a “satisfactory” ROIC when compared to the IEG WACC-based benchmark.
The table below shows a few of the more important financial performance indicators and compares them
with targets (projections in the Board Report and with IFC’s benchmarks for SME-oriented banks. More
detailed financial performance information is contained in the XPSR and Evaluative Note.
Indicator
Non-performing loans
Return on Average Assets
(ROAA)
Return on Average Equity
(ROAE)
Baseline
(2007)
1.8%
0.9%
2012 Target
2012 Actual
1.2%
2.3%
IFC’s SME Bank
Benchmark
< 4%
>= 1.5%
13.5%
20.0%
>= 15%
10.1%
2.3%
1.0%
The Bank’s non-performing loans (including SME and mortgage financing as well as other loans) worsened
from 1.8% in 2007 to 2.3% in 2012, underperforming the 2012 target of 1.2% but within the SME bank
benchmark of less than 4%. The Bank’s ROAA increased marginally from 0.9% in 2007 to 1.0% in 2012,
compared to a target of 2.3% and the SME bank benchmark of 1.5%. The Evaluative Note remarks that
IFC’s ROAA and ROAE benchmarks are for SME banks that have only a moderate rate of branch expansion,
and that due to the Bank’s extensive branch expansion program its ROAA and ROAE would be expected to
be lower than the benchmarks. Given the Project’s aim to expand the Bank’s branch network to
underserved areas of the Country, it is not clear why lower ROAA and ROAE targets were not projected at
appraisal.
Economic Efficiency: Not Rated
As noted above, no financial rate of return (FRR) was calculated for the Project. Thus, there is no
calculation of an Economic Rate of Return (ERR) based on the project’s FRR and adjusted for taxes and
subsidies, net effects on other stakeholders, and externalities.
The Evaluative Note reports that the Bank paid an effective income tax rate of between 21% and 37%
before the Project, and estimated that the Bank paid at least US$240 million in income taxes during 20072012. There is no information to determine whether the Project generated consumer surplus in the form
of interest rates lower than consumers’ willingness to pay.
As noted above, the XPSR and Evaluative Note present information on the profitability of the Bank as a
whole. The Evaluative Note states that the Bank’s 2011 merger with another bank adversely affected the
financial and operating performance of the Bank due to merger-related costs and integration adjustments
in operations. The rapid expansion of branches also negatively affected profitability during 2007-2010
because the new branches did not achieve profit break-even until about two years after starting
operations. Efficiency improvements from the merger are expected to become significant starting in
2013. The Evaluative Note also states that the Bank’s low ROAA and ROAE during 2007-2010 were also
the result of the global financial crisis that began in late 2007.
- 47 -
Implementation Efficiency: Satisfactory
The Project was completed on schedule and there was no discussion of implementation delays in the
XPSR or Evaluative Note.
IFI Investment Profitability: Satisfactory
According to the XPSR, IFC’s return on equity (ROE) for the project was 5.75%. The XPSR and Evaluative
Note do not compare this figure with the IFC’s target return on capital or an overall profitability objective.
The XPSR reports that the Bank had not been in any arrears on payments and that the asset was rated as
being of good quality on IFC’s books. Both the XPSR and the Evaluative Note rate IFC’s Investment
Outcome as “satisfactory”.
5. SUSTAINABILITY: LIKELY
Outcome Sustainability: Likely
The Bank continues to have a strategy of expanding its SME financing. The Bank aims to be not only an
SME bank but also an advisor to SMEs. In line with this target, the Bank held a total of 53 SME Academy
meetings in 33 cities since 2005; about 11,300 business men and women participated in these meetings
and attended the training programs. Since 2006 the Bank has organized search meetings in thirteen cities
and for four sectors to facilitate the preparation of SME strategies. The Bank has developed other
products targeted specifically to SMEs, including a special hotline, SME TV, an SME training academy, and
an SME Consultancy. Within SME Banking, the Bank introduced SME insurance services.
Commercial Sustainability: Likely
In explaining the Bank’s relatively low profitability despite its relatively low ratios of non-performing loans
and of operating costs to average net loans, the Evaluative Note suggests that the Bank’s fee income is
not high enough and/or its average interest rate on loans is not sufficiently high compared to its average
borrowing cost. Nevertheless, the Evaluative Note argues that the synergies and efficiency improvements
from the Bank’s recent merger will become significant starting in 2013. The profitability of new branches
also is expected to improve.
As noted above, the Bank’s capital adequacy ratio is below the regulatory minimum of 12%, and the Bank
will need Tier I and Tier II capital. However, Tier I and Tier II capital are intended to be only short- to
medium-term solutions to the undercapitalization of banks in IFC client countries. Eventually, this capital
will need to be replaced by equity through retained earnings and/or injections of new equity. Because of
the structure of the IFC investment, the Bank will need to raise equity from its existing shareholders when
the IFC Hybrid Tier I subordinated debt reaches its 10th anniversary mark (mid-2017), when a step-up
interest rate begins unless IFC decides to exercise its option to convert the subordinated debt to equity.
Conversion to equity is not assured if the Country’s stock market is bearish at that time. On the positive
side, the Bank’s two main shareholders appear to have the combined resources to refinance the IFC loan
if necessary.
The Bank’s portfolio risk is considered to be medium-high due to exposure to some medium-high risk
sectors such as food, beverages, and tobacco (8%), metal (8%), chemicals (3.6%), and construction (8.2%).
- 48 -
Compliance with Safeguards: Satisfactory
The Project was classified as a Category FI (Financial Intermediary) project according to IFC’s
Environmental and Social Review Procedure. As an existing client, the Bank had a Social and
Environmental Management System (SEMS) and the capacity to implement it. However, at project entry
the SEMS along with the Bank’s own Environmental Risk Management Plan (ERMP) had not fully
incorporated IFC’s Exclusion List and needed an improved system for monitoring the Bank’s investments.
The Bank also needed a mechanism to update the SMES based on changes in local laws and ensure that it
complies with IFC requirements.
The Bank updated its SEMS giving further details on objectives and policies, a revised exclusion list
complying with IFC guidelines, and additional sections on Community Engagement Policy and Social Risk
Policy. However, during the last supervision mission it was noted that implementation of the SEMS still
lacks adequate procedures for medium-high risk projects as no information was provided on risk
categorization and IFC Exclusion List screening. Since the project met IFC’s at-approval and current
requirements, and the Bank has the capacity to implement the SEMS, the Evaluative Note rated the
Project’s environmental and social performance as “satisfactory” at evaluation.
6. OVERALL PROJECT PERFORMANCE: PARTLY SATISFACTORY
Based on a Satisfactory rating for Relevance, Partly Unsatisfactory for Results, Partly Satisfactory for
Efficient Use of Resources, and Likely Sustainability, overall project performance is rated Partly
Satisfactory.
7. IFI PERFORMANCE: PARTLY SATISFACTORY
Quality at Entry: Partly Satisfactory
The Evaluative Note notes that IFC’s Tier I perpetual subordinated loan was an excellent investment
structuring. The step-up interest rate is designed to encourage redemption of the IFC loan, which is
important because the loan has no final maturity date. The conversion option allows IFC to improve its
return if the conversion price and the future prospects of the Bank’s shares are favorable. However, the
instrument involves risks for the Bank because its two main shareholders may not want to be diluted at
that point in time.
IFC’s project team assisted in the elaboration of regulations for the Hybrid Tier I instrument by
highlighting some inconsistencies in the Tier I Communiqué of the regulatory authority, and the
Communiqué subsequently was amended. According to the Country’s banking authorities, IFC played a
major role in clarifying regulatory intent as well as refining specific, regulatory-permissible product
features not initially understood by the authorities.
Environmental and social risks were not adequately identified during appraisal. Although the project was
correctly categorized as Category FI, considering the type of IFC investment and that the Bank’s portfolio
included medium-high risk sectors, the applicable requirements should have included the IFC
Performance Standards as well.
As noted above, the project’s ROAA and ROAE targets may have been inconsistent with the Project’s aim
to expand the Bank’s branch network to underserved areas of the Country.
- 49 -
The Board Report contained clear development objectives and associated key performance indicators to
measure their achievement. However, the monitoring and evaluation system did not include a
methodology to allow attribution of results to the IFC investment.
Project preparation was efficient, with only seven weeks between the date of the mandate letter and
commitment, including a full Board meeting.
Quality of Supervision: Satisfactory
IFC provided regular advice and recommendations on environmental and social safeguards. Annual
supervision reports, quarterly credit rating reviews, and environmental reviews were conducted. IFC
maintained a good working relationship with the Bank, going beyond its investment to invite the Bank to
regional SME conferences.
8. CLIENT PERFORMANCE: SATISFACTORY
Non-Financial Performance of the Company: Satisfactory
The Bank was cooperative in providing IFC with regular and timely reporting, and the reports were of high
quality. The Bank provided AEPRs on time with adequate portfolio details. Together with a consulting
firm, the Bank updated its SEMS to ensure consistency with IFC requirements.
There is no further information in the XPSR or Evaluative Note on other non-financial aspects of the
Bank’s performance such as compliance with financial sector regulatory requirements.
9. RATINGS SUMMARY
Relevance
Strategic Alignment
Rationale for IFI Involvement
Satisfactory
Partly Satisfactory
Satisfactory
Results
Operational Performance
Contribution to Intended Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Partly Unsatisfactory
Partly Unsatisfactory
Partly Unsatisfactory
Not Rated
Not Rated
Efficient Use of Resources
Financial Performance of the Project/Company
Economic Efficiency
Implementation Efficiency
IFI Investment Profitability
Partly Satisfactory
Partly Unsatisfactory
Not Rated
Satisfactory
Satisfactory
Sustainability
Outcome Sustainability
Commercial Sustainability
Compliance with Safeguards
Likely
Likely
Likely
Satisfactory
- 50 -
Overall Project Performance
Partly Satisfactory
IFI Performance
Quality at Entry
Quality of Supervision
Partly Satisfactory
Partly Satisfactory
Satisfactory
Client Performance
Non-Financial Performance of the Company
Satisfactory
Satisfactory
10. LESSONS
Regulations for new and untested financial products need to be thoroughly discussed with the regulator
to avoid problems with their issuance. In this project, the regulation on Hybrid Tier I instrument had
never been tested, and there were some legal loopholes that needed to be taken care of before the
Project could proceed. Although the regulator understood the problems, it took considerable time and
effort to make the amendments.
The commitment of the main project shareholders to the project’s development objectives is the key to
achieving them. In this project, IFC expected that the Bank would diversify its geographic coverage of
lending from highly urban areas to less urban areas. The Bank did increase the number of branches, but
mainly through a merger with another bank that had branches mostly in urban areas.
Public Sector Application: World Bank Rural Roads Project
1. PROJECT INFORMATION
Project Name
Country
Borrower
Approval Date
Closing Date
Project Cost at Appraisal
Project Cost at Completion
WB Financing
Co-financing
Rural Roads Project
India
Government of India
23 September 2004
31 March 2012
US$ 310 million
US$ 312 million
US$400 million (IBRD loan US$100 million, IDA credit US$300 million)
None
Context
At the time of appraisal of the Rural Roads Project in 2004, the rural population in many states of India
was suffering from poor physical access due to a lack of all-weather roads. This was constraining
economic activities in rural areas and preventing the rural population from being fully integrated into the
economy and accessing essential services. At that time, an estimated 300,000 habitations (about 40
percent of the 825,000 habitations in India) were without all-weather road access. The low levels of
access of habitations to all-weather roads was partly a result of the lack of adequate maintenance on the
existing large rural road network of approximately 2.7 million km and partly a lack of capital investment.
In addition, the rural road sector was suffering from relatively low levels of technical and planning
- 51 -
capacity at the state level that hindered the ability of road agencies to absorb new investment funds and
manage their assets properly.
In order to address the problem of poor rural access and improve planning and management of funds, the
Prime Minister’s Rural Road Program (Pradhan Mantri Gram Sadak Yojana, or PMGSY)) was announced in
late 2000. By the time of project appraisal, PMGSY had completed or was close to completing works
connecting 22,100 habitations with a population per habitation of less than 1,000. This was against
PMGSY targets of providing new connectivity to about 178,000 habitations, involving construction of
about 375,000 km of roads, and upgrading 372,000 km of existing rural roads in poor condition.
Challenges faced at that time included a shortfall in funds, capacity constraints of road agencies and
contractors, and inadequate road maintenance.
Project Objectives
To achieve broader and more sustainable access to markets and social services in participating districts of
India (Loan Agreement Schedule 2, p. 19).
The main beneficiaries of the project were the people in the habitations that were to be connected by allweather roads. Anticipated project benefits included all-season access to economic opportunities and
social services, employment generation for new construction and maintenance, and improved access to
transport services. Participating states also would benefit indirectly from improvements in the economy
resulting from the project.
Project Description
The project had four components:

Component 1: New connection and upgrading of core rural roads networks (appraisal estimate
US$430.9 million; actual cost not available). New construction and upgrading of 9,900 km of the
core rural road network in four states (Himachal Pradesh, Jharkhand, Rajasthan, and Uttar
Pradesh) to provide all-weather road access to habitations in identified project districts. This
component would also finance consultants in each state to undertake independent technical
examination of WB-funded works and to verify compliance with relevant safeguards, technical
standards, and the quality of bidding documents.

Component 2: Periodic and routine maintenance of core rural road network in project areas
(appraisal estimate US$235 million, to be financed by state governments; actual cost not
available): Annual implementation of the periodic and routine maintenance program in every
district where the WB was to provide additional investment (about 100,000 km of roads).

Component 3: Institutional development (appraisal estimate US$13.6 million, actual cost not
available): Technical assistance for institutional development at both the national and state
levels.

Component 4: Incremental operating cost (appraisal estimate US$4.4 million; actual cost not
available).
- 52 -
2. RELEVANCE: SATISFACTORY
Strategic Alignment: Satisfactory
Project objectives remained consistent with the Country Assistance Strategies at appraisal and
completion. The FY 2009-2012 Country Assistance Strategy (CAS) aimed at achieving rapid, inclusive
growth, ensuring that development is sustainable, and increasing the effectiveness of service delivery. To
achieve rapid, inclusive growth, the CAS aimed to assist in removing infrastructure and skills constraints to
growth in both urban and rural areas.
Project objectives were in line with the Government’s priority of addressing the problem on poor rural
accessibility and improving the planning and management of rural roads. The Prime Minister’s Rural
Roads Program was launched in late 2000 and the project was part of that program. The Program target
was to connect all habitations of more than 1000 people (500 in the case of hilly, desert, and tribal areas)
by the year 2009 and upgrade 194,000 rural roads by the year 2010. These targets also were reflected in
India’s Eleventh Five Year Plan (2007-2012).
Rationale for IFI Involvement: Satisfactory
The project sought to introduce the notion of partly linking PMGSY grants to sound management of the
entire core rural road network. Further, by demonstrating the value of sector planning and management
reforms in the pilot states or districts in partnership with the Government of India (GOI), the project
sought to increase the likelihood that similar reforms would be implemented in other states and districts.
GOI had set itself very ambitious connectivity targets that required substantial additional resources if they
were to be met on time, and the World Bank, both through IDA and as needed through IBRD, had the
financial capacity to contribute to investment in this public good. Thus, the ex-ante financial additionality
of the project was positive.
3. RESULTS: PARTLY SATISFACTORY
Operational Performance: Satisfactory
Achievement of Outputs: Satisfactory

By project closure, the physical works for new construction were completed. About 9,625 km of
rural roads were constructed (97 percent of the target of 9,900 km), providing better access for
78 participating districts.

Institutional development was supported by the development of a pavement design manual;
independent reviews of engineering designs through technical examiners; and the preparation of
a rural roads manual by the Ministry of Roads and Development with separate technical
specifications for rural roads. However, not all of the changes were implemented, and the use of
cost-effective optimal designs employing locally-available material was inadequate.

The project provided training in maintenance management systems, quality control,
environmental management, and e-procurement.
- 53 -
Contribution to Intended Outcomes: Partly Satisfactory
The project contributed to increased access to markets and social services due to the construction of
roads. Overall, about 93 percent of eligible habitations were connected, compared to the target of 60
percent. About 50 percent of the routes were in good condition in 2012, compared to the target of 43
percent. However, routine maintenance funding is not fully assured, raising concern about the extent to
which “sustainable access” will be achieved (see also the Sustainability section). Due to inadequate
evidence on some of the intended outcomes (e.g., access to social services) and also on attribution of the
reported outcomes to the Project, Contribution to Intended Outcomes is rated Partly Satisfactory.
Information on access to social services and markets is limited to data on the Government’s entire rural
road program, rather than to the roads funded by the Project. The data show state-level achievements
that surpassed the Government’s targets.
Percent of Eligible Habitations with All-Weather Access to Social Services and Markets
Uttar Pradesh
Himachal Pradesh
Jharkhand
Rajasthan
Baseline (%)
50
40
35
40
Target (%)
55
60
60
65
Actual (%)
97
78
69
86
To evaluate the economic and social benefits of the program, the WB undertook a survey in the four
states supported by the Project. Road user satisfaction was assessed on the basis of four factors:
reliability, transit time, connectivity, and user-friendliness. Overall, the satisfaction level was high. The
survey also found:

Child vaccination increased by 8 percent due to improved access to health centers.

The number of female patients visiting private doctors increased 8 percent.

The population in the income category of “more than 10,000 Rupees per month” increased by
3.46 percent.

The area cultivated increased by 12 percent.

Travel time was reduced.
However, these improvements cannot be completely attributed to the Project.
Contribution to Corporate Goals: Not Rated
No other contributions were reported.
Unintended Outcomes: Partly Satisfactory
As noted below under Sustainability, in 2006 serious deficiencies in safeguard preparation in Uttar
Pradesh were discovered, and corrective measures were required on at least some of the roads financed
by the project. There is no evidence on the severity of environmental damage or the extent to which it
was reversed or mitigated.
- 54 -
The project’s training and capacity building activities have been mainstreamed. Simple maintenance
management systems were developed by Rajasthan, Himachal Pradesh, and Uttar Pradesh (but not by
Jharkhand), and these were used to prepare the annual maintenance plans for the respective states.
4. EFFICIENT USE OF RESOURCES: SATISFACTORY
Economic Efficiency: Satisfactory
The ex-post ERR was relatively high at 15 percent, although it was below the appraisal estimate of 19
percent. The lower-than-estimated ERR was due to increased construction cost and time overruns. Cost
escalation was very high in Himachal Pradesh (34.5 percent), followed by Uttar Pradesh (7.6 percent). The
ERR calculation assumed a 12 percent cost of capital and reinvestment rate.
Himachal Pradesh
Jharkhand
Rajasthan
Uttar Pradesh
Total
Completed Road Length
(km)
984
126
6,287
2,228
9625
Ex-Ante ERR
(%)
21
18
17
25
19
Ex-Post ERR
(%)
11
17
15
18
15
Implementation Efficiency: Partly Satisfactory
There were initial delays in the institutional strengthening and maintenance components. The project
closed after a two-year delay in order to: (i) consolidate the institutional development initiatives; (ii)
complete the ongoing and additional rural roads works reallocated to Uttar Pradesh (US$27 million) and
Rajasthan (US$9 million); and (iii) undertake further capacity building activities planned for rural roads
agencies.
5. SUSTAINABILITY: SOMEWHAT UNLIKELY
Outcome Sustainability: Somewhat Unlikely
Routine maintenance funding is not fully assured. Experience has been mixed with respect to the
maintenance of the core road network in the participating districts. Initially, maintenance of the core
road network was neglected, and it received attention mainly towards the last two years of the project.
Only after the Ministry of Rural Development stipulated to the States that construction funds would be
released only if the States would specifically earmark funds for maintenance, did the situation improve.
Routine maintenance until 2017 is funded under the 13th Finance Commission, which approved US$4.2
billion for rural roads. However, there is no information on whether this amount is adequate. Routine
maintenance will be carried out by the same contractor who constructed the roads under the Project.
However, there is no information on who will guarantee and supervise the contractor.
Compliance with Safeguards: Partly Satisfactory
The Project was assigned an Environment Category A. The following safeguards policies were triggered:
Environmental Assessment, Natural Habitats, Cultural Property, Involuntary Resettlement, Indigenous
Peoples, and Forests.
- 55 -
Environment: At appraisal, the Government carried out an environmental assessment in the four project
states and formulated an Environment and Social Management Framework (ESMF) to address
environmental issues in the Project. A stand-alone document called Environmental Codes of Practice was
prepared covering aspects such as sub-project selection and planning, construction camp management,
topsoil conservation, debris management, public consultation, drainage, and worker safety. This
approach helped minimize the need for sub-project level environmental assessments and environmental
management plans by mainstreaming environmental issues into the selection, planning, design, and
construction stages of the project.
In almost all of the participating states, some roads, especially in the initial phases, were delayed due to
lack of forest clearances. In subsequent phases, with the insistence of having a prior forestry clearance in
hand before the bids were invited, such situations were almost completely avoided.
Since the construction of rural roads was on existing alignments, the extent of impact on environmental
features such as fertile farmland, orchards, trees, sacred groves, water bodies, and religious structures
was minimal.
Social: At appraisal, the Government carried out a social assessment in all of the Project states through an
independent consulting agency. The assessment found minimal land requirements for road widening, and
community willingness to volunteer for land donation. The ESMF adopted a voluntary land donation
approach.
Following the identification of serious deficiencies in safeguard preparation in Uttar Pradesh in 2006, the
WB conducted a complete audit of social safeguards preparation and implementation for all project roads
and developed a time-bound action plan for retrospective corrective measures. A series of training
programs for field officers were conducted at the Divisional level.
Indigenous Peoples: There is no information on compliance with Indigenous Peoples safeguards.
Compliance with Safeguards is rated Partly Satisfactory rather than Satisfactory due to a lack of
information on compliance with Indigenous Peoples safeguards.
6. OVERALL PROJECT PERFORMANCE
Based on a Satisfactory rating for Relevance, Partly Satisfactory for Results, Satisfactory for Efficiency, and
Somewhat Unlikely for Sustainability, overall project performance is rated Partly Satisfactory.
7. IFI PERFORMANCE: PARTLY SATISFACTORY
Quality at Entry: Partly Satisfactory
Project objectives were clearly stated, and there was a logical link between project objectives and project
activities. Project preparation involved extensive public participation. The project was technically sound
and benefitted from the WB’s experience with the implementation of the Government’s rural road
program in other states. Safeguards assessment was adequate. However, the design of the monitoring
and evaluation framework was weak, as the performance indicators referred to the performance of the
Government’s program as whole rather than the performance of the part supported by the Project.
- 56 -
Some risks were underestimated, including the risk of poor road designs and the risk of delayed or
improper procurement by rural road agencies. In the case of Jharkhand, the planned construction of
roads was substantially reduced because of weak capacity and resulting delays, which should have been
foreseen. Another risk which was not identified during preparation was the reluctance of states to use
WB funds when funds could also be accessed through the Prime Minister’s Rural Roads Program, which
had fewer requirements in terms of safeguard and fiduciary compliance.
Quality of Supervision: Partly Satisfactory
A total of 17 supervision missions were carried out, on average twice in each fiscal year. The supervision
missions identified issues which were likely to affect the achievement of development outcomes. With
the help of close monitoring by the WB, some states showed substantial improvement. Supervision of
safeguard and fiduciary aspects was satisfactory. However, an earlier reallocation of funds from
Jharkhand to Uttar Pradesh and Rajasthan would have made an extension of the closing date
unnecessary. Also, when funds were reallocated, corresponding targets were not revised, which made it
difficult to monitor the performance of different states.
8. CLIENT PERFORMANCE: PARTLY SATISFACTORY
Government Performance: Satisfactory
The Government was fully committed to the project. Detailed guidelines were prepared by the Ministry
of Rural Development for planning, standards and specifications, and procurement. The Ministry of Rural
Development and the National Rural Roads Development Agency provided close monitoring whenever
slow decision-making threatened to delay certain activities. For example, the initially slow procurement,
poor compliance with safeguards, and slow roll-out of the computerized financial management system in
states such as Uttar Pradesh, Himachal Pradesh, and Jharkhand were met with increased pressure and
resources from the National Rural Roads Development Agency. The Agency undertook state-by-state
monitoring reviews and provided further training, improved documentation, and stricter enforcement.
Implementing Agency Performance: Partly Satisfactory
Of the four states, the performance of the State Road Agencies in Rajasthan and Himachal Pradesh was
satisfactory. Uttar Pradesh initially had frequent changes in senior management, inadequate staffing, and
weak coordination among agencies, but managed to overcome these problems with assistance from the
WB. The implementing capacity of Jharkhand was unsatisfactory from the beginning, especially with
respect to institutional capacity and a willingness to resolve implementation problems in a timely manner.
Compliance with fiduciary requirements as well as inadequate ownership and staffing remained
problematic. Eventually, in September 2009, proceeds earmarked for Jharkhand were reallocated to
Rajasthan and Uttar Pradesh.
Implementation of the Environment and Social Management Framework was weak in Uttar Pradesh.
Following actions by the WB and the Government, there was a substantial improvement in safeguards
compliance. Rajasthan and Himachal Pradesh performed well on social safeguards implementation.
Internal and external audit reports found weaknesses with respect to delays in WB reconciliations,
inadequate controls over fund authorization, stale bank guarantees, and non-settlement of advances.
The states took remedial actions to address these weaknesses, but the results varied across states. An
Interim Performance Review carried out by the WB suggested mitigation measures for weak contract
management.
- 57 -
Procurement capacity varied considerably among the states. Rajasthan demonstrated adequate
procurement capacity, but procurement in Jharkhand remained slow throughout the project. Goods,
works, and services were generally procured in accordance with WB guidelines except for some
procedural errors in the procurement of works.
9. RATINGS SUMMARY
Relevance
Strategic Alignment
Rationale for IFI Involvement
Satisfactory
Satisfactory
Satisfactory
Results
Operational Performance
Contribution to Intended Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Partly Satisfactory
Satisfactory
Partly Satisfactory
Not Rated
Partly Satisfactory
Efficient Use of Resources
Economic Efficiency
Implementation Efficiency
Satisfactory
Satisfactory
Partly Satisfactory
Sustainability
Outcome Sustainability
Compliance with Safeguards
Somewhat UnLikely
Somewhat Unlikely
Partly Satisfactory
Overall Project Performance
Partly Satisfactory
IFI Performance
Quality at Entry
Quality of Supervision
Partly Satisfactory
Partly Satisfactory
Partly Satisfactory
Client Performance
Government Performance
Implementing Agency Performance
Partly Satisfactory
Satisfactory
Partly Satisfactory
10. LESSONS
In projects that fund a small portion of a large government program, it is important to agree on
harmonized standards for fiduciary and safeguard requirements across the Program and the Project. In
this project, differences in procurement guidelines between the WB and the Prime Minister’s Rural Roads
Program caused problems in implementation.
In projects that are to be implemented in different states or regions with very different implementation
capacities, project design should build in some flexibility to allow tailoring of technical assistance to the
needs of each state. This should be based on a critical analysis of the strengths and weaknesses in
institutional capacity at the state level.
- 58 -
Annex 1: Glossary of Terms and Definitions
Term
Definition
Aggregate Project Performance
Indicator
In the Public GPS, a single measure of overall project performance
constructed from ratings on the core evaluation criteria (Relevance,
Effectiveness, Efficiency, and Sustainability).
Benchmark
Reference point or standard against which performance or achievements
can be assessed (OECD-DAC).
Borrower Performance
The adequacy of the Borrower’s assumption of ownership and
responsibility during all project phases, including government and
implementing agency performance in ensuring quality preparation and
implementation, compliance with covenants and agreements, establishing
the basis for sustainability, and fostering participation by the project’s
stakeholders.
Broad economic and social
goals
Sector-wide and/or economy-wide goals that are not included in the
project’s statement of objectives but nevertheless are of interest in the
evaluation.
Completion Report
A record of an operation’s performance at the end of its implementation
phase, undertaken as a self-evaluation by an IFI operations unit.
Completion Report Validation
A review of Completion Report findings by the Central Evaluation
Department, normally as a desk study.
Core Criteria
In the Public GPS, the principal criteria that form the basis for evaluating
project performance. For evaluations of investment/TA loans, the core
criteria are Relevance, Effectiveness, Efficiency, and Sustainability. For
evaluations of PBLs, the core criteria are Relevance, Effectiveness, and
Sustainability.
Corporate goals
Areas of special focus of the IFI, such as poverty reduction, rural poverty
reduction, transition impact, social cohesion, gender equality, shared
prosperity, inclusive growth, environmental sustainability, EU policy, propoor innovation, etc.
Cost-benefit analysis
A quantitative analysis performed to establish whether the present value of
benefits of a given project exceeds the present value of costs.
Cost-effectiveness analysis 1.
A quantitative analysis that compares the relative costs and outcomes of
two or more courses of action. Cost-effectiveness analysis can be used to
show whether the outcomes were delivered at least cost compared to
alternative ways of achieving the same outcomes.
Direct evaluation
Evaluations undertaken directly by the CED, as opposed to indirectly by the
IFI. Equivalent to independent evaluation in the Public GPS.
Economic Rate of Return (ERR)
The internal rate of return of the time series of the project’s economic
costs and benefits. Measures the quantifiable net economic benefits to
society. In addition to those identified in financial analysis, may include
taxes paid to the government, consumer surplus, effects on competitors,
and benefits to suppliers, including labor. Also includes externalities to the
extent that they can be quantified.
Economic Return on Invested
Capital (EROIC)
The internal rate of return on the economic costs and benefits of a project,
including costs and benefits to customers, employees, government,
suppliers, competitors, local residents, etc. The EROIC is calculated only
when the ERR on the project cannot be calculated. It is calculated by
- 59 -
Term
Definition
adjusting the ROIC for factors normally taken into consideration when
adjusting the FRR to the ERR.
Effectiveness
The extent to which the project achieved (or is expected to achieve) its
stated objectives, taking into account their relative importance (Public GPS,
OECD-DAC)
Efficiency
The extent to which the project has converted its resources economically
into results (Public GPS, OECD-DAC).
Evaluability
The extent to which the value generated or the expected results of a
project are verifiable in a reliable and credible fashion.
Extended Annual Supervision
Report (XASR)
A stand-alone, ex-post private sector project evaluation conducted by an IFI
operations department (Private GPS)
Extended Annual Supervision
Report Assessment (XASR-A)
A CED validation of an XASR (Private GPS).
Financial Rate of Return (FRR)
The internal rate of return of a time series of real, after-tax cashflows
describing the project’s financial investments and returns.
Fund Weighted Average Cost of
Capital (FWACC)
The cost of capital for a private equity or listed fund, estimated by
calculating the average cost of debt based on the country composition of
the fund, and then levying a premium for the combined equity instrument
and project risk.
Goal
The higher-order objective to which a development intervention is
intended to contribute (OECD-DAC).
IFI Performance
The quality of services provided by the IFI during all project phases,
including the IFI’s performance in ensuring project quality at entry,
satisfactory implementation, and future operation (Public GPS).
Impact
Positive and negative, primary and secondary long-term effects produced
by a development institution, directly or indirectly, intended or unintended
(OECD-DAC).
Impact evaluation
An evaluation that quantifies the net change in outcomes that can be
attributed to a specific project or program, usually by the construction of a
plausible counterfactual.
Independent evaluation
In the Public GPS, a project evaluation conducted by the CED. Equivalent to
a “direct evaluation” in the Private GPS. Independent evaluation
instruments include Performance Evaluation Reports (PERs) and Expanded
Annual Supervision Report Assessments (XASRs) for private sector
operations, and Performance Evaluation Reports (PERs) and Completion
Report Validations (CR Validations) for public sector operations.
International Financial
Institution
A financial institution created by a group of countries that provides
financing and advisory services for projects and programs in member
countries. The term includes the World Bank, regional development banks,
and other regional financial institutions. It does not include internationallyoperating commercial banks.
Net present value
The sum of the present values of the time series of project costs and
benefits.
Outcome
The final level in the results chain, reflecting the objectives of the project.
If necessary, a distinction can be made between “intermediate outcomes”
- 60 -
Term
Definition
(the uptake of project outputs by beneficiaries, behavioral changes on the
part of beneficiaries, results achieved once beneficiaries use project
outputs, and/or results achieved in the short-to-medium term) and “final
outcomes” (the ultimate outcomes or project goals arising from
intermediate outcomes). “Final outcomes” are called “impacts” by some
IFIs.
Output
A product, capital good, or service which results from an IFI intervention
(OECD-DAC).
Output
The tangible goods and services that the project activities produce,
generally under the direct control of the implementing agency (OECD-DAC).
Performance Evaluation Report
An independent project evaluation conducted by a Central Evaluation
Department, which normally includes field work (Public GPS, Private GPS).
Project
A public or private sector investment, technical assistance, or program that
is supported by an IFI loan or grant.
Project Outcome Rating
In the Private GPS, a synthesis rating reflecting the ratings on the core
criteria of Financial Performance and Fulfillment of Project Business
Objectives, Economic Sustainability, Contribution to IFI Mandate
Objectives, and Environmental and Social Performance.
Relevance
Consistency of the project’s objectives with beneficiary needs, the
country’s development or policy priorities and strategy, and the IFI’s
assistance strategy and corporate goals.
Results
The output, outcome or impact (intended or unintended, positive and/or
negative) of a development intervention (OECD-DAC).
Results chain
A model that sets out the sequence of inputs, activities, and outputs that
are expected to lead to the project’s intended outcomes. Describes the
causal relationships, indicators, and the assumptions or risks that may
influence project success and failure. Alternatively called a “results
framework”, “causal chain”, or “logical framework (logframe)”.
Return on Invested Capital
(ROIC)
The internal rate of return on invested capital in real terms, i.e., the FRR on
the costs and benefits to the company as a whole on a before-after, rather
than a with-without, basis.
Self evaluation
In the Public GPS, a project evaluation conducted by IFI Management.
Equivalent to an “indirect evaluation” in the Private GPS. Self evaluation
instruments include Completion Reports (CRs) for public sector operations
and Expanded Annual Supervision Reports (XASRs) for private sector
operations.
Sustainability
The likelihood of continued long-term benefits, and the resilience to risk of
net benefit flows over time (OECD-DAC).
Theory-based evaluation
An analysis that establishes a plausible association between the various
links in the project’s results chain, using quantitative and qualitative
evidence as well as evidence from other evaluations and academic
literature.
Weighted Average Cost of
Capital (WACC)
The weighted average after-tax cost to the company of the yields it must
provide on its borrowings and the equity investors’ minimally acceptable
terms, all adjusted for inflation.
- 61 -
Annex 2: Recent Developments in ECG Members
European Investment Bank
Rather than publishing evaluations of individual projects, the Evaluation Department (EV) at EIB
conducts thematic evaluations that contain evaluations of each of the projects covered in the
thematic report. Some of these thematic evaluations cover both public sector and private
sector operations. EV uses a common set of evaluation criteria that apply to both types of
evaluations.
Although EV’s evaluation methodology has not yet been formalized in the form of written
guidelines, it can be seen in three recent thematic evaluations: Evaluation of EIB Investment
Loans for Economic and Social Cohesion in France, Portugal, and the United Kingdom (“Cohesion
evaluation”, April 2011); Evaluation of EIB’s Energy Efficiency Financing in the EU, 20002011(“Energy Efficiency evaluation”, May 2012); and Evaluation of EIB’s Intermediated Lending
to SMEs (“SME evaluation”, June 2012). A review of pairs of projects from each of these three
thematic evaluations – one public sector, one private sector – reveals how the evaluation
criteria and benchmarks were defined. The six projects were:

Cohesion evaluation, public sector: Construction of new waste incinerator and a
storage and transfer facility for organic waste. Financed by an EIB loan to a municipal
government agency.

Cohesion evaluation, private sector: Redevelopment of a former military airfield into a
commercial airport and associated business and commercial development. Financed by
an EIB loan to a private company.

Energy Efficiency evaluation, public sector: regional investments in solar photovoltaic
equipment on the roofs of public buildings; other renewable energy installations
including solar water heaters and other renewable energy sources such as biomass or
geothermal; equipment to improve energy efficiency in public buildings, as well as in
street and traffic lighting. Financed by an EIB loan to a regional government and
intermediated by participating financial institutions.

Energy Efficiency evaluation, private sector: development of remote digital electricity
metering infrastructure throughout the client country, replacing existing
electromechanical equipment, thereby allowing remote reading and several interactive
services. The project included investments in data network and management centers.
Financed by an EIB loan to a private power distribution company.

SME evaluation, public sector: credit to SMEs throughout the client country, in
particular for energy efficiency and renewable energy sources and for restarting
entrepreneurs after a period of inactivity. Financed by an EIB global loan to a majority
state-owned bank.
- 62 -

SME evaluation, private sector: credit to SMEs and microenterprises. Financed by an
EIB global loan to a private credit and leasing company.
Table A1 shows the definitions of the evaluation criteria that were applied to both public and
private sector operations.
Table A1: EIB Evaluation Criteria in Six Project Evaluations
Criterion
Definition
Relevance
The extent to which the objectives of a project are consistent with EU policies; EIB
priorities; national, regional, and local strategies; and beneficiaries’ needs.
Relevance also takes account of the internal coherence of objectives and the
relevance of the design. It also covers the identification of risks and measures to
mitigate them.
Effectiveness
The extent to which the objectives of the project have been achieved, or are
expected to be achieved, taking into account their relative importance, while
recognizing any change introduced in the project since loan approval.
Effectiveness considers the achievement of operational objectives (“outputs”),
intermediate objectives (“results” or “outcomes”) and global objectives (direct
and indirect effects following from the outcomes, corresponding to, often longerterm, “impacts”). For example, in the Energy Efficiency evaluation of a private
sector project, the operational objectives were modernizing the company’s
electricity monitoring; intermediate objectives were a reduction in operational
costs and an improved competitive position; and global objectives were to
develop a building block for smart grids, reduce greenhouse gas emissions,
maximize the effects of market liberalization, and support regional development.
Efficiency
The extent to which project benefits/outputs are commensurate with
resources/inputs.
At appraisal, project efficiency is normally measured through the financial and
economic rates of return. In public sector projects a financial rate of return is
often not calculated at appraisal, in which case the efficiency of the project is
estimated by cost effectiveness analysis.
Efficiency also includes such aspects as the project’s physical implementation,
schedule, and procurement; costs and financing; operation (management, use,
and employment); and cooperation and coordination with other counterparts.
Sustainability
The likelihood of continued long-term benefits and the resilience to risk over the
intended life of the project.
The assessment covers physical and operational sustainability; organizational,
institutional and regulatory sustainability; environmental and social sustainability;
and financial and economic sustainability.
Environmental,
Economic, and Social
Impact
Environmental impact considers two aspects:


Compliance with environmental safeguards, including EU and/or national as
well as EIB guidelines.
Environmental impact beyond safeguards compliance.
- 63 -
Criterion
Definition
Economic and social impact includes, for example, employment created outside of
the project, the creation of new businesses, etc.
EIB Contribution
EIB Contribution considers two aspects:


EIB Management of
the Project Cycle
(or “EIB
Management”)
EIB Financial Contribution -- the financial contribution provided in relation to
the alternatives available, including facilitating co-financing from other
sources (catalytic effect).
Other EIB Contribution (optional) relates to any significant non-financial
contribution to the operation provided by EIB – e.g., improvements in the
technical, institutional, economic, or other aspects of the project.
EIB’s handling of the operation, from project identification and selection to postcompletion monitoring.
An overall project performance rating combines the ratings on Relevance, Effectiveness,
Efficiency, and Sustainability. The Environmental, Economic, and Social Impact criterion is rated
“below the line”.
All criteria are rated on a four-point scale:

Relevance, Effectiveness, Efficiency, Sustainability, are rated as Excellent, Satisfactory,
Partly Unsatisfactory, and Unsatisfactory. The overall assessment reflects the individual
assessments within the same scale.

Individual assessments on EIB Contribution are rated as High, Significant, Moderate, and
Low.

Individual assessments on EIB Management of the Project Cycle are rated Excellent,
Satisfactory, Partly Unsatisfactory, and Unsatisfactory.
Inter-American Development Bank
IDB recently re-designed their approach to development effectiveness for IDB Non-Sovereign
Guarantee (NSG) operations.7 The new approach is aligned with the broader discussion
regarding the IDB Group’s private sector operations, where development effectiveness and the
measurement of development impact are placed front and center. The envisioned changes are
intended to “raise the bar” for NSG projects by requiring broader and deeper analysis of project
logic, sector context, and other factors.
7
IDB (2013). Revised Development Effectiveness Framework for Non-Sovereign Guarantee Projects.
Revised version. Policy and Evaluation Committee, 28 June. Private sector operations in the IDB Group
are also provided by the Inter-American Investment Corporation (IIC) and the Multilateral Investment
Facility (MIF).
- 64 -
The previous Development Effectiveness Matrix (DEM) was designed to improve project
evaluability at entry, monitoring during implementation, and evaluation at completion.
Experience with the DEM revealed some shortcomings. The DEM scoring system created
incentives for project teams to try to score as many sub-sections as possible, regardless of their
importance for the project. This lessened the focus and relevance of the DEM. In addition, the
DEM’s information requirements placed a burden on IDB staff, suggesting that a more
streamlined approach was needed.
The new Development Effectiveness (DE) Toolkit attempts to harmonize the evaluability
approach between IDB’s NSG and Sovereign Guarantee (SG) projects. The DE Toolkit has four
components: (i) a DEM Summary; (ii) a DEM Worksheet; (iii) an Evaluability checklist; and (iv) a
Results Matrix.
The DEM Worksheet has three main sections that correspond to the evaluation criteria in the
ECG Private Sector GPS, fourth edition:

Development Outcome, based on (i) Project/Company Business Performance; (ii)
Economic Sustainability; (iii) Private Sector Development; and (iv) Environmental and
Social Policy Assessment.

Additionality, consisting of (i) Financial Additionality and (ii) Non-Financial Additionality.

Strategic Alignment, against IDB Strategic Development Objectives and Country Strategy
Development Objectives.
Instead of performance area scores being calculated as a weighted average of all possible subsections, the DEM performance areas are to be scored based only on factors or sub-sections
that are most relevant within each performance area. This is intended to provide a clearer
picture of each project’s intended development results.
The Evaluability Checklist has three sections:

the justification for IDB intervention, the project’s developmental objectives, how the
project addresses the identified market failure or development problem, and the
intended beneficiaries. It assesses the quality of the empirical evidence provided to
support the intervention, and a verification that the vertical logic of the project holds.

an assessment of the quality of the project’s financial and economic analysis, in
accordance with the standards of the ECG GPS. For real sector projects, ex ante costbenefit analysis is carried out to calculate FRR and ERR (or ROIC and EROIC); for financial
intermediary projects, the analysis includes the financial performance of financial
institutions as well as the economic sustainability of their portfolio.

an assessment of the monitoring and evaluation plan, in order to ensure meaningful ex
post evaluation as well as tracking and reporting on development results. The list of
indicators includes not only those in the DEM and Results Matrix, but also the tracking
indicators useful for monitoring the achievement of IDB corporate goals.
- 65 -
The Results Matrix includes output and outcome indicators with baseline and target values. The
Results Matrix allows for a sharpened focus on those indicators that are most relevant for
measuring the project’s achievements of their stated development objectives.
The DEM Summary contains summary scores from the DEM Worksheet and the Evaluability
Checklist.
The DE Toolkit is to be supported by IT solutions to increase efficiency. All information and data
points are to be entered only once, and then are automatically fed into different tables including
the DEM Summary and the Results Matrix.
DEMs and DEM scores are to be updated annually as part of NSG Project Supervision Reports.
Monitoring activities will focus mainly on the tracking of performance indicators included in the
Results Matrix, which will be reflected mainly in the updated DEM scores mostly for
Project/Company Business Performance and Economic Sustainability. For other areas, any
major events or milestones will be tracked, and if there is any difference between actual and
expected project performance, the corresponding DEM scores will be updated.
European Bank for Reconstruction and Development
EBRD’s Evaluation Department (EvD) is currently preparing a guidance note on EBRD’s
evaluation performance rating system. The objective is to strengthen the existing project
performance rating system and extend it to a wider range of evaluation products. At the time of
the writing of the present paper, the guidance note was still being written. However, some of
its features can be gleaned from the Approach Paper8 and from conversations with EvD
management.
Among the principles underlying the new evaluation performance rating system are the
following:

The rating system should provide a robust measure of performance. This means that
the criteria and sub-criteria included in the overall performance assessment are those
that are directly attributable (or largely attributable) to EBRD. Those criteria to which
the operation contributes but is not solely or largely responsible, or which are the
consequence of success rather than a determinant of it, may be assessed but not
included in the overall performance rating.

To the extent possible, a common framework should apply to all types of evaluation
that rate performance (including self evaluation and independent evaluation) as well as
all objects of evaluation – investments, frameworks, technical cooperation (TC), non-TC
grants, strategies, and policies. There may be some differentiation and customization at
lower levels (sub-criteria and benchmarks) and to the weights used in aggregating
criteria ratings to the overall performance rating.
8
EBRD (2013). Guidance Note: Evaluation Performance Rating System. Approach Paper. Evaluation
Department, March.
- 66 -

The rating methodology should follow internationally accepted good practice standards,
principally those of ECG and the Evaluation Network of OECD-DAC.

Guidance should indicate the type of evidence that should be provided.

The evaluation criteria should reflect EBRD’s unique mandate
EvD currently uses seven evaluation criteria for the performance rating of operations, two of
which have sub-criteria:
Transition Impact
Realized TI at the time of evaluation
TI that can still be achieved
Risk attached to achieving remaining TI
Environmental and social performance and change
Environmental and social performance of the project and the sponsor
Extent of environmental and social change
Additionality
Project and Company Financial Performance
Fulfillment of Project Objectives
EBRD Investment Performance
Bank Handling of the Project
The Approach Paper suggests that a mapping of the existing criteria to the standard OECD-DAC
criteria could be a useful starting point, and gives an example of how to do this:
Relevance
Effectiveness
Efficiency
Sustainability
Impact
Additionality
Fulfillment of Project Objectives
Outputs and outcomes associated with transition,
environmental, and social impact that are largely
attributable to the project (mostly client-level transition
impact, and extent of environmental and social change,
and environmental and social performance of the
project).
Project and company financial performance
FIRR and/or other financial ratios demonstrating
efficient funds use
Bank investment performance
Bank handling in terms of efficiency of process
Environmental and social sustainability (rather than the
sustainability of results). However, it would be possible
to consider company financial performance at the time
of evaluation under “effectiveness” and projected future
financial performance as evidence of the sustainability of
results.
Remaining transition impact and the risk to achieving
this could be included under Sustainability.
Transition Impact at the sector and economy-wide levels
Environmental and social performance of the sponsor
- 67 -