Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Approaches to Harmonizing Public and Private Sector Project Evaluation Draft: October 21, 2013 Acronyms CAS CED CPS DAC DE DEM E&S EBRD ECG EIB EP ERMP EROIC ERR ESMF ESMS EV EvD FRR GOI GPS ICRR IDB IEG IFC IFI ILO IT M&E NPV NSG OECD-DAC OP PMGSY QAE ROAA ROAE ROIC SEMS SG SME TC WACC WB WBG Country Assistance Strategy Central Evaluation Department Country Partnership Strategy Development Assistance Committee (of OECD) Development Effectiveness Development Effectiveness Matrix Environmental and Social European Bank for Reconstruction and Development Evaluation Cooperation Group European Investment Bank Evaluation Principle Environmental Risk Management Plan Economic Return on Invested Capital Economic Rate of Return Environmental and Social Management Framework Environmental and Social Management System Evaluation Department (EIB) Evaluation Department (EBRD) Financial Rate of Return Government of India Good Practice Standards Implementation Completion and Results Report (WBG) Inter-American Development Bank Independent Evaluation Group (WBG) International Finance Corporation International Financial Institution International Labor Organization Information Technology Monitoring and Evaluation Net Present Value Non-Sovereign Guarantee Organization for Economic Cooperation and Development – Development Assistance Committee Operational Practice Pradhan mantra Gram Sadak Yojana (Prime Minister’s Rural Road Program) (India) Quality at Entry Return on Average Assets Return on Average Equity Return on Invested Capital Social and Environmental Management System Sovereign Guarantee Small- and Medium-Scale Enterprise Technical Cooperation Weighted Average Cost of Capital World Bank World Bank Group Contents 1. Introduction ............................................................................................................................. 1 2. Similarities and Differences Between the Public GPS and Private GPS ................................... 3 3. Option 1: A Common Reporting Framework .......................................................................... 9 Mapping the Existing GPS Criteria to OECD-DAC Criteria........................................................ 9 Private Sector Application: IFC Telecommunications Project .............................................. 15 Public Sector Application: World Bank Telecommunications Project .................................. 18 4. Option 2: A Common Evaluation Methodology ................................................................... 21 The Basis of the Evaluation.................................................................................................... 23 Evaluation Criteria ................................................................................................................. 24 Evidentiary Requirements, Analytical Methods, and Benchmarks ....................................... 31 Private Sector Application: IFC Tier I Bank Financing Project ............................................... 42 Public Sector Application: World Bank Rural Roads Project................................................. 51 Annex 1: Glossary of Terms and Definitions................................................................................. 59 Annex 2: Recent Developments in ECG Members ....................................................................... 62 Tables Table 1: Evaluation Criteria in the Public GPS ................................................................................ 4 Table 2: Evaluation Criteria in the Private GPS............................................................................... 5 Table 3: Mapping the Existing GPS Criteria to OECD-DAC Criteria ............................................... 12 Table 4: The Basis of the Evaluation ............................................................................................. 23 Table 5: Option 2 Criteria and Sub-Criteria .................................................................................. 28 Table 6: Evidentiary Requirements, Analytical Methods, and Benchmarks ................................ 33 Figures Figure 1: Option 1 Criteria and Sub-Criteria ................................................................................. 15 Figure 2: Option 2 Criteria and Sub-Criteria ................................................................................. 27 This paper was prepared for the Evaluation Cooperation Group meeting in October 2013 by Kris Hallberg (consultant and lead author) and Nicholas Burke (contributing consultant) under the direction of Cheryl Gray (Director, Office of Evaluation and Oversight, Inter-American Development Bank) and Marvin Taylor-Dormond (Director of Private Sector Evaluation, Independent Evaluation Group, World Bank Group). 1. Introduction 1.1 Members of the Evaluation Cooperation Group (ECG) have worked over many years to harmonize their evaluation methodologies through the development and application of Good Practice Standards (GPS) for evaluation. GPSs have been developed in several areas, including for the evaluation of individual public sector and private sector operations. The latest versions of the Private GPS and Public GPS were approved by ECG in 2011 and 2012 respectively.1 1.2 While the Public and Private GPSs have been instrumental in moving forward the evaluation agendas and practices in many IFIs, the Boards of Directors of some IFIs have questioned the reasons for the differences in public and private sector evaluation approaches and have asked whether it might be possible to attain greater consistency among them. In addition, comparisons are inevitably drawn between the aggregate evaluation results of public sector and private sector operations when they are reported together in periodic Central Evaluation Department (CED) synthesis reports, inviting the question of whether differences in aggregate ratings are due to actual differences in performance or to differences in the way these operations are evaluated. And in sector, thematic, and country evaluations, CEDs often find themselves evaluating public and private sector operations using different criteria, making it difficult to present comparable results. 1.3 For these reasons, several ECG members have been experimenting with more harmonized approaches to evaluating public sector and private sector operations. The Inter-American Development Bank (IDB) has created a development effectiveness framework for private sector operations. The European Bank for Reconstruction and Development (EBRD) is preparing a guidance note on evaluation criteria that can be applied to both public and private sector projects. The European Investment Bank (EIB) is using a common set of evaluation criteria for public and private sector projects in its thematic evaluations. 1.4 At the April 2013 ECG meeting, Nicholas Burke (Consultant) presented a paper describing areas of commonality and divergence between the Public GPS and Private GPS.2 The paper found that, for the most part, differences between the two GPSs reflect the unique features of public and private sector interventions, the IFI’s corresponding institutional mandates, their operational structures, their instruments of support, and the nature of their client relationships. In another session at the same meeting, Alejandro Soriano (IDB) and Kris Hallberg (Consultant) applied the Public GPS and Private GPS to projects with a financial intermediary that had received both sovereign guaranteed and non-sovereign guaranteed loans from IDB. It was found that the different criteria and evidentiary requirements in the two GPSs could potentially lead to different assessments and ratings. 1.5 Using these efforts as a starting point, this paper proposes ways to reduce the divergence in how public sector and private sector evaluations are prepared and reported. The proposed 1 Evaluation Cooperation Group (2011). Good Practice Standards for the Evaluation of Private Sector Operations. Fourth Edition, November 8; and Evaluation Cooperation Group (2012). Good Practice Standards for the Evaluation of Public Sector Operations. 2012 Revised Edition, February. 2 Burke, Nicholas (2013). The Private and Public Sector Good Practice Standards: A Background and Comparison. Paper presented at the April 2013 ECG meeting. -1- approaches apply to the evaluation of individual projects, rather than higher-level evaluations at the sector, thematic, or country level, although there may be applicability to these evaluation products as well. As is the case in the Public GPS and Private GPS, the paper focuses on investment operations as opposed to non-lending advisory services and technical assistance. Applicability to policy-based lending has not yet been tested. 1.6 ECG members differ in their visions of what a common approach would look like. Some have expressed a desire to develop a “common reporting framework”. This would involve mapping the existing criteria in the Public and Private GPSs to common headings – for example, to the OECD-DAC evaluation criteria – with “modules” or “branches” to the existing criteria in the two GPSs. A common reporting framework, called Option 1 in this paper, would not require significant changes to the criteria or definitions in the current GPSs. 1.7 Other ECG members would like to go further to define a “common methodology” -- including criteria, guidelines for their application, ratings scales, and evidentiary requirements -- that could be applied to both public and private sector operations. A common methodology, called Option 2 in this paper, would involve changes to the standards in both GPSs, some of them significant.3 1.8 Even under a common methodology, some differences in approach may be warranted to accommodate the unique characteristics of public and private sector projects. In these cases, branches to the different approaches are presented. In addition, in areas in which there may be more than one approach to solve a problem, the paper presents the options and discusses the advantages and disadvantages of each. 1.9 It should be noted that differences in evaluation results between public sector and private sector operations may arise not just because of differences in evaluation methodologies, but also because of differences in the rigor with which those methodologies are applied in practice. Determining the extent to which the GPSs are being applied would require benchmarking a representative sample of evaluations from all ECG members, which is outside the scope of this paper. Instead, the paper focuses only on differences in the standards contained in the two GPSs. 1.10 The next section of the paper discusses the key similarities and differences between the Public GPS and Private GPS, and the relative strengths and weaknesses of each. Option 1 (a common reporting framework) and Option 2 (a common evaluation methodology) are presented with a discussion of the advantages and disadvantages of each. For illustration, both of these options are applied to examples of private sector and public sector projects. 1.11 Annexes provide definitions of terms used (Annex 1) and recent efforts of ECG members to harmonize evaluation criteria (Annex 2). 3 Other options could be considered, such as the so-called “Option 3” which proposes to address the weaknesses and gaps in the existing GPSs and be implemented in combination with Option 1. -2- 2. Similarities and Differences Between the Public GPS and Private GPS 2.1 The Public GPS and the Private GPS are already harmonized or mostly harmonized in a number of areas. The main area of full harmonization is the independence of the CED. The Private GPS contains a section on independence which is similar in content to the separate GPS on CED independence.4 The Public GPS does not have its own section on independence, instead referring to the Independence GPS. 2.2 Areas related to CED reporting, dissemination, and learning from evaluations are substantially harmonized. These include standards on the CED’s delivery of synthesis reports to the Board, CED reports on the quality of the IFI’s self evaluation system, the dissemination of evaluation results to a wide range of audiences, and CED monitoring of IFI Management’s implementation of CED recommendations. With respect to disclosure, the Public and Private GPSs have a similar standard, namely that the CED’s disclosure policy is explicit and is consistent with the IFI’s general disclosure policy. In practice, to protect the proprietary information of private sector clients, few evaluations of private sector operations are disclosed, and if they are, it is in a redacted or summarized form. In terms of evaluation processes and report preparation, the Public GPS and the Private GPS are mostly similar. 2.3 In other areas, there are notable differences in standards between the two GPSs. In order of importance for the comparability of evaluation results, the most significant differences are: 2.4 Evaluation criteria: The evaluation criteria currently used in the Public GPS and the Private GPS are shown in Tables 1 and 2. There are differences in the terminology and definitions of criteria as well as the methodology of the assessment. The Public GPS core criteria correspond directly to the OECD-DAC evaluation criteria of Relevance, Effectiveness, Efficiency, and Sustainability. The correspondence of the Private GPS criteria to OECD-DAC is less obvious, although there is at least partial correspondence of Financial Performance, Economic Sustainability, Contribution to IFI Mandate Objectives, and Environmental and Social Performance. Both GPSs contain criteria to assess IFI performance. In the Public GPS, the performance of the client is assessed separately in a single criterion (Borrower Performance). In the Private GPS, the performance of the client is manifest in the financial performance and compliance of the project company, and is therefore considered in the GPS criteria of Financial Performance and Environmental and Social Performance. 4 Evaluation Cooperation Group (2010). Good Practice Standards on the Independence of Central Evaluation Departments. June. -3- Table 1: Evaluation Criteria in the Public GPS Criteria and Sub-Criteria Relevance Effectiveness Efficiency Definition Consistency of the project’s objectives with beneficiary needs, the country’s development or policy priorities and strategy, and the IFI’s assistance strategy and corporate goals (the relevance of objectives); and the adequacy and coherence of the project’s components to achieve those objectives (the relevance of design). The extent to which the project achieved, or is expected to achieve, its stated objectives, taking into account their relative importance. The extent to which the project converted its resources economically into results. Sustainability The likelihood of continued long-term benefits, and the resilience to risk of net benefit flows over time. IFI Performance The quality of services provided by the IFI during all project phases, including the IFI’s performance in ensuring project quality at entry, satisfactory implementation, and future operation. Borrower Performance The adequacy of the Borrower’s assumption of ownership and responsibility during all project phases. Assessment Methodology The assessment of the relevance of objectives also considers the extent to which the project’s objectives were clearly stated and focused on outcomes as opposed to outputs. The relevance of design also considers whether the project’s financial instrument was appropriate to achieve project objectives and country needs. Tests the validity of the anticipated links between the project’s activities, outputs, and intended outcomes (i.e., the results chain). Outcomes are evaluated against a counterfactual. Uses cost-benefit analysis (NPV or ERR) to the extent that data is available and it is reasonable to place a monetary value on benefits. In addition, in all cases, cost effectiveness analysis is conducted. The assessment of Sustainability considers various aspects such as technical, financial, economic, social, political, and environmental sustainability. Summary Rating: Aggregate Project Performance Indicator -4- The assessment of quality at entry covers the IFI’s role in ensuring project quality and in ensuring that effective arrangements were made for satisfactory implementation and future operation of the project. The assessment of the quality of supervision is based on the extent to which the IFI proactively identified and resolved problems during project implementation. IFI Performance includes the application of relevant environmental and social safeguards. The assessment of Borrower performance considers the effectiveness of both the government and the implementing agency in ensuring quality project preparation and implementation, compliance with covenants and agreements, establishing the basis for sustainability, and fostering participation by the project’s stakeholders. Table 2: Evaluation Criteria in the Private GPS Criteria and Sub-Criteria Financial Performance and Fulfillment of Project Business Objectives Definition Financial performance: the project’s contribution to the company’s financial results, or the company’s financial results where the project is indistinguishable from the company. Fulfillment of business objectives: the extent to which the project delivered on its process and business objectives. Economic Sustainability The project and/or company’s contribution to growth in the economy. Contribution to IFI Mandate Objectives Depending on the IFI, mandate objectives may include private sector development, the development of efficient financial and capital markets, or transition to a market economy. Environmental and Social Performance The project company’s overall environmental and social performance in the area of influence of the project. An optional supplementary sub-criterion of Environmental and Social Performance is called “Environmental and Social Impact”: the extent of improvement (or deterioration) in the client’s environmental and social performance since project approval. Assessment Methodology Based on the FRR or ROIC, the achievement of returns projected at appraisal, the achievement of process and business goals articulated at approval, performance indicators in financial statements compared to appraisal projections, and the company’s business prospects. For financial intermediary operations, if possible, the assessment also considers the performance of the sub-portfolio or fund. Financial performance is assessed on a with-vs.-without project basis, or a before-vs.- after project basis. Considers the impact of the project on all key economic stakeholders – including and beyond the company’s owners and financiers (e.g., employees, suppliers, competitors, neighbors). Economic Sustainability is determined by the project’s ERR or EROIC, along with qualitative stakeholder analysis. For financial intermediary operations, Economic Sustainability includes the economic activities of sub-borrowers or the economic viability of fund investees. Economic Sustainability is assessed on a with-vs.-without or before-vs.-after basis. Considers positive and negative contributions to, e.g., competition; market expansion; private ownership and entrepreneurship; frameworks for markets; transfer and dispersion of skills; demonstration effects; standards for corporate governance and business conduct; development of financial institutions and financial/capital markets; attracting FDI flows; and development of physical infrastructure. A rating of “Neutral” is permitted in cases where a project has no observable or attributable impacts relevant to the IFI’s mandate objectives. The assessment is based on the project company’s compliance primarily with the IFI’s specified standards at approval and, secondarily, with those prevailing at the time of the evaluation. For financial intermediary operations, the assessment considers the financial intermediary’s or fund manager’s Environmental and Social Management System (ESMS) and its implementation. If required by the IFI’s specified standards, it considers the environmental and social effects of sub-projects or fund investee companies. A rating of “Not Applicable” is permitted where, by virtue of the project’s expected lack of environmental and social impacts, the IFI did not prescribe relevant safeguards. Summary Rating: Project Outcome -5- Criteria and Sub-Criteria IFI Investment Profitability Definition The profitability of each of the IFI’s investment(s) in the project company. IFI Work Quality/Bank Handling The quality of the IFI’s pre- and post-commitment work. IFI Additionality The IFI’s value proposition (value added) in providing support to the project. -6- Assessment Methodology Based on either (i) the investment’s net profit contribution measured in risk-adjusted, discounted cash flow terms, or (ii) the quality of the investment’s gross profit contribution. The quality of the IFI’s pre-commitment work considers all aspects of the IFI’s work in screening, appraising, and structuring the project and the IFI’s associated investment. The quality of post-commitment work considers all aspects of the IFI’s portfolio responsibilities in monitoring and supervising the project and the IFI’s associated investment. Considers the IFI’s financial additionality in providing funding and/or catalyzing other funding; and its non-financial additionality in improving the project’s risk profile, design, or functioning. The rating is based on a comparison with the counterfactual of how the project would have proceeded without IFI support. 2.5 Basis of the evaluation: The Public GPS says that evaluations are primarily objectives-based. This means that projects are evaluated against the statement of objectives found in the design documents, or as reconstructed by the evaluator. In contrast, the Private GPS does not require a project-specific statement of its development, policy, or transition objectives. However, consistent with international banking and investment practices, project-specific business objectives and market-based benchmarks are used to assess commercial performance. 2.6 Attribution: The Private GPS calls for with-versus-without-project comparisons, or beforeversus-after-project comparisons, for two criteria (Financial Performance and Fulfillment of Business Objectives, and Economic Sustainability). This means that, if a before-versus-after approach is taken, outcomes would not necessarily be attributable to the project. Environmental and Social Performance is based primarily on compliance with safeguards rather than on environmental and social performance compared to a counterfactual (an optional subcriterion, Environmental and Social Impact, does not call for comparison with a counterfactual). IFI Additionality calls for comparison with a counterfactual of no IFI support. The Public GPS calls for comparison with a counterfactual in the Effectiveness criterion, and is more specific on how causality is to be established: either through an impact evaluation or through a theorybased approach. It does not say anything about using a with-versus-without project analysis under the Efficiency criterion, although this is implied because the benefits used in the analysis are those evidenced under Effectiveness. 2.7 Safeguards: Environmental and social safeguards receive extensive treatment in the Private GPS, and the rating on this criterion is included in the overall project performance rating. In the Public GPS, safeguards are given much less emphasis. They enter only as part of the IFI Performance and Borrower Performance assessments, which are not included in the overall project performance rating. 2.8 Definitions of performance indicators, rating scales, and threshold values: The Public GPS does not define the rating scales (e.g., what “Moderately Satisfactory” means). With the exception of the economic rate of return, performance indicators and targets are presumed to be projectspecific and thus there are no GPS-defined indicators or threshold values for a satisfactory rating. In contrast, the Private GPS contains detailed standards on performance indicators, rating scales, and threshold values for aspects related to business performance and the project’s ERR/EROIC. Like the Public GPS, however, it does not have performance indicators or benchmarks for development, policy, or transition results.5 5 Confusion sometimes arises because of differences in the use of the term “indicator” in the Public GPS and Private GPS. The Public GPS refers to “performance indicators” or “outcome indicators” as variables that can be expressed in quantitative terms (e.g., the project’s ERR; the number of project beneficiaries; the public sector share of GDP). This is consistent with OECD-DAC definition of “performance indicator” (“a variable that allows the verification of changes in the development intervention or shows results relative to what was planned”). In the Private GPS, some of the “outcome indicators” are qualitative in nature, consistent with the OECD-DAC definition of “indicator” (“a quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor”). These qualitative factors would be called “sub-criteria” or “performance attributes” or “performance dimensions” in the Public GPS. In this paper, “performance indicator” means a measurable variable. The qualitative factors to be considered are contained in the definitions of sub-criteria and the evidentiary requirements for rating criteria and sub-criteria. -7- 2.9 Evaluation methods for projects of different types: The Private GPS defines four main types of projects, and tailors the evaluation methodology to the particular type of project. The project categories are A – direct investments; B - corporate investment programs; C – financial diversification or short-term funding; and D – investments in multiple sub-projects, either through a financial institution (D1) or a fund (D2). The Public GPS distinguishes only two types of projects – investment operations (including technical assistance loans) and policy-based operations – and has few differences in evaluation methodology for the two types. 2.10 CED involvement in the self evaluation system: The Public GPS establishes an institutional model in which the CED has little involvement with the self evaluation system. The CED supports project evaluability and the quality of self evaluation reports (e.g., through IFI staff training) but does not write guidelines for self evaluation. Instead, the CED and IFI Management share responsibility for harmonizing self- and independent evaluation guidelines. In the Private GPS, the CED is responsible for developing guidance for both self- and independent evaluation, in consultation with IFI Management. As a result, the self- and independent evaluation methodologies are automatically harmonized. 2.11 Selection of projects for evaluation: In the Private GPS, the CED chooses the set of projects for which self evaluations will be prepared – either the entire population of projects ready for evaluation, or a representative sample. All of the self evaluations are then subjected to independent validations or more in-depth independent evaluations. In the Public GPS, self evaluations are prepared for the entire population of completed projects, and the CED may choose to independently evaluate either 100 percent or a representative sample. Since the sampling methods are similar in the two GPSs, the end result would be similar selection and coverage for independent evaluations. 2.12 Timing of evaluations: Both GPSs state the principle that evaluations should be timed so that outcomes are observable. The Private GPS defines a specific condition for determining when self evaluations should be conducted: when the project reaches “early operating maturity”. Thus, there is specific guidance on when business outcomes are observable, but no equivalent guidance on when development, policy, or transition outcomes are observable. The Public GPS calls for self evaluations to be prepared within a fixed period of time determined by the IFI, typically six to twelve months after project completion. This timeframe may or may not correspond to the length of time needed to observe desired outcomes. However, independent evaluations have no such established timeframe. 2.13 Both GPSs have their strengths and weaknesses. The Private GPS is stronger in terms of explicit definitions of performance indicators, benchmarks, and ratings guidelines, at least for the project’s commercial objectives; on the attention given to environmental and social safeguards; on the criteria for deciding when to conduct and evaluation (although this defined on the basis of commercial indicators rather than development, policy, or transition indicators); and on the application of evaluation principles to different types of projects. The Public GPS is stronger in its coverage of project evaluability; its attention to the clarity of project objectives and results frameworks; and in its delineation of methodologies to establish the attribution of observed outcomes to the project. -8- 2.14 The sections below propose ways to achieve greater consistency in how public sector and private sector operations are evaluated, and in how evaluation results are reported. The next section focuses on a common reporting framework (called Option 1), and the subsequent section on a common evaluation methodology (Option 2). A comprehensive, harmonized GPS would also include sections on evaluability (evaluation-related guidelines for project design at entry, including the definition of what the project aims to achieve, the market failures it intends to address, beneficiaries and target groups, the logic connecting project activities with outputs and outcomes, and performance indicators and targets); evaluation processes (the coverage and selection of projects for evaluation, evaluation products, and the timing of evaluations); and reporting, disclosure, and learning. These topics are not covered in this paper because there is less divergence between the Public GPS and the Private GPS in these areas. 3. Option 1: A Common Reporting Framework 3.1 This section proposes a common reporting framework by mapping the evaluation criteria in the existing Public GPS and Private GPS into the standard OECD-DAC evaluation criteria: Relevance, Effectiveness, Efficiency, Sustainability, and Impact. The aim is to be able to compare ratings for public sector and private sector operations, both for the individual criteria and for aggregations of criteria. The section concludes with an application of the Option 1 approach to two projects, one private sector and one public sector. 3.2 The mapping approach under Option 1 has several advantages: Since the mapping framework sits on top of the existing criteria, it would minimize the need to revise the existing GPSs and could be implemented relatively easily by ECG members. Nevertheless, there would be a need for some changes in both GPSs, since some now-optional criteria would become mandatory. Future GPSs in other areas (e.g., the evaluation of technical assistance) could similarly be mapped to the OECD-DAC criteria. The scheme could be adapted to hybrid projects with both public and private sector characteristics by applying either the Public GPS or the Private GPS as appropriate for each criterion under the common OECD-DAC framework. The approach also has some shortcomings which are described in the introduction to Option 2. Mapping the Existing GPS Criteria to OECD-DAC Criteria 3.3 Table 3 and Figure 1 show that most of the Public GPS criteria map relatively easily to OECD-DAC criteria, which stands to reason because OECD-DAC criteria were the starting point for the Public GPS. The Public GPS criteria of Relevance, Effectiveness, Efficiency, and Sustainability map to the same OECD-DAC criteria.6 The optional criteria of Achievement of Corporate Goals and 6 The Public GPS does not include an “Impact” criterion. OECD-DAC’s definition of Impact is “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”. Most of the elements of this definition (effects that are positive and -9- Unintended Outcomes could be mapped to the OECD-DAC criterion of Impact. An overall project performance rating could be calculated from the ratings on Relevance, Effectiveness, Efficiency, Sustainability, and Impact. IFI Performance and Borrower Performance would not figure into the overall rating. 3.4 The mapping of Private GPS criteria into OECD-DAC criteria is less straightforward. Some of the Private GPS criteria can be mapped directly, but for others, only a component of the criterion maps to OECD-DAC. The mapping of the Private GPS criteria shown in Table 3 and Figure 1 reflects the following reasoning: Relevance is captured by the Contribution to IFI Mandate Objectives. Effectiveness considers the extent to which the client achieved the project’s business and process objectives (which are measured mainly in terms of outputs) and forecasted financial performance targets (which can be considered as outcomes). The assessment considers both internal (to the client) and external factors influencing the achievement of business objectives, and the extent to which the project/company’s operational performance met expectations. This is captured by one of the component parts of the Private GPS’s indicator, Financial Performance and Fulfillment of Project Business Objectives – namely, the Fulfillment of Project Business Objectives. Efficiency considers the efficiency of the project from the perspective of financial stakeholders in the company. The efficiency with which the client company deploys project funds to meet objectives is captured in the profitability of the operation, since this is a measure of the extent to which the client utilizes project assets to maximize revenues and minimize costs. Thus, the most appropriate Private GPS criterion to measure the efficiency of the company/project is one component of Financial Performance and Fulfillment of Project Business Objectives, namely Financial Performance, as measured by the FRR or ROIC or other profitability indicators as appropriate to the project type. Impact considers the observed and expected effects of the project on a wider group of stakeholders (beyond its financial stakeholders), including customers, suppliers, employees, neighbors, competitors, and taxpayers. The applicable Private GPS criteria are Economic Sustainability and the optional supplementary sub-criterion of Environmental and Social Performance called Environmental and Social Impact. The latter assesses the extent of improvement (or deterioration) in the client’s environmental and social performance since project approval. negative, primary and secondary, direct and indirect, and intended) are picked up in the Public GPS’s Effectiveness criterion. The realization of long-term effects depends on the timing of the evaluation, so these are not separated into a different criterion. Unintended outcomes are assessed under an optional criterion but are not included in the overall project performance rating, due to the Public GPS’s reliance on an objectives-based approach. Similarly, the Public GPS allows for the use of additional criteria to assess the achievement of corporate goals (to the extent that they are not expressed in the project’s statement of objectives), but their ratings are not included in the overall project performance rating. - 10 - Sustainability takes into account both the project/company’s continued commercial viability and prospects for growth (since other project benefits rely on the continued existence of the company in a competitive market), and the company’s compliance with environmental and social performance standards. Thus, the Private GPS criteria that are mapped into Sustainability are part of the Fulfillment of Business Objectives and Financial Performance (namely the part that relates to the client company’s prospects for sustainability and growth), and Environmental and Social Performance. 3.5 A summary rating for overall project performance for private sector operations could be calculated from the ratings on Contribution to IFI Mandate Objectives (which maps to Relevance); Fulfillment of Project Business Objectives (Effectiveness); Financial Performance (Efficiency); Economic Sustainability and Environmental and Social Performance (Impact); part of Fulfillment of Business Objectives and Financial Performance and Environmental and Social Performance (Sustainability). IFI Additionality, IFI Investment Performance, and IFI Work Quality/Bank Handling would not figure into the overall rating. 3.6 To illustrate how Option 1 would work in practice, below are two applications of the approach: one to a private sector operation and one to a public sector operation, both in the telecommunications sector. The projects are: 3.7 an IFC telecommunications project to finance the expansion and upgrade of a company’s GSM cellular telephone network. a World Bank telecommunications project to improve delivery of communications services to the Government and people of Afghanistan. For the private sector project, information is taken from the IFC’s XPSR (equivalent to the XASR) and IEG’s Evaluation Note (XASR-A). For the public sector project, the sources are the WB’s Implementation Completion and Results Report (ICRR, equivalent to the CR) and IEG’s ICRR Review (CR validation). The projects are rated using the ratings systems in the Private GPS and Public GPS respectively. - 11 - Table 3: Mapping the Existing GPS Criteria to OECD-DAC Criteria (existing criteria shown in bold) OECD-DAC Criteria Relevance OECD-DAC Definition1 The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities, and partners’ and donors’ policies. Note: Retrospectively, the question of relevance often becomes a question as to whether the objectives of an intervention or its design are still appropriate given changed circumstances. Effectiveness The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results. Public GPS Criteria Relevance considers both the relevance of objectives and the relevance of design. Relevance of objectives: assessed against beneficiary needs, the country’s development or policy priorities and strategy, and the IFI’s assistance strategy and corporate goals. Projects dealing with global public goods also assess relevance against global priorities. Relevance of design: assesses the extent to which project design adopted the appropriate solutions to the identified problems. It is an assessment of the internal logic of the operation (the results chain) and the validity of underlying assumptions. Effectiveness: the extent to which the project achieved (or is expected to achieve) its stated objectives, taking into account their relative importance. The assessment uses appropriate methods to determine the contribution of the project to intended outcomes in a causal manner. Efficiency: The extent to which the project has converted its resources economically into results. The assessment uses both cost-benefit analysis (if feasible and practical) and costeffectiveness analysis (in all cases). - 12 - Private GPS Criteria Contribution to IFI Mandate Objectives: the project’s contribution to the IFI’s mandate objectives, be they to stimulate development of the private sector, develop efficient financial/capital markets, or transition to a market economy. Includes both positive and negative contributions. Fulfillment of Project Business Objectives (part of the Financial Performance and Fulfillment of Project Business Objectives criterion): the extent to which the project delivered on the process and business objectives stated at approval and achieved operational performance targets. Financial Performance (part of the Financial Performance and Fulfillment of Project Business Objectives criterion): the incremental effect of the project on the company, including all financial stakeholders in the project and/or company. The incremental effect of the project is assessed on a with-vs.without project basis, or a before-vs.-after OECD-DAC Criteria Impact OECD-DAC Definition1 Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Public GPS Criteria Cost-benefit: the extent to which the benefits of the project (achieved or expected to be achieved) exceed project costs. A form of a cost-benefit calculation is an ERR. Cost-effectiveness: the extent to which the project achieved its benefits at least cost, compared to alternative ways of achieving the same results. Achievement of Corporate Goals: currently an optional criterion in the Public GPS to assess the project’s contributions to broad corporate goals such as poverty reduction, social cohesion, rural development, institutional development, etc. that are not included in the project’s statement of objectives. Unintended Outcomes: currently an optional criterion in the Public GPS to cover positive or negative impacts of the project that were not included in the project’s statement of objectives. Sustainability The continuation of benefits from a Sustainability: the risk that changes may occur - 13 - Private GPS Criteria project basis. Impact considers both the project’s Economic Sustainability (effects on wider stakeholders) and its Environmental and Social Impact. Economic Sustainability: The project and/or company’s contribution to growth in the economy, assessed on a with-vs.-without or before-vs.-after basis. The effect of the project on all key economic stakeholders – including and beyond the company’s owners and financiers. Quantitative methods include ERR and EROIC. Environmental and Social Impact: The extent of environmental and social change, considering both the ex ante and ex post conditions of the project compared with the IFI’s requirements at approval and, therefore, the extent of progress or regress in the project’s environmental and social performance. This is an optional supplementary criterion under Environmental and Social Performance, which is otherwise concerned with compliance with safeguards. Sustainability considers both the OECD-DAC Criteria OECD-DAC Definition1 development intervention after major development assistance has been completed. The probability of longterm benefits. The resilience to risk of the net benefit flows over time. Public GPS Criteria that are detrimental to the continued benefits associated with the achievement or expected achievement of the project’s objectives, and the impact on that stream of benefits if some or all of these changes were to materialize. Both the probability and the likely impact of various threats to outcomes are assessed. Overall Project Performance Not prescribed in OECD-DAC. Currently calculated from the ratings on Relevance, Effectiveness, Efficiency, and Sustainability. Could be calculated from the ratings on Relevance; Effectiveness; Efficiency; Sustainability; and Achievement of Corporate Goals and Unintended Outcomes (Impact). Private GPS Criteria project/company’s continued viability and its compliance with environmental and social safeguards. The former is reflected in a component of the Fulfillment of Business Objectives and Financial Performance criterion, namely the client company’s prospects for sustainability and growth. Thus, the relevant indicators are part of Fulfillment of Business Objectives and Financial Performance and Environmental and Social Performance. Currently calculated from the ratings on Financial Performance and the Fulfillment of Business Objectives, Economic Sustainability, Environmental and Social Performance, and Contribution to IFI Mandate Objectives. Could be calculated from the ratings on Contribution to IFI Mandate Objectives (Relevance); Fulfillment of Project Business Objectives (Effectiveness); Financial Performance (Efficiency); the forward-looking part of Fulfillment of Project Business Objectives and Financial Performance, and Environmental and Social Performance (Sustainability); and Economic Sustainability and Environmental and Social Impact (Impact). 1 Organization for Economic Cooperation and Development (2002), Glossary of Key Terms in Evaluation and Results Based Management. Paris. The definitions of the five criteria are slightly different from those shown in another source on the DAC website, DAC Criteria for Evaluating Development Assistance, which is based on OECD (1991), The DAC Principles for the Evaluation of Development Assistance; OECD (1986), Glossary of Terms Used in Evaluation, in ‘Methods and Procedures in Aid Evaluation’; and OECD (2000), Glossary of Evaluation and Results Based Management (RBM) Terms. The definitions in the 2002 Glossary of Key Terms in Evaluation and Results Based Management are used in this paper because the source publication is more recent and because it is also used for definitions of other evaluation terms. - 14 - Figure 1: Option 1 Criteria and Sub-Criteria Private GPS Criteria Public GPS Criteria OECD-DAC Criteria Relevance Relevance Contribution to IFI Mandate Objectives Effectiveness Effectiveness Fulfillment of Project Business Objectives Efficiency Efficiency Financial Performance Economic Sustainability Achievement of Corporate Goals (opt.) Impact Environmental and Social Impact (opt. suppl. criterion) Unintended Outcomes (opt.) Sustainability (the forward-looking part of) Fulfillment of Project Business Objectives and Financial Performance Sustainability Environmental and Social Performance Overall Project Performance Rating IFI Additionality IFI Performance IFI Investment Performance Borrower Performance IFI Work Quality/Bank Handling Private Sector Application: IFC Telecommunications Project 1. PROJECT INFORMATION The Project consisted of financing the expansion and upgrade of a Company’s GSM cellular telephone network, at an estimated cost of USD 1.95 billion. It included: (i) the funding of the Company's network expansion with associated working capital requirement; and (ii) the refinancing of USD 1.1 billion equivalent of short-term debt. The Company had previously been awarded one of two cellular licenses in the country, aimed at increasing competition and private sector participation in mobile telephony. 2. RELEVANCE Rating under Private GPS Contribution to IFI Mandate Objectives: Satisfactory Rating under OECD-DAC Framework Relevance: Satisfactory The IFI’s global telecommunications sector strategy was to: (i) foster new private investment and competition in order to improve access to basic communication services; (ii) increase the availability of advanced technologies; and (iii) support the development of downstream industries. The Project was expected to fit well with these objectives. Moreover, the results reinforce this alignment and justify the Satisfactory rating as follows: i) Objective: To increase the public's access to affordable mobile services and quality of service through increased private investment and competition in the country. Result: The Project enabled the Company to invest in its 3G network in turn contributing to the overall enhancement of technology used by mobile - 15 - subscribers in the country. The percentage of 3G network users reached 48% of total mobile subscribers by 2011, which is significantly higher than the EU average of 30%. ii) Objective: To provide the recently established Company much needed long term capital, to complete its investment program and allow it to emerge as a legitimate competitor against the two large incumbent mobile operators. Result: It enabled the Company to resist the strong competition from the dominant operators and increase its market share, albeit by less than hoped for, from 15% to 20%. iii) Objective: To support broader economic growth through increased availability of advanced technologies and introduction of new value added services such as mobile banking. Result: The Company established partnerships with more than 75 major retailers and banks, it developed new service applications, and won the ‘Best Product Award’ for its Credit-Card-In-SIM application. 3. EFFECTIVENESS Rating under Private GPS Fulfillment of Business Objectives: Partly Unsatisfactory Rating under OECD-DAC Framework Effectiveness: Partly Unsatisfactory In respect of the Project’s business objectives: (i) The Company invested heavily in its 3G network. It exceeded forecast annual capital expenditure in order to accelerate its roll-out of services, investing on average USD 310 million per annum since 2009. The funding also helped the Company to establish a research & development department, which invested more than USD 65 million over five years and has provided innovative solutions to its business partners. (ii) The Company successfully refinanced USD 1.1 billion of short-term debt, replacing it with long-term finance more suited to the tenor of its network assets. At the time of board approval, it was estimated that the Company would achieve a market share of 23% by 2010. Despite the higher than forecasted increase in mobile penetration and subscribers in the country, its market share had only reached 20% by the end of 2011, due mainly to intense competition from the incumbent operators. Although outputs of the project were achieved, one of the principal outcome objectives (market share) was not. Therefore, effectiveness is rated Partly Unsatisfactory. 4. EFFICIENCY Rating under Private GPS Financial Performance: Unsatisfactory Rating under OECD-DAC Framework Efficiency: Unsatisfactory The financial performance of the Company, measured in FRR terms, was Unsatisfactory. Lower than expected market share resulted in lower revenue growth, impeding the financial returns for the Company. Recorded revenues of USD 1,251 million in 2011, in real terms, were 16% lower than forecasts. The Company’s EBITDA margin was expected to increase gradually from 17% in 2007 to 34% in 2011, getting closer to the margin level of the dominant operator. However, in order to increase its market share, marketing and promotional expenses were much higher than planned on a per-subscriber basis. This had an adverse impact on the EBITDA margin, which averaged only 13% from 2007 to 2011. A combination of lower revenues, tighter margins, and the accelerated capital program, resulted in negative free cash flow for the Company. To reduce interest costs, USD 2.3 billion of shareholder loans were converted into equity. From the perspective of financial stakeholders in the Company, returns to shareholders have therefore been negative, whilst the financial performance has increased the risk for senior lenders. These results are consistent with an Unsatisfactory rating. - 16 - 5. SUSTAINABILITY Rating under Private GPS Future Business Prospects: Satisfactory Env. and Social Performance: Partly Unsatisfactory Rating under OECD-DAC Framework Sustainability: Partly Unsatisfactory The Company’s management is now focusing more on profitable and sustainable growth. It is expanding data services and aiming to increase its post-paid subscriber base (higher Average Revenue Per User ARPU). The percentage of post-paid customers increased from 37% in Q1 2010 to 44% in Q1 2012, and the Company has now achieved the highest post-paid customer base among mobile operators in the country. Furthermore, the share of data revenues in total revenues increased from 4.9% in 2009, to 13.4% in 2011. The Company continues to receive strong support from its shareholders, and with the balance sheet now restructured and de-leveraged, has a much more robust basis for future operations. Finally, the regulatory environment has also improved, with implementation of pro-market competition rules such as network sharing, number portability, asymmetric termination tariffs, and unsubsidized onnet tariffs. Business prospects going forward are therefore rated Satisfactory. The Project, originally appraised in 2004, was classified as Category B for its E&S risk. The Company started implementation of a Safety Health and Environment (SHE) Management System. Although, it committed to obtain ISO 9001 and ISO 14001 certificates by the end of 2007 and OHSAS 18001 certification by the end of 2008, these are still outstanding. During a 2010 site visit, the Company reported that there was an ongoing updating process for adaptation of SHE, which will be applicable to the mobile sector. However, by 2012, this adaptation process had not been finalized. The lack of a fully developed and formalized SHE management system and centralized EHS monitoring mechanism and organization results in a Partly Unsatisfactory rating for Performance Standard (PS) 1. The project performance in regard to PS2, PS3, and PS4 has been Satisfactory at appraisal and evaluation, but most of the Company’s mobile base stations have been installed without the required permissions and are not in compliance with PS5 and regulatory requirements. Overall, Environmental and Social Performance is rated Partly Unsatisfactory. 6. IMPACT Rating under Private GPS Environmental and Social Impact: No Change Economic Sustainability: Satisfactory Rating under OECD-DAC Framework Impact: Satisfactory Deficiencies in the Company’s SHE Management System were identified at approval and requirements for these to be addressed were incorporated into the Project agreements. Since the Company has yet to meet these requirements in full and deficiencies still exist, the rating of environmental and social impact is No Change. The Project's ERR is estimated at 17%. Quantified net benefits to society in NPV terms are approximately USD 721 million, comprising largely of tax and fee payments to the Government (roughly equivalent to 46% of the Company’s total revenues). Although the Project has not performed well for the Company’s financial stakeholders, it has had a number of Satisfactory impacts on the market, on related businesses, and for customers, as follows: Consumers: The Company increased its subscribers from 7.5 million to 12.8 million, and has offered more affordable services than its competitors. Consumers also benefited from the high quality of service offered by the Company as it increased its geographical/population coverage. - 17 - Market: As the third-biggest player, the Company has had some positive impact on the competitiveness of the market. Prices have decreased dramatically, with ARPU across the country falling from approximately USD$ 19 in 2004 to USD$ 11 in 2011. At the same time, the quality of mobile services increased as evidenced by lower unsuccessful call and blocked call rates, and increased customer relations activities. Employment: By 2012, the Company had 2,790 employees compared to 1,833 employees at approval. It established eight outsourced call centers in frontier regions of the country, boosting local employment in these areas. Direct and outsourced call center employees are provided with special training. Through its franchised dealers, the Company has increased indirect employment from 514 to 3,656. 7. OVERALL PROJECT PERFORMANCE Project Development Outcome, based on Private GPS: Mostly Unsuccessful Overall Project Performance Rating, based on OECD-DAC Framework: Mostly Unsuccessful Public Sector Application: World Bank Telecommunications Project 1. PROJECT INFORMATION The project aimed to improve delivery of communications services to the Government and to the people of Afghanistan, via three components: (i) A USD 14 million component to support the supply, installation, and commissioning of a communications network as well as improved and extended coverage to Ministries and other key Government organizations in Kabul as well as Provincial Capitals and selected other sites, and operations and maintenance and training of Afghan Tel staff. (ii) A USD 7 million component to provide consulting services, training, technical advisory services and equipment to (a) separate MOC’s policy, regulation, operational and ownership functions, (b) strengthen the MOC’s policy and project management capacity, (c) further establish and “bed down” independent regulatory capacity, (d) establish the monitoring equipment required to properly administer the MOC’s spectrum plan, and (e) establish Afghan Tel as an efficient State owned corporation. (iii) A USD 0.5 million component to support consulting services and equipment for the separation of the MOC’s policy and operational functions in the postal sector, and to help develop the sector. - 18 - 2. RELEVANCE Rating under Public GPS Relevance: Substantial Rating under OECD-DAC Framework Relevance: Substantial Relevance of objectives: ICT remains a priority in the Afghanistan National Development Strategy (ANDS) and an important priority according to the most recent Bank Interim Strategy Note. This includes broad based work to improve infrastructure services in most areas, including telecommunications, to address overall business constraints. The objectives related to postal services are not mentioned as priorities of the Strategy Note, but they clearly require improvement in Afghanistan. For example, the average number of letter-post items posted per inhabitant in 2007 was only 0.24, significantly below the average for other neighboring countries with similar income per capita level such as Bangladesh (0.79 in 2007), Nepal (2.56 in 2008), and Pakistan (2.03 in 2008). Relevance of design: Project design was based on solid background analysis during a one-year engagement with the Ministry and other donors in the telecommunications sector. However, there were weaknesses in the results framework, which did not allow proper monitoring of outcomes. Although design was kept relatively simple, in line with the conflict-affected situation in Afghanistan, the addition of the postal sector was inappropriate. It added to complexity while at the same time investing inadequate resources to achieve the goal of improving services. 3. EFFECTIVENESS Rating under Public GPS Effectiveness: Substantial Rating under OECD-DAC Framework Effectiveness: Substantial The delivery of communications services to the government has been significantly improved and all targets regarding the Government communication network (GCN) were met. The GCN is fully functional (voice, data, and video) and connects all 34 provincial capitals and 40 line ministries/agencies in Kabul. Telephone connectivity of Government has reached over 70 thousand lines — ten times over the original target of 6,550 lines. Regarding the delivery of communications services to the people of Afghanistan: Teledensity has grown from 0.2% in 2002 to about 35% by 2009 and total number of subscriptions is about 11 million. Service providers increased from two in 2002 to seven plus seven operational Internet service providers (ISPs). Mobile signal coverage has reached over 70 percent of the population. Tariffs have fallen to about 10 US cents a minute from US$2 per minute in 2002. Investments in the sector are above US$1 billion and surpassed the target of US$950 million. These achievements cannot be attributed solely to the project, and it is hard to determine the relative contribution of the project in the context of rapid ICT sector growth worldwide, along with complementary support from other Bank projects, other donors and technical assistance providers. More modest results were achieved regarding postal services. Although new services such as express mail have been introduced, the Postal Commission has been instituted and the Postal Law has been drafted, indicators of postal performance remain significantly low. - 19 - 4. EFFICIENCY Rating under Public GPS Efficiency: Modest Rating under OECD-DAC Framework Efficiency: Modest As the project was prepared as an emergency credit, no economic and financial analysis was conducted during preparation. Nor were economic or financial calculations made at evaluation, nor was costeffectiveness analysis carried out. At least an approximate estimate of the economic benefits could have been made based on the opportunity revenues generated from the government network if the lines/or the services were provided to private customers rather than to the government. Furthermore, financial calculation should have been made based on the revenues from the government network paid to Afghan Tel by the government, especially given the significant increase in cost of the GCN component (US$17 million compared with US$ 14 million at appraisal). Research suggests that every 10 percentage-point increase in mobile telephony penetration in developing countries leads to an acceleration of GDP growth by 0.75 percentage point. These calculations, however, should be considered with caution as such extrapolations and estimates may well suffer from econometric estimation problems. Some indirect benefits to which the project made a partial contribution are: the increase in affordability of telephone services (tariffs dropped from US$2 per minute of talk-time to about 10 US cents per minute over the period of the project), the social impact due to the growth of the telecommunications sector (creation of 60,000 jobs directly or indirectly) and the favorable fiscal impact. However, these achievements cannot be attributed solely to the project. 5. IMPACT Rating under Public GPS Achievement of Corporate Goals: Not Rated Unintended Outcomes: Not Rated Rating under OECD-DAC Framework Impact: Not Rated The project’s broader impacts on corporate goals such as poverty reduction, social cohesion, rural development and institutional development were not assessed. There were no unintended outcomes. 6. SUSTAINABILITY Rating under Public GPS Sustainability: Moderately Likely Rating under OECD-DAC Framework Sustainability: Moderately Likely The government infrastructure will be maintained, and the reforms undertaken and the degree of competition achieved in the sector are unlikely to be reversed. However, the volatile and deteriorating security situation in Afghanistan poses moderate risks to development outcome. - 20 - 7. OVERALL PROJECT PERFORMANCE Overall Project Performance Rating, based on Public GPS: Moderately Satisfactory Overall Project Performance Rating, based on OECD-DAC Framework: Moderately Satisfactory The operation substantially achieved most of its telecommunications objectives, and relevance of objectives and design was also substantial. Achievements in the postal segment, however, were modest, although this was a small part of the project. Limited evidence was presented to demonstrate efficiency and attribution of benefits to the project. Outcome is rated as moderately satisfactory. 4. 3.8 Option 2: A Common Evaluation Methodology The Option 1 approach goes part of the way toward the harmonization objective, but there are still some differences between the public sector and private sector criteria and methodologies that create inconsistencies and make comparisons difficult. For private sector operations, the achievement of development, policy, or transition objectives that are specific to the project is not considered in the Effectiveness rating. Instead, the Effectiveness rating is based only on the achievement of the project’s business and process objectives. In contrast, for public sector operations, Effectiveness assesses the achievement of the project’s specific development, policy, or transition objectives. Attribution of outcomes to the project differs between the two GPSs. For the Effectiveness criterion, the Public GPS requires the comparison of observed outcomes to a counterfactual, either using an impact evaluation or a theory-based method. In contrast, the Private GPS generally uses before-and-after comparisons to describe the results of the project, and does not have an explicit methodology for establishing a counterfactual. The project’s Economic Rate of Return (ERR) is an indicator of Impact for private sector operations but an indicator of Efficiency for public sector operations. For private sector operations, the Efficiency criterion reflects only the financial rate of return to the client, rather than the broader economic benefits and costs of the project that are measured for public sector operations. Compliance with environmental and social safeguards is the main factor in Sustainability for private sector operations, but for public sector operations it enters into IFI and Borrower Performance. The mapping of existing criteria to the OECD-DAC criterion of Impact is straightforward for the Private GPS criteria but not for the Public GPS. The latter incorporates most of the elements of the OECD-DAC Impact criterion into the Effectiveness criterion. The other elements of the OECD-DAC Impact criterion, reflected in Achievement of Corporate Goals and Unintended Outcomes, are optional. - 21 - Contribution to IFI Mandate Objectives is mapped to Relevance for private sector operations. Since this criterion includes both planned and actual contributions, it is not equivalent to the Relevance criterion for public sector operations, which only assesses the ex-ante relevance of project objectives to IFI mandate objectives. IFI Additionality is assessed below the line for private sector operations, but would be assessed above the line (under Relevance) for public sector operations. 4.1 Option 2 aims to go further than the mapping of existing GPS criteria into a common reporting framework, to the definition of an evaluation methodology that could be applied to both public sector and private sector operations. Option 2 attempts to solve the problems described above, as well as to draw on the strengths and address the weaknesses of both GPSs. 4.2 The proposed methodology attempts to: 4.3 be relevant to the institutional structures and mandates of the wide variety of IFIs in ECG. be applicable to both public sector and private sector operations, allowing but minimizing the use of modules specific to private sector or public sector operations. allow comparisons of ratings of individual criteria across public sector and private sector operations (e.g., Relevance to Relevance, Sustainability to Sustainability) as well as comparisons of the overall project performance rating. minimize overlaps among evaluation criteria. sharpen the focus of evaluations on the project’s specific intended results, rather than requiring projects to be evaluated against a common set of outcomes that may not all be relevant to the project. separate the results that are easily attributable to the project without comparison with a counterfactual, from those that may be influenced by external factors and thus require comparison with a counterfactual. clarify how attribution should be taken into account in the assignment of ratings. Each section below begins with the highlights of the proposed standards and how they differ from the current GPSs, and suggests other options that might be considered. Both the text and tables are formatted in landscape to allow for easier reading. - 22 - The Basis of the Evaluation 4.4 Highlights: The evaluation approach begins with an understanding of the basis of the evaluation (Table 4). Both public and private sector evaluations are based on the specific, intended results of the project, either as drawn from project design and/or approval documents, or as reconstructed by the evaluator. This is the current standard in the Public GPS, but for private sector operations it internalizes the project’s private sector development, financial sector development, transition, or regional economic impact objectives into a set of intended results that is specific to the project. Similarly, intentions to reach particular populations, or to reduce or mitigate market failures and externalities, would be brought into the project’s intended outcomes. 4.5 Other options: Maintain the approach of the Private GPS, which evaluates all projects against common objectives under the Contribution to IFI Mandate Objectives criterion (e.g., Private Sector Development in the case of IFC and Transition Impact in the case of EBRD). However, this would be harder to do for public sector operations since the IFI mandate objectives for public sector operations are often quite broad (e.g., poverty reduction in the case of WB). Table 4: The Basis of the Evaluation Evaluation Principles and Practices Projects are evaluated primarily against the outcomes that the project intended to achieve, as contained in the project’s design and/or approval documents. Projects that are intended to reach specific populations, or to address market failures or externalities, also are evaluated against these intended outcomes. If intended project results are unclear, or if results are expressed solely in terms of outputs, the evaluator retrospectively constructs an outcome-oriented set of intended results based on the expected benefits and beneficiaries of the project, project components, performance indicators, and/or other elements of project design. If intended results were revised during implementation, the project is assessed against both the original and the revised intended results. Application to: Public Private Sector Sector X X X X X X X X Comments Same as Public GPS OP 12.1. Private sector operations are evaluated against both their intended development, policy, or transition outcomes, and their intended business outcomes. A clarifying addition to both the Public and Private GPS. Similar to Public GPS 12.3 – 12.5. In some IFIs, private sector operations are less likely than public sector operations to have a formal statement of objectives. It will therefore be necessary for the evaluator to construct set of intended results as well as the results chain linking them with project activities and outputs. Same as Public GPS EP 13. - 23 - Evaluation Principles and Practices Evaluations also may assess the contribution of the project to broad corporate goals to the extent they are not expressed in the project’s intended results. Evaluations also may assess the project’s contribution to unintended outcomes, both positive and negative. Application to: Public Private Sector Sector X X X X Comments This goes beyond the objectives-based methodology of the Public GPS to include the project’s contribution to non-project-specific corporate goals. Depending on the IFI, these may include poverty reduction, rural poverty reduction, social cohesion, institutional development, EU policy, environmental sustainability, inclusive growth, etc. Goes beyond the Public GPS, which calls for unintended outcomes to be discussed and documented but not rated. Thus, an element of OECD-DAC “Impact” is added to the evaluation of public sector operations. Evaluation Criteria 4.6 Highlights: Harmonized evaluation criteria apply to both public sector and private sector operations, but some of the criteria have sub-criteria that are applicable only to public sector or private sector operations. The structure is shown in Figure 2 and Table 5. The Relevance criterion assesses the relevance of intended project results against country needs and priorities, IFI country strategies, and IFI corporate goals (Strategic Alignment) as well as the ex-ante justification for IFI involvement (Rationale for IFI Involvement). For operations that could potentially compete with the private sector (which may be the case for both public and private sector operations), the Relevance assessment assesses the rationale for public sector intervention. Where applicable, it assesses the evidence behind claims of market failures that justify the intervention, and the rationale for targeting specific populations. The Rationale for IFI Involvement also discusses the intended value added of the IFI vis-à-vis other donors and/or private investors or lenders. The Results criterion comprises four sub-criteria: Operational Performance, Contribution to Intended Outcomes, Contribution to Corporate Goals, and Unintended Outcomes. For the Public GPS, this adds some of the elements of OECD-DAC Impact criterion and some elements of the Private GPS’s Economic and Social Performance criterion to the existing Effectiveness criterion. The Operational Performance sub-criterion assesses results that are under the control of the project (outputs and the fulfillment of project business objectives) and thus easily attributable to it. The other Results sub-criteria (Contribution to Intended Outcomes, - 24 - Contribution to Corporate Goals, and Unintended Outcomes) are likely to be influenced by external factors and thus require comparison with a counterfactual. This is clarified by using the term “contribution” rather than “achievement”. For private sector operations, the project’s development, policy, or transition impacts, as well as its regional economic impacts, are covered under Contribution to Intended Outcomes. Contribution to Corporate Goals is a criterion that accommodates the current practice of many IFIs of assessing the project’s results in achieving broad corporate objectives – e.g., poverty reduction, social inclusion, shared prosperity, institutional development, pro-poor innovation – to the extent these are not contained in the project’s intended results. The rating for Results is based on the ratings for Operational Performance, Contribution to Intended Outcomes, Contribution to Corporate Goals, and Unintended Results. The rating on Contribution to Intended Outcomes criterion would normally be given greater weight than the ratings on the other three criteria. The Efficient Use of Resources criterion assesses the profitability of the project to all stakeholders: the project’s financial stakeholders (for private sector operations), society at large (for both public and private sector operations), and the IFI (for private sector operations). It also separates Economic Efficiency from Implementation Efficiency. This is because implementation delays may affect the timing of both benefits and costs, and may not be captured in ERRs. Sustainability covers the sustainability of outcomes achieved or expected to be achieved as well as the project’s environmental and social sustainability. The latter is assessed by the project’s compliance with environmental and social safeguards (“do no harm”). This is an addition to the Public GPS, which places less emphasis on safeguards compliance than does the Private GPS. For private sector operations, Sustainability also includes an assessment of the likelihood of continued viability of the project’s company (or, for Group D operations, the continued viability of the financial institution/fund and sub-borrowers/fund investees). For both types of operations, negative environmental and social effects of the project, or positive effects that go beyond the “do no harm” resulting from compliance with safeguards, are covered under Unintended Outcomes. For both public and private sector operations, IFI Performance covers Quality at Entry (QAE) and the Quality of Supervision. QAE includes an assessment of the logic of the operation – a divergence from the Public GPS, which covers this aspect under the Relevance of Design. This would eliminate the substantial overlap between the Relevance of Design and QAE that currently plagues the Public GPS. - 25 - 4.7 Client Performance assesses the non-financial performance of the project/company for private sector operations, and the performance of the government and implementing agency for public sector operations. Other Options: Under the Results criterion, for private sector operations, include a sub-criterion for Regional Economic Impact. This would mimic the current Private GPS criterion of Economic Sustainability, which assesses the project’s impact on stakeholders external to the company (e.g., employees, suppliers, competitors, and neighbors). The reason for not doing this is that these results should be included from the beginning in the project’s results chain, and assessed during evaluation along with other intended outcomes. Rather than putting Financial Performance of the Project/Company under Efficient Use of Resources, include it under Results. The argument is that private sector operations are aimed mainly at promoting commercially-viable enterprises, a prerequisite for achieving development, policy, and transition impacts. However, the reason for putting Financial Performance under the Efficient Use of Resources is because it is more consistent with the concept of efficiency, i.e., the profitable use of both private and public resources. It also preserves the concept of Results as being the “end” of the intervention, as opposed to the “means” of a viable commercial enterprise. Rather than including IFI Profitability under Efficient Use of Resources, put it below the line as part of IFI Performance. The argument is that the internal efficiency of the IFI has no bearing on the project's performance from a development point of view. The reason for not doing so is that the Efficient Use of Resources conceptually captures the efficiency of resource allocation by all of the project’s stakeholders, including the IFI. - 26 - Figure 2: Option 2 Criteria and Sub-Criteria Private Sector Sub-Criteria Harmonized Criteria Public Sector Sub-Criteria Strategic Alignment Rationale for IFI Involvement Relevance Strategic Alignment Rationale for IFI Involvement Results Achievement of Outputs Fulfillment of Project Business Objectives Operational Performance Achievement of Outputs Contribution to Intended Outcomes Contribution to Corporate Goals Unintended Outcomes Financial Performance of the Project/Company Economic Efficiency Implementation Efficiency IFI investment profitability Efficient Use of Resources Outcome Sustainability Commercial Sustainability Compliance with Safeguards Sustainability Economic Efficiency Implementation Efficiency Outcome Sustainability Compliance with Safeguards Overall Project Performance Rating Quality at Entry Quality of Supervision Non-Financial Performance of the Company IFI Performance Client Performance - 27 - Quality at Entry Quality of Supervision Government Performance Implementing Agency Performance Table 5: Option 2 Criteria and Sub-Criteria Criteria and Sub-Criteria Relevance Strategic Alignment Rationale for IFI Involvement Results Operational Performance Achievement of Outputs Fulfillment of Project Business Objectives Contribution to Intended Outcomes Definition Application to: Public Private Sector Sector The consistency of the project’s intended results with beneficiary needs; the country’s development, policy, or transition priorities and strategy; and the IFI’s assistance strategy and corporate goals. X X X X X X The justification for public sector involvement and the IFI’s involvement in the project. Achievement of outputs and intermediate outcomes that are under the control of the project. The extent to which the project achieved its targeted outputs. X The extent to which the project delivered on its process and business objectives. The extent to which the project contributed, or is expected to contribute, to its intended development, policy, or transition results, taking into account their relative importance. Also considers the project’s reach to intended beneficiaries and target groups, and the - 28 - X X X X Comments Similar to Public GPS OP 17.1 and the Relevance of Objectives. In contrast to the Public GPS, the Relevance criterion does not include the Relevance of Design, which is covered instead under Quality at Entry. Similar to Private GPS criterion, IFI Additionality (OP 22.1), but focuses on what was expected at project entry rather than what was achieved at the time of evaluation. A new criterion. In practice, some IFIs separate the discussion of outputs and outcomes in the Effectiveness assessment. Similar to Private GPS OP 16.1 (Financial Performance and Fulfillment of Project Business Objectives). Corresponds to Effectiveness in the Public GPS (OPs 18.1 and 19.1). For private sector operations, this subcriterion also includes regional economic impacts that are now covered under Economic sustainability (Private Criteria and Sub-Criteria Contribution to Corporate Goals Unintended Outcomes Efficient Use of Resources Financial Performance of the Project/Company Economic Efficiency Implementation Efficiency Definition Application to: Public Private Sector Sector project’s success in reducing or compensating for market failures identified at entry. The project’s contribution (or expected contribution) to broad corporate goals that are not included in the project-specific intended results. X X X X Other results caused by the project, positive or negative, that are not covered in the above sub-criteria. The profitability of the project to all financial stakeholders in the project and/or company. The profitability of the project from a social perspective. The extent to which the costs involved in achieving results were reasonable in comparison with both the project’s benefits and with recognized norms. Other aspects of project design and/or - 29 - X X X X X Comments GPS OP 17.1). Adds target groups and market failures. Includes broad corporate goals that are not covered in the Contribution to Intended Outcomes (e.g., poverty reduction, gender equality, social cohesion, inclusive growth). For private sector operations, includes IFI Mandate Objectives (Private GPS OP 18.1) and the optional sub-criterion, Environmental and Social Impact. Differs from the treatment of unintended (unanticipated) outcomes in the Public GPS (OPs 14.1 – 14.3) because in the Public GPS they do not enter into the overall project performance rating. Covers one aspect of the OECD-DAC Impact criterion. Includes positive or negative environmental impacts caused by the project. Similar to Private GPS OP 16.1. May be applicable to public sector operations that have a commercial orientation. Similar to Public GPS OP 20.1 and Private GPS OP 17.1. In the Public GPS, implementation Criteria and Sub-Criteria Definition Application to: Public Private Sector Sector implementation that either added to or reduced costs (e.g., implementation delays). IFI Investment Profitability Sustainability Outcome Sustainability The extent to which the IFI deployed its own resources efficiently to generate a net profit from its investment. X The resilience to risk of results achieved or expected to be achieved. X Commercial Sustainability Compliance with Safeguards IFI Performance Quality at Entry Prospects for the continued viability of the company, financial institution, and/or subborrowers/fund investees. X X The project’s compliance with applicable environmental and social safeguards in the area of influence of the project. The IFI’s role at project entry in ensuring quality design and implementation arrangements. - 30 - X X X X Comments delays are considered in the assessment of Efficiency (OP 20.5). Here, implementation efficiency is explicitly separated from Economic Efficiency. Same as IFI Investment Profitability in the Private GPS (OP 20.1). Draws on Public GPS OP 21.1 but extends the outcomes considered to include regional economic impact (mainly for private sector operations), Contribution to Corporate Goals, and Achievement of Unintended Outcomes. The forward-looking part of the Financial Performance and Fulfillment of Project Business Objectives criterion. See Private GPS EP 16 (B) and OP 16.2 (5). May be applicable to public sector projects with a commercial orientation. Similar to the Private GPS criterion, Environmental and Social Performance (OP 19.1). Adds an explicit sub-criterion on safeguards to Public GPS. Ensures that consideration is given to the actual implementation of safeguards, not just the adoption of a safeguards system. Similar to Public GPS OPs 22.1 – 22.7. The quality of the results framework appears here rather than under the Criteria and Sub-Criteria Quality of Supervision Client Performance Non-Financial Performance of the Company Government and Implementing Agency Performance Definition Application to: Public Private Sector Sector The extent to which the IFI proactively identified and resolved problems during implementation and adequately executed its portfolio responsibilities The company’s execution of its responsibilities for achieving process and business objectives and its compliance with relevant government regulations. The performance of the government and of the implementing agency during all project phases in meeting their responsibilities as owners of the project. X X X X Comments Relevance of Design. Includes the part of the IFI Work Quality/Bank Handling criterion in Private GPS that deals with pre-commitment work (OPs 21.1- 21.2). Similar to Public GPS OPs 22.8 – 22.10. Includes the part of the IFI Work Quality/Bank Handling criterion in Private GPS that deals with monitoring and supervision (OPs 21.1 - 21.3). Part of Private GPS OP 16.1 (Financial Performance and Fulfillment of Project Business Objectives). See Public GPS OPs 23.1 – 23.6 (Borrower Performance, covering Government Performance and Implementing Agency Performance). Evidentiary Requirements, Analytical Methods, and Benchmarks 4.8 Option 2’s evidentiary requirements, analytical methods, and benchmarks for a positive rating are shown in Table 6. 4.9 Highlights: Compared to both the Public GPS and Private GPS, the proposed evaluation methodology has more detail on evidentiary requirements and analytical methods to assess the achievement of development, policy, and transition objectives. It maintains the Private GPS’s evidentiary requirements for project, company, and IFI financial performance. - 31 - 4.10 To establish results attributable to the project, three of the four sub-criteria under Results require a comparison of observed outcomes with a counterfactual. The fourth sub-criterion – Operational Performance – is focused on the achievement of outputs and intermediate outcomes that are under the control of the project (for example, the physical implementation of investments in physical assets, the creation of credit lines) and thus does not require counterfactual analysis. The ratings guidance for the Results sub-criteria goes beyond the usual definitions that do not mention causality (e.g., in WB evaluation guidelines: “a positive rating requires that objectives be substantially achieved”) to bring attribution explicitly into the ratings. Thus, for example, a positive rating on Contribution to Intended Outcomes requires that there is strong evidence that the project caused the targeted improvement in outcome indicators compared to a without-project counterfactual. If outcome indicators improved but it cannot be shown that the improvement was due to the project, the Contribution to Intended Outcomes criterion would receive a less than satisfactory rating. Symmetrically, if outcome indicators deteriorated but it can be demonstrated that they would have deteriorated more in the absence of the project (meaning the project made a positive contribution compared to the counterfactual), the Contribution to Intended Outcomes criterion would receive a positive rating. All three attribution-related sub-criteria – Contribution to Intended Outcomes, Contribution to Corporate Goals, and Unintended Outcomes – allow for a “Not Rated” rating on an exceptional basis. If “Not Rated” is assigned, an explanation is provided (e.g., not applicable; insufficient evidence). A summary rating for each of the criteria (Relevance, Results, Efficient Use of Resources, Sustainability, IFI Performance, and Client Performance) is based on the ratings for the corresponding sub-criteria, with weights determined by the evaluator. An overall project performance rating would be based on the summary ratings for Relevance, Results, Efficient Use of Resources, and Sustainability, with each receiving equal weight. Other options: Do not allow the option of a “Not Rated” rating for Contribution to Intended Outcomes. The argument is that insufficient evidence should not be a reason for not rating, since this provides a perverse incentive to not provide any evidence if the project results are bad. The reasons for allowing a “Not Rated” rating are that projects with poor M&E systems should not be presumed to have produced negative results, and that there are other ways of holding IFI Management accountable for failing to provide sufficient evidence. At the project level, the quality of the M&E system enters into IFI Performance. At the portfolio level, the CED’s reports to the Board on portfolio performance and on the quality of the self evaluation system can include data on the proportion of projects that were not rated under each criterion, thus holding IFI Management accountable for providing insufficient evidence. - 32 - Table 6: Evidentiary Requirements, Analytical Methods, and Benchmarks Criteria/Sub-Criteria Relevance Strategic Alignment Evidentiary Requirements and Analytical Methods A comparison of the project’s intended results with the country’s development, policy, or transition priorities and with IFI country and sector assistance strategies and corporate goals as expressed in Poverty Reduction Strategy Papers, country strategies, sector strategies, and operational guidelines. For IFIs that do not prepare country strategies, consistency with other analyses of country conditions and needs. Also considers the clarity and realism of the project’s intended results. Benchmarks Substantial clarity and realism of project objectives and consistency with needs, policies, and priorities are required for a positive rating. Private sector operations: Same as above. Also considers the implementation of the IFI’s screening mechanisms at the pre-commitment stage. Rationale for IFI Involvement Strategic Alignment is assessed against priorities and conditions at the time of project entry. For operations that could potentially compete with the private sector (which may be the case for both public and private sector operations), the assessment requires evidence of the market failures that justify the intervention. Where applicable, it also discusses the rationale for targeting specific populations. If the rationale for intervention is based on social goals (such as redistribution), these are explained. Public sector operations: Also considers the comparative advantage of the IFI vis-à-vis other providers (national and sub-national governments, other donors, NGOs). Private sector operations: Assesses the rationale for IFI intervention as opposed to the client’s own resources or other external sources of support. Considers (i) financial additionality (whether the client would have been able to obtain sufficient financing/insurance from private sources on appropriate terms; the - 33 - A clear, evidence-based justification for IFI involvement and anticipated value added is required for a positive rating. Criteria/Sub-Criteria Evidentiary Requirements and Analytical Methods IFI’s catalytic role in mobilizing funds from other investors and lenders; the IFI’s role in reducing risks and providing comfort); and (ii) non-financial additionality (the IFI’s role in bringing about a fair allocation of risks and responsibilities; the IFI’s role in ensuring quality of project design and in supporting client capacity). Benchmarks The Rationale for IFI Involvement is assessed from the perspective of what was expected to occur at entry (at the appraisal or pre-commitment stage). Thus, the timing of the assessment is parallel to the timing of the Strategic Alignment sub-criterion. Results Operational Performance Achievement of Outputs Fulfillment of Project Business Objectives Contribution to Intended Outcomes Compares achieved values with both initial targets and revised targets, if the latter were formally revised during project implementation. In general, counterfactual analysis is not needed because by definition the outputs are under the control of the project and not influenced by external factors. For private sector operations, outputs would include the physical investments realized. Evidence on process objectives might include, e.g., the implementation of an investment plan, the establishment of a strong management team, or the introduction of an accounting system. In most cases, the project’s process and business outcomes will be attributable to the project, so there is no need for comparison to a without-project counterfactual. Achievements are compared to appraisal projections or performance targets. Relevant evidence would include baseline and at-evaluation values of performance indicators defined under the project as well as other quantitative and qualitative information relevant to the intended outcomes, including the project’s reach to target groups and mitigation of market failures. Outcomes are assessed against the project’s intended results as contained in appraisal and/or approval documents or as reconstructed by the evaluator. If the intended results and/or targets were revised during implementation, the assessment considers the project’s achievements against both the original and revised intended results and/or targets. - 34 - For a positive rating, there must be evidence that the project substantially achieved its targeted outputs. For a positive rating, there must be substantial achievement of the project’s process and business objectives compared to appraisal projections or performance targets. A positive rating on Contribution of Intended Outcomes requires that there is strong evidence that the project contributed (or is likely to contribute) substantially to intended outcomes. When the desired outcome is achieved but there is evidence that the results are primarily due to other factors, the rating is adjusted downward, accordingly. Criteria/Sub-Criteria Evidentiary Requirements and Analytical Methods Outcomes are compared against a without-project counterfactual. If feasible and practical, this is done using an impact evaluation; if not, plausible causality is established using a theory-based method: presenting evidence on the achievement of all levels in the results chain and testing the validity of assumptions. For private sector operations, the analytical method includes stakeholder analysis, i.e., the extent to which the project had its intended impact on employees, suppliers, competitors, and neighbors. For projects through financial intermediaries, the stakeholders include sub-borrowers and fund investees. The assessment also considers the extent to which financial intermediary projects reached their target groups. If the IFI-supported subborrowers cannot be identified, a before-and-after comparison of the financial intermediary’s portfolio is made to determine whether the intermediary increased its exposure to the target group, and then a theory-based method is used to established plausible causality to the project (for example, evidence that the intermediary had improved its marketing, screening, and credit procedures as a result of the project with the intention of increasing its reach to SMEs). Contribution to Corporate Goals Benchmarks The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether the observed outcomes moved in the “right” or “wrong” direction. For example, If outcome indicators met or exceeded targets, but there is evidence that the change was due mainly to external factors, a less than satisfactory rating is warranted. If outcome indicators deteriorated, but there is evidence that the decline would have been worse in the absence of the project, a positive rating is warranted. Assesses the incremental contribution of the project to broad corporate goals that are not part of the project’s intended results – for example, poverty reduction, rural poverty reduction, shared prosperity, gender equality, institutional development, social cohesion, EU policy, etc. On an exceptional basis,“Not Rated” is a possible rating when evidence is missing or weak. A substantial and plausible contribution of the project to the achievement of corporate goals must be shown to merit a positive rating. The assessment covers actual achievements in these areas, as opposed to expected achievements (the latter is covered under Relevance). It uses a theory-based approach to establish plausible association between the project and corporate goals. It also discusses other factors that could have affected the achievement of those goals. The assessment may be supported by evidence from other evaluations and research. As with the Contribution to Intended Outcomes, the rating reflects the project’s incremental contribution to observed outcomes, regardless of whether the observed outcomes moved in the “right” or “wrong” direction. On an exceptional basis, “Not Rated” is a possible rating when evidence is missing - 35 - Criteria/Sub-Criteria Unintended Outcomes Evidentiary Requirements and Analytical Methods Assesses positive or negative results of the project that were not included in the project’s statement of objectives. Uses a theory-based approach to establish plausible causality between the project and unintended outcomes. To be included, unintended outcomes must be truly unanticipated, attributable to the project, quantifiable, of significant magnitude, and at least as well evidenced as the project’s other outcomes. Where there are unintended benefits, an assessment should be made of why these were not "internalized" through project restructuring by modifying the project’s intended results. Benchmarks or weak. A substantial and plausible contribution of the project to the achievement of unanticipated outcomes must be shown to merit a positive rating. Positive impacts that are attributable to the project merit a positive rating; negative impacts that are attributable to the project merit a negative rating. The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether the observed outcomes moved in the “right” or “wrong” direction. “Not Rated” is a possible rating when there were no unintended outcomes or when evidence is missing or weak. Efficient Use of Resources Financial Performance of the Project/Company Assesses the incremental effect of the project on the financial performance of the company (or for Group D projects, the financial intermediary or fund). Financial performance is measured by the FRR or ROIC for the project; a comparison of appraisal financial projections; and other performance indicators from the company’s financial statements. For financial intermediary projects, the required information includes the financial performance of the financial intermediary or fund, and the financial performance of the sub-portfolio or fund portfolio. In evaluating financial performance, observed changes in performance are compared with a without-project counterfactual. The choice of method is appropriate to the project type. - 36 - For a positive rating, the project’s FRR or ROIC is greater than or equal to the company’s Weighted Average Cost of Capital (WACC), or actual performance meets or exceeds appraisal projections, and/or other financial performance indicators are in line with appraisal projections. For intermediation projects, there is adequate evidence that the marginal performance of the subportfolio had a positive effect on the financial intermediary’s profitability, or that the projected or realized net return Criteria/Sub-Criteria Economic Efficiency Evidentiary Requirements and Analytical Methods Assesses the extent to which the costs involved in achieving project objectives were reasonable in comparison with the project’s benefits, and the extent to which the project was implemented at least cost compared to alternative ways of achieving the same results. Cost-benefit analysis is conducted to the extent that data is available and it is reasonable to place a monetary value on project benefits. The costs and benefits of the project include both private and social costs and benefits, and extend to all affected stakeholders. Cost-effectiveness analysis compares the unit costs of the project, or component costs, with those of similar projects. Requires information on traditional measures of efficiency, e.g., FRR, ERR, NPV, unit rate norms (cost per unit of input or cost per unit of output), and service standards, as well as information on the cost of projects with similar objectives, scope, and design. For private sector operations, the calculation of the ERR begins with the financial benefit to the project company (the FRR or ROIC) and adjusts for taxes paid and subsidies received. It also includes any marginal employment effects and incremental impacts on other stakeholders. Care should be taken to not count the gross increase in employment or compensation without considering what they would have been in the absence of the project. The assumptions behind the calculations should be fully explained. The analysis shows the incremental impact of the project, i.e., the costs and benefits compared to the without-project counterfactual. The project’s Economic Efficiency should not be confused with the achievement of improved efficiency of the sector or program being supported. The latter is an outcome and would be included in the assessment the Contribution to Intended Outcomes. - 37 - Benchmarks on equity or net IRR to the fund’s investors is greater than or equal to the fund’s WACC. For a positive rating on Economic Efficiency, the Private GPS’s benchmark is that the ERR or EROIC should be greater than or equal to 1.2 times the project company’s WACC. The Public GPS’s benchmark is that the ERR should be greater than or equal to the opportunity cost of capital. In addition, project costs should have been equal to or less than the costs of alternative ways of achieving the same objectives. Substantial cost overruns would usually lead to a negative rating on Economic Efficiency, but if were unrealistically estimated at appraisal, they should instead enter into the rating for Quality at Entry. Criteria/Sub-Criteria Implementation Efficiency IFI Investment Profitability Evidentiary Requirements and Analytical Methods Measures other aspects of efficiency not included in Economic Efficiency, such as aspects of design and implementation that either contributed to or reduced efficiency. Implementation delays are a typical implementation inefficiency. The timeline of implementation is compared with the projected timeline at entry (the appraisal or pre-commitment stage), and reasons for differences are discussed. Measures the profitability of the IFI’s investment(s) in the project. The calculation of the profitability of the project to the IFI is based on either the investment’s net profit contribution or the quality of the investment’s gross profit contribution. The net profit contribution method is preferred, if cost accounting data is available. Gross profit contribution is applied in a largely qualitative manner as a proxy for likely investment performance. Sustainability Outcome Sustainability The assessment is based on (i) the likelihood that some changes may occur that are detrimental to the continuation of the project’s results or expected results; and (ii) the impact on the operation’s results of some or all of these changes materializing. Benchmarks Significant delays or other implementation inefficiencies would suggest a negative rating for Implementation Efficiency. For a positive rating, the net profit contribution is sufficient relative to the IFI’s target return on capital or overall profitability objectives. Detail by type of operation is contained in Private GPS OPs 20.2 – 20.5. A positive rating requires strong evidence that the expected value of risks is moderate to low. The risks may include technical, financial, economic, social, political, environmental, government ownership/commitment, other stakeholder ownership, institutional support, governance, and exposure to natural disasters. Commercial Sustainability Uses the evaluator’s judgment of the uncertainties faced by the operation’s results (intended outcomes, unintended outcomes, contribution to corporate goals) over its expected remaining useful life, taking account of any risk mitigation measures already in place at the time of evaluation. The forward-looking commercial viability of the company, financial intermediary, and/or sub-borrowers/fund investees. Considers the company’s (or financial institution’s or sub-borrowers’/fund investees’) adaptability and prospects for sustainability and growth. - 38 - An expectation of continued commercial viability in projected market conditions is required for a positive rating. Criteria/Sub-Criteria Compliance with Safeguards IFI Performance Quality at Entry Evidentiary Requirements and Analytical Methods Based on projected future financial performance and the performance of the company, financial institution, and/or sub-borrowers/fund investees relative to the market or sector peers. The Client’s compliance with applicable safeguard policies, if any, including implementation of the mitigation plan. Based on the degree of compliance with the IFI’s standards in effect at project entry, and the standards prevailing at the time of the evaluation. For public sector operations, based on the implementing agency’s management of its environmental and social impacts. For private sector operations, based on the project company’s management of its environmental and social impacts (and for financial intermediary operations, the environmental and social performance of sub-projects or fund investee companies). Measures the extent to which the IFI identified, facilitated preparation of, and appraised the operation such that it was most likely to achieve its planned outcomes and was consistent with the IFI’s fiduciary role. For public sector operations, this includes the quality of the IFI’s analytical work, the quality of the results framework; the quality of design of the monitoring and evaluation system; stakeholder and institutional analysis; risk assessment and plans for managing risks; the quality of implementation arrangements; the appraisal of environmental and social risks, and the inclusion of safeguards to mitigate them; and the appropriateness of the investment instrument selected. For private sector operations, this includes the quality of the IFI’s assessment of the operation as being relevant to the IFI’s corporate, country, and sector strategies; the quality of the results framework and the design of the monitoring and evaluation system; the assessment of sponsors, company, management, country conditions, market dynamics, project concept, configuration and cost; the appraisal of the financial plan, source of project funds, and assumptions used in the project’s financial projections; the assessment of project and political risks, and steps taken to mitigate them; the appraisal of environmental and social risks, and the inclusion of safeguards to mitigate them; and the appropriateness of the investment instrument selected. - 39 - Benchmarks For a positive rating, the project’s implementing agency or company (or financial institution, fund, sub-borrowers, and investees) must be in material compliance with applicable safeguards. For a positive rating, the IFI should have materially met its operational standards in these areas, and there were no significant shortcomings in project results due to the IFI’s performance at project entry. Criteria/Sub-Criteria Quality of Supervision Evidentiary Requirements and Analytical Methods For public sector operations, considers the IFI’s focus on development, policy, or transition outcomes throughout project implementation; the supervision of applicable fiduciary and safeguards aspects; the quality of supervision inputs and processes; the implementation of the monitoring and evaluation system; the candor and quality of performance reporting; and IFI’s role in ensuring adequate transition arrangements for regular operation of supported activities after project closing. Benchmarks For a positive rating, the IFI should have materially met its operational standards in these areas, and there were no significant shortcomings in project results due to the IFI’s supervision performance. For private sector operations, includes the completeness of supervision reports in documenting project status and risk; the monitoring of the client company’s compliance with the terms of the investment; the monitoring of the client company’s environmental and social performance, and adherence to relevant government regulations and IFI requirements; the adequacy and timeliness of the IFI’s response to emerging problems or opportunities; and the effectiveness of hand-over procedures should there be changes in IFI staff monitoring responsibilities. Client Performance Non-Financial Performance of the Company Government and Implementing Agency Performance Non-financial performance covers compliance with relevant government regulations and IFI requirements. For financial intermediary operations, includes compliance by the financial intermediary and sub-borrowers/fund investees. Assesses the extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, towards the achievement of intended outcomes. Includes such aspects as government ownership and commitment; the enabling environment for the project; adequacy of consultations with stakeholders; readiness for implementation; timely resolution of implementation problems; fiduciary management; compliance with environmental and social safeguards; adequacy of monitoring and evaluation arrangements; relationships with other donors and stakeholders; and adequacy of arrangements for the transition to normal - 40 - For a positive rating, the company or intermediary is in material compliance with relevant government regulations and IFI requirements, and subborrowers/fund investees are in material compliance with relevant government regulations and IFI requirements. For a positive rating, there were at most moderate shortcomings in the performance of the government and implementing agency or agencies. Criteria/Sub-Criteria Evidentiary Requirements and Analytical Methods operations after project closing. Benchmarks The evaluator should take account of the operational, sector, and country context in weighing the relative importance of each aspect of government and implementing agency performance as they affected outcomes. The next section illustrates the Option 2 methodology by applying it to two projects, one private sector and one public sector: an IFC investment operation with a mid-tier bank in an emerging market, designed to introduce a new financial instrument and expand the bank’s SME and mortgage financing, particularly in under-served areas of the country. Evaluative evidence was taken from IFC’s Expanded Project Supervision Report and IEG’s Evaluation Note. a World Bank rural roads project in India designed to achieve broader and more sustainable access to markets and social services in participating districts of the country. Evaluative evidence was taken from the WB’s Implementation Completion and Results Report and IEG’s ICRR Review. All ratings are on a four-point scale: Satisfactory, Partly Satisfactory, Partly Unsatisfactory, and Unsatisfactory (for Relevance, Results, Efficient Use of Resources, Overall Project Performance, IFI Performance, and Client Performance) and Likely, Somewhat Likely, Somewhat Unlikely, and Unlikely (for Sustainability). - 41 - Private Sector Application: IFC Tier I Bank Financing Project 1. PROJECT INFORMATION Project Name Country Borrower Approval Date Commitment Date IFC Financing Co-financing “Tier 1 Bank Financing Project”, or “The Project” “The Country” “The Bank” 5 July 2007 27 July 2007 Up to US$100 million subordinated Hybrid Tier I debt investment None Context Following macroeconomic stabilization in 2002 and a subsequent fall in interest rates, the loan portfolios of the Country’s banks had grown rapidly, becoming a significant part of their balance sheets for the first time in over 30 years. Despite the recent growth, access to finance remained a key constraint for a number of market segments. Small- and medium-scale enterprises (SMEs) continued to face inadequate access to long term credit, the central and eastern regions of the Country were underserved, and nascent mortgage financing had not received attention by the Country’s banks. On the funding side, most banks faced constraints in raising long-term funding as the Country’s capital markets lacked depth and sophistication, and capital resources were largely dependent on equity funding. A recent decision of the Country’s banking regulatory authority to increase the minimum capital adequacy ratio to 12 percent for banks expanding their branch networks, combined with rapid growth in riskweighted assets, made it imperative for many banks to raise capital. Hybrid Tier I capital was used widely since the 1990s by financial institutions in Europe and the USA, and in the years before the Project had become an attractive and useful instrument for banks in emerging markets. In an effort to make this capital instrument available to the Country’s banks, the regulatory authority introduced enabling legislation and regulations for Hybrid Tier I capital (called “Tier I” capital in the Country) in November 2006. However, local financial institutions and banks, unfamiliar with this instrument, had yet to take advantage of this regulation to provide themselves with much-needed capital and financial flexibility. Project Development Objectives According to the Board Report (pp. i, ii, 2, 4, and 5), the project intended to: open the market for Hybrid Tier I financing in the Country by (i) introducing the Hybrid Tier I financing instrument; (ii) demonstrating the feasibility of the Hybrid Tier I financing instrument and the effectiveness of its enabling regulations; and (iii) building market appetite for both Tier II capital and Hybrid Tier I capital by potential syndication of IFC’s existing Tier II capital investment in the Bank and eventually selling part of its Hybrid Tier I investment in the Project. increase the Bank’s SME financing in the Country, particularly in the underserved central and eastern regions of the Country. expand the availability of residential mortgage finance throughout the Country, thus increasing the level of home ownership. - 42 - Key performance indicators and targets were the following: Intended Outcome Catalyze the introduction of the Hybrid Tier I financial instrument; clarify the terms and intent of new regulations Demonstrate the feasibility of the Hybrid Tier I instrument and the effectiveness of its enabling regulations Increase the Bank’s financing of SMEs Expand the Bank’s residential mortgage financing Key Performance Indicators and Targets Successful issuance, with regulatory approval, of the Hybrid Tier I debt instrument Three financial institutions in the Country adopt similar Hybrid Tier I financing by 2009 Growth of the Bank’s total residential mortgage portfolio by at least 20% per year through 2012 (baseline: US$200 million in December 2006) Growth of the Bank’s residential mortgage portfolio by at least 20% per year through 2012 (baseline: US$200 million in December 2006) Increase in mortgage portfolio to 10% of total Bank loan portfolio by 2012 (baseline: 5.8% in December 2006) Expand the Bank’s SME and mortgage financing in underserved regions of the Country Increase in the Bank’s market share of the Country’s mortgage market to 5% by 2012 (baseline: 1.5% in June 2007). Increase in branches outside major urban centers by at least 100% by 2012 (baseline: 215). Most of the projected total of more than 500 branches would be in locations outside major urban areas. Project Business Objectives In addition to expanding the Bank’s SME and mortgage portfolios and the branch network, the Project aimed to strengthen the Bank’s capital structure, allowing the Bank to raise up to US$1.25 billion from other sources to increase its financial intermediation. Project Description The Bank was a leading mid-tier bank in the Country. It was 84 percent owned by a privately-owned financial services holding company that was in turn a 50/50 joint venture between a national business group with interests in steel, shipping, power, shipping containers, and financial services (42.1 percent ownership), and a large European bank (42.1 percent). Other shareholders owned 15.8 percent of the Bank. Since partnering with the European bank in 2005, the Bank significantly modified its strategy to become a full-service bank with a strong focus on SME and consumer finance segments to complement its historically strong corporate, private banking, and trade finance divisions. The Bank was a long-standing client of IFC in the Country. At entry, the indicative terms and conditions of the IFC investment were the following: Currency and amount Type Maturity Up to US$100 million Hybrid Tier I subordinated debt instrument eligible as Tier I capital under national banking regulations Perpetual legal maturity with repayment at the Bank’s option on the 10th anniversary of disbursement - 43 - Alternative payment mechanism/ conversion Interest rate Front-end fee Arrangement fee Commitment fee Subject to regulatory approval, payment of interest or principal to be made in the form of common shares of the Bank. Floating at 6 month LIBOR + 3.50% per annum with a step-up in spread on the 10th anniversary of disbursement 1.00% 0.30% 0.50% Hybrid Tier I instruments rank lower than Tier II subordinated debt but higher than common or preferred equity, and contain some equity-like features. The note is payable or pre-payable by the borrower with only regulatory approval, which in turn is based on capital adequacy requirements. As common practice, there is a significant increase in the interest rate after ten years, which serves as a strong incentive for the borrower to repay at that time. In accordance with national banking regulations, the Bank could withhold payment of interest on an interest payment date, and such interest would be non-cumulative. The alternative payment mechanism was to provide for option of payment in shares if case interest is not payable, subject to approval by the regulator and relevant corporate approvals. If the Bank did not repay the principal on the tenth anniversary of disbursement, IFC would have the option of requiring payment in shares of the Bank, subject to national regulatory and corporate approvals. 2. RELEVANCE: SATISFACTORY Strategic Alignment: Partly Satisfactory The Project’s intention to increase long-term financing of SMEs was likely to have been relevant to the development needs of the financial sector, although the Board Report does not provide complete evidence. The Board Report reports that credit penetration in the Country was only 39% of GDP, lower than emerging market peers, and that credit penetration in the SME segment was even lower. It also notes that the Country had the largest under-banked population in the region. However, the Board Report did not provide sufficient data to conclude that SME financing as a proportion of the SME sector’s GDP was less than in comparator countries. With respect to mortgage finance, the Board Report reports that the Country’s ratio of mortgage loans to GDP was only 4 percent in 2005, compared to 38 percent in the EU and 63 percent in the USA, and was low compared to some other emerging markets. The Board Report argues convincingly that there was a need to introduce a new and innovative capital instrument – Hybrid Tier I financing – to allow banks to support continued loan growth. The Hybrid Tier I product would allow banks to move toward better and more diversified capital funding which in turn would enhance the structure of their balance sheets and offer more competitive loan pricing to their borrowers. Prior to the Project, the Bank defined its SME portfolio as loan sizes less than US$1 million. The Board Report is not clear whether this definition would be used under the Project, or whether loan size was a good proxy for the size of borrowers. The XPSR provides data on the Bank’s SME portfolio using the IFC definition for the Country (loans in excess of US$2 million). The project’s intended outcomes were consistent with the World Bank Group’s Country Assistance Strategy of November 2003, which supported among other things the establishment of a sound banking sector. IFC’s country strategy aimed inter alia at expanding its client base to less developed regions of the Country and addressing regional inequalities. IFC’s Financial Markets Strategy aimed at strengthening financial systems and increasing access to finance, widening the financial product base as well as - 44 - deepening capital markets and broadening the investor base. The goal of increasing access to finance included delivery of financial products to underserved sectors, in particular SMEs and low-income households, while addressing regional inequalities. In line with this strategy, IFC’s SME financing was expected to remain an important activity, along with increasing emphasis on housing finance. Rationale for IFI Involvement: Satisfactory Although the enabling legislation governing Hybrid Tier I capital was adopted in November 2006, local financial institutions and banks, unfamiliar with this instrument, had not yet taken advantage of this instrument to access capital and enhance financial flexibility. By introducing this product and setting a market precedent, IFC was likely to provide greater comfort to international investors for such subordinated debt issues. The Board Report does not discuss the comparative advantage of IFC, vis-à-vis other sources of finance, in providing funding for SME and mortgage lending. The XPSR argues that, without the IFC investment, it would have taken the Bank longer to increase its SME portfolio and housing finance loans with respect to market position and regional presence. 3. RESULTS: PARTLY UNSATISFACTORY Operational Performance: Partly Unsatisfactory Achievement of Outputs: Not Rated The Project’s intended outcomes were expressed in terms of the volume of SME and mortgage financing provided by the Bank. In some other SME and housing finance projects, these would be considered “outputs” that would lead to broader development outcomes such as increased SME competitiveness and increased home ownership and housing quality. In this project, the expansion of SME and mortgage financing are considered to be intended outcomes, so they are not assessed in this section. Fulfillment of Project Business Objectives: Partly Unsatisfactory Strengthen the Bank’s capital structure: The Bank’s ratio of equity to total assets averaged only about 6.8 percent before the Project was approved in 2007, improving to 9.0 percent during 2007-2010 and to 10.9 percent in 2011-2012 after a capital increase resulting from a merger with another bank. The Bank therefore needed and continues to need Tier I and/or Tier II capital to meet the required regulatory capital adequacy ratio of at least 12 percent. Contribution to Intended Outcomes: Partly Unsatisfactory Catalyze the introduction of the Hybrid Tier I financial instrument; clarify the terms and intent of new regulations. The new instrument was introduced under the Project. Prior to the Project, IFC worked with the financial sector regulatory authority to clarify several aspects of the regulations and to obtain approval for this first issue. Demonstrate the feasibility of the Hybrid Tier I instrument and the effectiveness of its enabling regulations (target: three financial institutions in the Country adopt similar Hybrid Tier I financing by 2009). The Hybrid Tier I product was not replicated in the market as a new capital product. Right after the IFC deal was closed, the Lehman bankruptcy caused a significant increase in the cost of capital. It - 45 - became very expensive to issue such subordinated notes, as investors asked for large spreads. The lack of investor interest in the new product appears to have been mainly due to factors external to the project. Increase the Bank’s financing of SMEs. SME financing did not increase to the target level. According to the Reach Data provided by the Bank to IFC, the SME loan portfolio (using the IFC definition for the Country of up to US$2 million loan size) grew by 31 percent between 2007 and 2011 (target: 20 percent per year between 2007 and 2012, or 107 percent increase by 2011). The SME loan share of total loans actually fell from 67 percent in 2007 to 38 percent in 2011 due to the Bank’s diversification of the loan portfolio among mortgages, retail loans, micro loans, and corporate loans. Expand the Bank’s residential mortgage financing. The expansion in the Bank’s mortgage financing exceeded targets. The Bank’s mortgage loan portfolio grew by 400 percent between 2007 and 2011 (target: 20 percent per year between 2007 and 2012, or 107 percent increase by 2011), and the share of mortgages to total loans increased significantly from 6.5 percent in 2007 to 14.1 percent in 2011 (target: 10 percent by 2012). However, the Bank’s market share actually declined, from 1.5 percent in 2007 to below 1 percent at the end of 2011 (target: 5 percent). The XPSR explains the lack of growth of mortgage financing as the result of the 2008 sub-prime crisis which halted mortgage lending globally for at least three years. However, it is not clear why, in this environment, the Bank was able to increase its mortgage lending by 400% while at the same time losing market share. Expand the Bank’s SME and mortgage financing in underserved regions of the Country (target: increase of branches outside major urban centers by at least 100% by 2012). Although the Bank expanded the total number of branches from a baseline of 215 to 509 by the end of 2012 (2012 target: 500), its presence in less urban (underserved) areas did not increase. In 2011, the Bank merged with another bank, and the branches of the latter were mainly in urban areas. The XPSR notes that there has been no change in the Bank’s strategy of branch expansion to less urban areas, and that once the initial operational adjustments to the merger are made, the Bank will implement this strategy. Since the failure to achieve the targets for SME financing, mortgage financing, and expansion to underserved areas was under the control of the Bank and thus attributable to the Project, the Contribution to Intended Outcomes is rated Partly Unsatisfactory. Contribution to Corporate Goals: Not Rated None reported in the XPSR. Unintended Outcomes: Not Rated None reported in the XPSR. 4. EFFICIENT USE OF RESOURCES: PARTLY SATISFACTORY Financial Performance of the Project, Company, or Intermediary: Partly Unsatisfactory The XPSR does not provide information on the financial performance of the Project – i.e., the performance of the Bank’s additional SME and mortgage portfolio that resulted from the IFC investment. (In 2007, the Bank’s SME portfolio accounted for 59% of total loans, and the housing finance portfolio accounted for 9.3%.) The XPSR did not estimate any return on invested capital (ROIC) for the Project, even though the Board Report included the ROIC and the EROIC as indicators to be tracked or estimated annually by IFC. The - 46 - expected ROIC of 14.5% was projected to exceed the WACC by 5% and the expected EROIC of 15.3% was projected to exceed the WACC by 5.8%, both over the life of the project. In the absence of a ROIC and EROIC, the XPSR and Evaluative Note provide information on financial performance for the Bank as a whole. The Evaluative Note argues that, based on IEG’s evaluation experience with banking projects, projects that meet or exceed the ROAA “satisfactory” benchmark of 1.5% or better during the evaluation period will generally have a “satisfactory” ROIC when compared to the IEG WACC-based benchmark. The table below shows a few of the more important financial performance indicators and compares them with targets (projections in the Board Report and with IFC’s benchmarks for SME-oriented banks. More detailed financial performance information is contained in the XPSR and Evaluative Note. Indicator Non-performing loans Return on Average Assets (ROAA) Return on Average Equity (ROAE) Baseline (2007) 1.8% 0.9% 2012 Target 2012 Actual 1.2% 2.3% IFC’s SME Bank Benchmark < 4% >= 1.5% 13.5% 20.0% >= 15% 10.1% 2.3% 1.0% The Bank’s non-performing loans (including SME and mortgage financing as well as other loans) worsened from 1.8% in 2007 to 2.3% in 2012, underperforming the 2012 target of 1.2% but within the SME bank benchmark of less than 4%. The Bank’s ROAA increased marginally from 0.9% in 2007 to 1.0% in 2012, compared to a target of 2.3% and the SME bank benchmark of 1.5%. The Evaluative Note remarks that IFC’s ROAA and ROAE benchmarks are for SME banks that have only a moderate rate of branch expansion, and that due to the Bank’s extensive branch expansion program its ROAA and ROAE would be expected to be lower than the benchmarks. Given the Project’s aim to expand the Bank’s branch network to underserved areas of the Country, it is not clear why lower ROAA and ROAE targets were not projected at appraisal. Economic Efficiency: Not Rated As noted above, no financial rate of return (FRR) was calculated for the Project. Thus, there is no calculation of an Economic Rate of Return (ERR) based on the project’s FRR and adjusted for taxes and subsidies, net effects on other stakeholders, and externalities. The Evaluative Note reports that the Bank paid an effective income tax rate of between 21% and 37% before the Project, and estimated that the Bank paid at least US$240 million in income taxes during 20072012. There is no information to determine whether the Project generated consumer surplus in the form of interest rates lower than consumers’ willingness to pay. As noted above, the XPSR and Evaluative Note present information on the profitability of the Bank as a whole. The Evaluative Note states that the Bank’s 2011 merger with another bank adversely affected the financial and operating performance of the Bank due to merger-related costs and integration adjustments in operations. The rapid expansion of branches also negatively affected profitability during 2007-2010 because the new branches did not achieve profit break-even until about two years after starting operations. Efficiency improvements from the merger are expected to become significant starting in 2013. The Evaluative Note also states that the Bank’s low ROAA and ROAE during 2007-2010 were also the result of the global financial crisis that began in late 2007. - 47 - Implementation Efficiency: Satisfactory The Project was completed on schedule and there was no discussion of implementation delays in the XPSR or Evaluative Note. IFI Investment Profitability: Satisfactory According to the XPSR, IFC’s return on equity (ROE) for the project was 5.75%. The XPSR and Evaluative Note do not compare this figure with the IFC’s target return on capital or an overall profitability objective. The XPSR reports that the Bank had not been in any arrears on payments and that the asset was rated as being of good quality on IFC’s books. Both the XPSR and the Evaluative Note rate IFC’s Investment Outcome as “satisfactory”. 5. SUSTAINABILITY: LIKELY Outcome Sustainability: Likely The Bank continues to have a strategy of expanding its SME financing. The Bank aims to be not only an SME bank but also an advisor to SMEs. In line with this target, the Bank held a total of 53 SME Academy meetings in 33 cities since 2005; about 11,300 business men and women participated in these meetings and attended the training programs. Since 2006 the Bank has organized search meetings in thirteen cities and for four sectors to facilitate the preparation of SME strategies. The Bank has developed other products targeted specifically to SMEs, including a special hotline, SME TV, an SME training academy, and an SME Consultancy. Within SME Banking, the Bank introduced SME insurance services. Commercial Sustainability: Likely In explaining the Bank’s relatively low profitability despite its relatively low ratios of non-performing loans and of operating costs to average net loans, the Evaluative Note suggests that the Bank’s fee income is not high enough and/or its average interest rate on loans is not sufficiently high compared to its average borrowing cost. Nevertheless, the Evaluative Note argues that the synergies and efficiency improvements from the Bank’s recent merger will become significant starting in 2013. The profitability of new branches also is expected to improve. As noted above, the Bank’s capital adequacy ratio is below the regulatory minimum of 12%, and the Bank will need Tier I and Tier II capital. However, Tier I and Tier II capital are intended to be only short- to medium-term solutions to the undercapitalization of banks in IFC client countries. Eventually, this capital will need to be replaced by equity through retained earnings and/or injections of new equity. Because of the structure of the IFC investment, the Bank will need to raise equity from its existing shareholders when the IFC Hybrid Tier I subordinated debt reaches its 10th anniversary mark (mid-2017), when a step-up interest rate begins unless IFC decides to exercise its option to convert the subordinated debt to equity. Conversion to equity is not assured if the Country’s stock market is bearish at that time. On the positive side, the Bank’s two main shareholders appear to have the combined resources to refinance the IFC loan if necessary. The Bank’s portfolio risk is considered to be medium-high due to exposure to some medium-high risk sectors such as food, beverages, and tobacco (8%), metal (8%), chemicals (3.6%), and construction (8.2%). - 48 - Compliance with Safeguards: Satisfactory The Project was classified as a Category FI (Financial Intermediary) project according to IFC’s Environmental and Social Review Procedure. As an existing client, the Bank had a Social and Environmental Management System (SEMS) and the capacity to implement it. However, at project entry the SEMS along with the Bank’s own Environmental Risk Management Plan (ERMP) had not fully incorporated IFC’s Exclusion List and needed an improved system for monitoring the Bank’s investments. The Bank also needed a mechanism to update the SMES based on changes in local laws and ensure that it complies with IFC requirements. The Bank updated its SEMS giving further details on objectives and policies, a revised exclusion list complying with IFC guidelines, and additional sections on Community Engagement Policy and Social Risk Policy. However, during the last supervision mission it was noted that implementation of the SEMS still lacks adequate procedures for medium-high risk projects as no information was provided on risk categorization and IFC Exclusion List screening. Since the project met IFC’s at-approval and current requirements, and the Bank has the capacity to implement the SEMS, the Evaluative Note rated the Project’s environmental and social performance as “satisfactory” at evaluation. 6. OVERALL PROJECT PERFORMANCE: PARTLY SATISFACTORY Based on a Satisfactory rating for Relevance, Partly Unsatisfactory for Results, Partly Satisfactory for Efficient Use of Resources, and Likely Sustainability, overall project performance is rated Partly Satisfactory. 7. IFI PERFORMANCE: PARTLY SATISFACTORY Quality at Entry: Partly Satisfactory The Evaluative Note notes that IFC’s Tier I perpetual subordinated loan was an excellent investment structuring. The step-up interest rate is designed to encourage redemption of the IFC loan, which is important because the loan has no final maturity date. The conversion option allows IFC to improve its return if the conversion price and the future prospects of the Bank’s shares are favorable. However, the instrument involves risks for the Bank because its two main shareholders may not want to be diluted at that point in time. IFC’s project team assisted in the elaboration of regulations for the Hybrid Tier I instrument by highlighting some inconsistencies in the Tier I Communiqué of the regulatory authority, and the Communiqué subsequently was amended. According to the Country’s banking authorities, IFC played a major role in clarifying regulatory intent as well as refining specific, regulatory-permissible product features not initially understood by the authorities. Environmental and social risks were not adequately identified during appraisal. Although the project was correctly categorized as Category FI, considering the type of IFC investment and that the Bank’s portfolio included medium-high risk sectors, the applicable requirements should have included the IFC Performance Standards as well. As noted above, the project’s ROAA and ROAE targets may have been inconsistent with the Project’s aim to expand the Bank’s branch network to underserved areas of the Country. - 49 - The Board Report contained clear development objectives and associated key performance indicators to measure their achievement. However, the monitoring and evaluation system did not include a methodology to allow attribution of results to the IFC investment. Project preparation was efficient, with only seven weeks between the date of the mandate letter and commitment, including a full Board meeting. Quality of Supervision: Satisfactory IFC provided regular advice and recommendations on environmental and social safeguards. Annual supervision reports, quarterly credit rating reviews, and environmental reviews were conducted. IFC maintained a good working relationship with the Bank, going beyond its investment to invite the Bank to regional SME conferences. 8. CLIENT PERFORMANCE: SATISFACTORY Non-Financial Performance of the Company: Satisfactory The Bank was cooperative in providing IFC with regular and timely reporting, and the reports were of high quality. The Bank provided AEPRs on time with adequate portfolio details. Together with a consulting firm, the Bank updated its SEMS to ensure consistency with IFC requirements. There is no further information in the XPSR or Evaluative Note on other non-financial aspects of the Bank’s performance such as compliance with financial sector regulatory requirements. 9. RATINGS SUMMARY Relevance Strategic Alignment Rationale for IFI Involvement Satisfactory Partly Satisfactory Satisfactory Results Operational Performance Contribution to Intended Outcomes Contribution to Corporate Goals Unintended Outcomes Partly Unsatisfactory Partly Unsatisfactory Partly Unsatisfactory Not Rated Not Rated Efficient Use of Resources Financial Performance of the Project/Company Economic Efficiency Implementation Efficiency IFI Investment Profitability Partly Satisfactory Partly Unsatisfactory Not Rated Satisfactory Satisfactory Sustainability Outcome Sustainability Commercial Sustainability Compliance with Safeguards Likely Likely Likely Satisfactory - 50 - Overall Project Performance Partly Satisfactory IFI Performance Quality at Entry Quality of Supervision Partly Satisfactory Partly Satisfactory Satisfactory Client Performance Non-Financial Performance of the Company Satisfactory Satisfactory 10. LESSONS Regulations for new and untested financial products need to be thoroughly discussed with the regulator to avoid problems with their issuance. In this project, the regulation on Hybrid Tier I instrument had never been tested, and there were some legal loopholes that needed to be taken care of before the Project could proceed. Although the regulator understood the problems, it took considerable time and effort to make the amendments. The commitment of the main project shareholders to the project’s development objectives is the key to achieving them. In this project, IFC expected that the Bank would diversify its geographic coverage of lending from highly urban areas to less urban areas. The Bank did increase the number of branches, but mainly through a merger with another bank that had branches mostly in urban areas. Public Sector Application: World Bank Rural Roads Project 1. PROJECT INFORMATION Project Name Country Borrower Approval Date Closing Date Project Cost at Appraisal Project Cost at Completion WB Financing Co-financing Rural Roads Project India Government of India 23 September 2004 31 March 2012 US$ 310 million US$ 312 million US$400 million (IBRD loan US$100 million, IDA credit US$300 million) None Context At the time of appraisal of the Rural Roads Project in 2004, the rural population in many states of India was suffering from poor physical access due to a lack of all-weather roads. This was constraining economic activities in rural areas and preventing the rural population from being fully integrated into the economy and accessing essential services. At that time, an estimated 300,000 habitations (about 40 percent of the 825,000 habitations in India) were without all-weather road access. The low levels of access of habitations to all-weather roads was partly a result of the lack of adequate maintenance on the existing large rural road network of approximately 2.7 million km and partly a lack of capital investment. In addition, the rural road sector was suffering from relatively low levels of technical and planning - 51 - capacity at the state level that hindered the ability of road agencies to absorb new investment funds and manage their assets properly. In order to address the problem of poor rural access and improve planning and management of funds, the Prime Minister’s Rural Road Program (Pradhan Mantri Gram Sadak Yojana, or PMGSY)) was announced in late 2000. By the time of project appraisal, PMGSY had completed or was close to completing works connecting 22,100 habitations with a population per habitation of less than 1,000. This was against PMGSY targets of providing new connectivity to about 178,000 habitations, involving construction of about 375,000 km of roads, and upgrading 372,000 km of existing rural roads in poor condition. Challenges faced at that time included a shortfall in funds, capacity constraints of road agencies and contractors, and inadequate road maintenance. Project Objectives To achieve broader and more sustainable access to markets and social services in participating districts of India (Loan Agreement Schedule 2, p. 19). The main beneficiaries of the project were the people in the habitations that were to be connected by allweather roads. Anticipated project benefits included all-season access to economic opportunities and social services, employment generation for new construction and maintenance, and improved access to transport services. Participating states also would benefit indirectly from improvements in the economy resulting from the project. Project Description The project had four components: Component 1: New connection and upgrading of core rural roads networks (appraisal estimate US$430.9 million; actual cost not available). New construction and upgrading of 9,900 km of the core rural road network in four states (Himachal Pradesh, Jharkhand, Rajasthan, and Uttar Pradesh) to provide all-weather road access to habitations in identified project districts. This component would also finance consultants in each state to undertake independent technical examination of WB-funded works and to verify compliance with relevant safeguards, technical standards, and the quality of bidding documents. Component 2: Periodic and routine maintenance of core rural road network in project areas (appraisal estimate US$235 million, to be financed by state governments; actual cost not available): Annual implementation of the periodic and routine maintenance program in every district where the WB was to provide additional investment (about 100,000 km of roads). Component 3: Institutional development (appraisal estimate US$13.6 million, actual cost not available): Technical assistance for institutional development at both the national and state levels. Component 4: Incremental operating cost (appraisal estimate US$4.4 million; actual cost not available). - 52 - 2. RELEVANCE: SATISFACTORY Strategic Alignment: Satisfactory Project objectives remained consistent with the Country Assistance Strategies at appraisal and completion. The FY 2009-2012 Country Assistance Strategy (CAS) aimed at achieving rapid, inclusive growth, ensuring that development is sustainable, and increasing the effectiveness of service delivery. To achieve rapid, inclusive growth, the CAS aimed to assist in removing infrastructure and skills constraints to growth in both urban and rural areas. Project objectives were in line with the Government’s priority of addressing the problem on poor rural accessibility and improving the planning and management of rural roads. The Prime Minister’s Rural Roads Program was launched in late 2000 and the project was part of that program. The Program target was to connect all habitations of more than 1000 people (500 in the case of hilly, desert, and tribal areas) by the year 2009 and upgrade 194,000 rural roads by the year 2010. These targets also were reflected in India’s Eleventh Five Year Plan (2007-2012). Rationale for IFI Involvement: Satisfactory The project sought to introduce the notion of partly linking PMGSY grants to sound management of the entire core rural road network. Further, by demonstrating the value of sector planning and management reforms in the pilot states or districts in partnership with the Government of India (GOI), the project sought to increase the likelihood that similar reforms would be implemented in other states and districts. GOI had set itself very ambitious connectivity targets that required substantial additional resources if they were to be met on time, and the World Bank, both through IDA and as needed through IBRD, had the financial capacity to contribute to investment in this public good. Thus, the ex-ante financial additionality of the project was positive. 3. RESULTS: PARTLY SATISFACTORY Operational Performance: Satisfactory Achievement of Outputs: Satisfactory By project closure, the physical works for new construction were completed. About 9,625 km of rural roads were constructed (97 percent of the target of 9,900 km), providing better access for 78 participating districts. Institutional development was supported by the development of a pavement design manual; independent reviews of engineering designs through technical examiners; and the preparation of a rural roads manual by the Ministry of Roads and Development with separate technical specifications for rural roads. However, not all of the changes were implemented, and the use of cost-effective optimal designs employing locally-available material was inadequate. The project provided training in maintenance management systems, quality control, environmental management, and e-procurement. - 53 - Contribution to Intended Outcomes: Partly Satisfactory The project contributed to increased access to markets and social services due to the construction of roads. Overall, about 93 percent of eligible habitations were connected, compared to the target of 60 percent. About 50 percent of the routes were in good condition in 2012, compared to the target of 43 percent. However, routine maintenance funding is not fully assured, raising concern about the extent to which “sustainable access” will be achieved (see also the Sustainability section). Due to inadequate evidence on some of the intended outcomes (e.g., access to social services) and also on attribution of the reported outcomes to the Project, Contribution to Intended Outcomes is rated Partly Satisfactory. Information on access to social services and markets is limited to data on the Government’s entire rural road program, rather than to the roads funded by the Project. The data show state-level achievements that surpassed the Government’s targets. Percent of Eligible Habitations with All-Weather Access to Social Services and Markets Uttar Pradesh Himachal Pradesh Jharkhand Rajasthan Baseline (%) 50 40 35 40 Target (%) 55 60 60 65 Actual (%) 97 78 69 86 To evaluate the economic and social benefits of the program, the WB undertook a survey in the four states supported by the Project. Road user satisfaction was assessed on the basis of four factors: reliability, transit time, connectivity, and user-friendliness. Overall, the satisfaction level was high. The survey also found: Child vaccination increased by 8 percent due to improved access to health centers. The number of female patients visiting private doctors increased 8 percent. The population in the income category of “more than 10,000 Rupees per month” increased by 3.46 percent. The area cultivated increased by 12 percent. Travel time was reduced. However, these improvements cannot be completely attributed to the Project. Contribution to Corporate Goals: Not Rated No other contributions were reported. Unintended Outcomes: Partly Satisfactory As noted below under Sustainability, in 2006 serious deficiencies in safeguard preparation in Uttar Pradesh were discovered, and corrective measures were required on at least some of the roads financed by the project. There is no evidence on the severity of environmental damage or the extent to which it was reversed or mitigated. - 54 - The project’s training and capacity building activities have been mainstreamed. Simple maintenance management systems were developed by Rajasthan, Himachal Pradesh, and Uttar Pradesh (but not by Jharkhand), and these were used to prepare the annual maintenance plans for the respective states. 4. EFFICIENT USE OF RESOURCES: SATISFACTORY Economic Efficiency: Satisfactory The ex-post ERR was relatively high at 15 percent, although it was below the appraisal estimate of 19 percent. The lower-than-estimated ERR was due to increased construction cost and time overruns. Cost escalation was very high in Himachal Pradesh (34.5 percent), followed by Uttar Pradesh (7.6 percent). The ERR calculation assumed a 12 percent cost of capital and reinvestment rate. Himachal Pradesh Jharkhand Rajasthan Uttar Pradesh Total Completed Road Length (km) 984 126 6,287 2,228 9625 Ex-Ante ERR (%) 21 18 17 25 19 Ex-Post ERR (%) 11 17 15 18 15 Implementation Efficiency: Partly Satisfactory There were initial delays in the institutional strengthening and maintenance components. The project closed after a two-year delay in order to: (i) consolidate the institutional development initiatives; (ii) complete the ongoing and additional rural roads works reallocated to Uttar Pradesh (US$27 million) and Rajasthan (US$9 million); and (iii) undertake further capacity building activities planned for rural roads agencies. 5. SUSTAINABILITY: SOMEWHAT UNLIKELY Outcome Sustainability: Somewhat Unlikely Routine maintenance funding is not fully assured. Experience has been mixed with respect to the maintenance of the core road network in the participating districts. Initially, maintenance of the core road network was neglected, and it received attention mainly towards the last two years of the project. Only after the Ministry of Rural Development stipulated to the States that construction funds would be released only if the States would specifically earmark funds for maintenance, did the situation improve. Routine maintenance until 2017 is funded under the 13th Finance Commission, which approved US$4.2 billion for rural roads. However, there is no information on whether this amount is adequate. Routine maintenance will be carried out by the same contractor who constructed the roads under the Project. However, there is no information on who will guarantee and supervise the contractor. Compliance with Safeguards: Partly Satisfactory The Project was assigned an Environment Category A. The following safeguards policies were triggered: Environmental Assessment, Natural Habitats, Cultural Property, Involuntary Resettlement, Indigenous Peoples, and Forests. - 55 - Environment: At appraisal, the Government carried out an environmental assessment in the four project states and formulated an Environment and Social Management Framework (ESMF) to address environmental issues in the Project. A stand-alone document called Environmental Codes of Practice was prepared covering aspects such as sub-project selection and planning, construction camp management, topsoil conservation, debris management, public consultation, drainage, and worker safety. This approach helped minimize the need for sub-project level environmental assessments and environmental management plans by mainstreaming environmental issues into the selection, planning, design, and construction stages of the project. In almost all of the participating states, some roads, especially in the initial phases, were delayed due to lack of forest clearances. In subsequent phases, with the insistence of having a prior forestry clearance in hand before the bids were invited, such situations were almost completely avoided. Since the construction of rural roads was on existing alignments, the extent of impact on environmental features such as fertile farmland, orchards, trees, sacred groves, water bodies, and religious structures was minimal. Social: At appraisal, the Government carried out a social assessment in all of the Project states through an independent consulting agency. The assessment found minimal land requirements for road widening, and community willingness to volunteer for land donation. The ESMF adopted a voluntary land donation approach. Following the identification of serious deficiencies in safeguard preparation in Uttar Pradesh in 2006, the WB conducted a complete audit of social safeguards preparation and implementation for all project roads and developed a time-bound action plan for retrospective corrective measures. A series of training programs for field officers were conducted at the Divisional level. Indigenous Peoples: There is no information on compliance with Indigenous Peoples safeguards. Compliance with Safeguards is rated Partly Satisfactory rather than Satisfactory due to a lack of information on compliance with Indigenous Peoples safeguards. 6. OVERALL PROJECT PERFORMANCE Based on a Satisfactory rating for Relevance, Partly Satisfactory for Results, Satisfactory for Efficiency, and Somewhat Unlikely for Sustainability, overall project performance is rated Partly Satisfactory. 7. IFI PERFORMANCE: PARTLY SATISFACTORY Quality at Entry: Partly Satisfactory Project objectives were clearly stated, and there was a logical link between project objectives and project activities. Project preparation involved extensive public participation. The project was technically sound and benefitted from the WB’s experience with the implementation of the Government’s rural road program in other states. Safeguards assessment was adequate. However, the design of the monitoring and evaluation framework was weak, as the performance indicators referred to the performance of the Government’s program as whole rather than the performance of the part supported by the Project. - 56 - Some risks were underestimated, including the risk of poor road designs and the risk of delayed or improper procurement by rural road agencies. In the case of Jharkhand, the planned construction of roads was substantially reduced because of weak capacity and resulting delays, which should have been foreseen. Another risk which was not identified during preparation was the reluctance of states to use WB funds when funds could also be accessed through the Prime Minister’s Rural Roads Program, which had fewer requirements in terms of safeguard and fiduciary compliance. Quality of Supervision: Partly Satisfactory A total of 17 supervision missions were carried out, on average twice in each fiscal year. The supervision missions identified issues which were likely to affect the achievement of development outcomes. With the help of close monitoring by the WB, some states showed substantial improvement. Supervision of safeguard and fiduciary aspects was satisfactory. However, an earlier reallocation of funds from Jharkhand to Uttar Pradesh and Rajasthan would have made an extension of the closing date unnecessary. Also, when funds were reallocated, corresponding targets were not revised, which made it difficult to monitor the performance of different states. 8. CLIENT PERFORMANCE: PARTLY SATISFACTORY Government Performance: Satisfactory The Government was fully committed to the project. Detailed guidelines were prepared by the Ministry of Rural Development for planning, standards and specifications, and procurement. The Ministry of Rural Development and the National Rural Roads Development Agency provided close monitoring whenever slow decision-making threatened to delay certain activities. For example, the initially slow procurement, poor compliance with safeguards, and slow roll-out of the computerized financial management system in states such as Uttar Pradesh, Himachal Pradesh, and Jharkhand were met with increased pressure and resources from the National Rural Roads Development Agency. The Agency undertook state-by-state monitoring reviews and provided further training, improved documentation, and stricter enforcement. Implementing Agency Performance: Partly Satisfactory Of the four states, the performance of the State Road Agencies in Rajasthan and Himachal Pradesh was satisfactory. Uttar Pradesh initially had frequent changes in senior management, inadequate staffing, and weak coordination among agencies, but managed to overcome these problems with assistance from the WB. The implementing capacity of Jharkhand was unsatisfactory from the beginning, especially with respect to institutional capacity and a willingness to resolve implementation problems in a timely manner. Compliance with fiduciary requirements as well as inadequate ownership and staffing remained problematic. Eventually, in September 2009, proceeds earmarked for Jharkhand were reallocated to Rajasthan and Uttar Pradesh. Implementation of the Environment and Social Management Framework was weak in Uttar Pradesh. Following actions by the WB and the Government, there was a substantial improvement in safeguards compliance. Rajasthan and Himachal Pradesh performed well on social safeguards implementation. Internal and external audit reports found weaknesses with respect to delays in WB reconciliations, inadequate controls over fund authorization, stale bank guarantees, and non-settlement of advances. The states took remedial actions to address these weaknesses, but the results varied across states. An Interim Performance Review carried out by the WB suggested mitigation measures for weak contract management. - 57 - Procurement capacity varied considerably among the states. Rajasthan demonstrated adequate procurement capacity, but procurement in Jharkhand remained slow throughout the project. Goods, works, and services were generally procured in accordance with WB guidelines except for some procedural errors in the procurement of works. 9. RATINGS SUMMARY Relevance Strategic Alignment Rationale for IFI Involvement Satisfactory Satisfactory Satisfactory Results Operational Performance Contribution to Intended Outcomes Contribution to Corporate Goals Unintended Outcomes Partly Satisfactory Satisfactory Partly Satisfactory Not Rated Partly Satisfactory Efficient Use of Resources Economic Efficiency Implementation Efficiency Satisfactory Satisfactory Partly Satisfactory Sustainability Outcome Sustainability Compliance with Safeguards Somewhat UnLikely Somewhat Unlikely Partly Satisfactory Overall Project Performance Partly Satisfactory IFI Performance Quality at Entry Quality of Supervision Partly Satisfactory Partly Satisfactory Partly Satisfactory Client Performance Government Performance Implementing Agency Performance Partly Satisfactory Satisfactory Partly Satisfactory 10. LESSONS In projects that fund a small portion of a large government program, it is important to agree on harmonized standards for fiduciary and safeguard requirements across the Program and the Project. In this project, differences in procurement guidelines between the WB and the Prime Minister’s Rural Roads Program caused problems in implementation. In projects that are to be implemented in different states or regions with very different implementation capacities, project design should build in some flexibility to allow tailoring of technical assistance to the needs of each state. This should be based on a critical analysis of the strengths and weaknesses in institutional capacity at the state level. - 58 - Annex 1: Glossary of Terms and Definitions Term Definition Aggregate Project Performance Indicator In the Public GPS, a single measure of overall project performance constructed from ratings on the core evaluation criteria (Relevance, Effectiveness, Efficiency, and Sustainability). Benchmark Reference point or standard against which performance or achievements can be assessed (OECD-DAC). Borrower Performance The adequacy of the Borrower’s assumption of ownership and responsibility during all project phases, including government and implementing agency performance in ensuring quality preparation and implementation, compliance with covenants and agreements, establishing the basis for sustainability, and fostering participation by the project’s stakeholders. Broad economic and social goals Sector-wide and/or economy-wide goals that are not included in the project’s statement of objectives but nevertheless are of interest in the evaluation. Completion Report A record of an operation’s performance at the end of its implementation phase, undertaken as a self-evaluation by an IFI operations unit. Completion Report Validation A review of Completion Report findings by the Central Evaluation Department, normally as a desk study. Core Criteria In the Public GPS, the principal criteria that form the basis for evaluating project performance. For evaluations of investment/TA loans, the core criteria are Relevance, Effectiveness, Efficiency, and Sustainability. For evaluations of PBLs, the core criteria are Relevance, Effectiveness, and Sustainability. Corporate goals Areas of special focus of the IFI, such as poverty reduction, rural poverty reduction, transition impact, social cohesion, gender equality, shared prosperity, inclusive growth, environmental sustainability, EU policy, propoor innovation, etc. Cost-benefit analysis A quantitative analysis performed to establish whether the present value of benefits of a given project exceeds the present value of costs. Cost-effectiveness analysis 1. A quantitative analysis that compares the relative costs and outcomes of two or more courses of action. Cost-effectiveness analysis can be used to show whether the outcomes were delivered at least cost compared to alternative ways of achieving the same outcomes. Direct evaluation Evaluations undertaken directly by the CED, as opposed to indirectly by the IFI. Equivalent to independent evaluation in the Public GPS. Economic Rate of Return (ERR) The internal rate of return of the time series of the project’s economic costs and benefits. Measures the quantifiable net economic benefits to society. In addition to those identified in financial analysis, may include taxes paid to the government, consumer surplus, effects on competitors, and benefits to suppliers, including labor. Also includes externalities to the extent that they can be quantified. Economic Return on Invested Capital (EROIC) The internal rate of return on the economic costs and benefits of a project, including costs and benefits to customers, employees, government, suppliers, competitors, local residents, etc. The EROIC is calculated only when the ERR on the project cannot be calculated. It is calculated by - 59 - Term Definition adjusting the ROIC for factors normally taken into consideration when adjusting the FRR to the ERR. Effectiveness The extent to which the project achieved (or is expected to achieve) its stated objectives, taking into account their relative importance (Public GPS, OECD-DAC) Efficiency The extent to which the project has converted its resources economically into results (Public GPS, OECD-DAC). Evaluability The extent to which the value generated or the expected results of a project are verifiable in a reliable and credible fashion. Extended Annual Supervision Report (XASR) A stand-alone, ex-post private sector project evaluation conducted by an IFI operations department (Private GPS) Extended Annual Supervision Report Assessment (XASR-A) A CED validation of an XASR (Private GPS). Financial Rate of Return (FRR) The internal rate of return of a time series of real, after-tax cashflows describing the project’s financial investments and returns. Fund Weighted Average Cost of Capital (FWACC) The cost of capital for a private equity or listed fund, estimated by calculating the average cost of debt based on the country composition of the fund, and then levying a premium for the combined equity instrument and project risk. Goal The higher-order objective to which a development intervention is intended to contribute (OECD-DAC). IFI Performance The quality of services provided by the IFI during all project phases, including the IFI’s performance in ensuring project quality at entry, satisfactory implementation, and future operation (Public GPS). Impact Positive and negative, primary and secondary long-term effects produced by a development institution, directly or indirectly, intended or unintended (OECD-DAC). Impact evaluation An evaluation that quantifies the net change in outcomes that can be attributed to a specific project or program, usually by the construction of a plausible counterfactual. Independent evaluation In the Public GPS, a project evaluation conducted by the CED. Equivalent to a “direct evaluation” in the Private GPS. Independent evaluation instruments include Performance Evaluation Reports (PERs) and Expanded Annual Supervision Report Assessments (XASRs) for private sector operations, and Performance Evaluation Reports (PERs) and Completion Report Validations (CR Validations) for public sector operations. International Financial Institution A financial institution created by a group of countries that provides financing and advisory services for projects and programs in member countries. The term includes the World Bank, regional development banks, and other regional financial institutions. It does not include internationallyoperating commercial banks. Net present value The sum of the present values of the time series of project costs and benefits. Outcome The final level in the results chain, reflecting the objectives of the project. If necessary, a distinction can be made between “intermediate outcomes” - 60 - Term Definition (the uptake of project outputs by beneficiaries, behavioral changes on the part of beneficiaries, results achieved once beneficiaries use project outputs, and/or results achieved in the short-to-medium term) and “final outcomes” (the ultimate outcomes or project goals arising from intermediate outcomes). “Final outcomes” are called “impacts” by some IFIs. Output A product, capital good, or service which results from an IFI intervention (OECD-DAC). Output The tangible goods and services that the project activities produce, generally under the direct control of the implementing agency (OECD-DAC). Performance Evaluation Report An independent project evaluation conducted by a Central Evaluation Department, which normally includes field work (Public GPS, Private GPS). Project A public or private sector investment, technical assistance, or program that is supported by an IFI loan or grant. Project Outcome Rating In the Private GPS, a synthesis rating reflecting the ratings on the core criteria of Financial Performance and Fulfillment of Project Business Objectives, Economic Sustainability, Contribution to IFI Mandate Objectives, and Environmental and Social Performance. Relevance Consistency of the project’s objectives with beneficiary needs, the country’s development or policy priorities and strategy, and the IFI’s assistance strategy and corporate goals. Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention (OECD-DAC). Results chain A model that sets out the sequence of inputs, activities, and outputs that are expected to lead to the project’s intended outcomes. Describes the causal relationships, indicators, and the assumptions or risks that may influence project success and failure. Alternatively called a “results framework”, “causal chain”, or “logical framework (logframe)”. Return on Invested Capital (ROIC) The internal rate of return on invested capital in real terms, i.e., the FRR on the costs and benefits to the company as a whole on a before-after, rather than a with-without, basis. Self evaluation In the Public GPS, a project evaluation conducted by IFI Management. Equivalent to an “indirect evaluation” in the Private GPS. Self evaluation instruments include Completion Reports (CRs) for public sector operations and Expanded Annual Supervision Reports (XASRs) for private sector operations. Sustainability The likelihood of continued long-term benefits, and the resilience to risk of net benefit flows over time (OECD-DAC). Theory-based evaluation An analysis that establishes a plausible association between the various links in the project’s results chain, using quantitative and qualitative evidence as well as evidence from other evaluations and academic literature. Weighted Average Cost of Capital (WACC) The weighted average after-tax cost to the company of the yields it must provide on its borrowings and the equity investors’ minimally acceptable terms, all adjusted for inflation. - 61 - Annex 2: Recent Developments in ECG Members European Investment Bank Rather than publishing evaluations of individual projects, the Evaluation Department (EV) at EIB conducts thematic evaluations that contain evaluations of each of the projects covered in the thematic report. Some of these thematic evaluations cover both public sector and private sector operations. EV uses a common set of evaluation criteria that apply to both types of evaluations. Although EV’s evaluation methodology has not yet been formalized in the form of written guidelines, it can be seen in three recent thematic evaluations: Evaluation of EIB Investment Loans for Economic and Social Cohesion in France, Portugal, and the United Kingdom (“Cohesion evaluation”, April 2011); Evaluation of EIB’s Energy Efficiency Financing in the EU, 20002011(“Energy Efficiency evaluation”, May 2012); and Evaluation of EIB’s Intermediated Lending to SMEs (“SME evaluation”, June 2012). A review of pairs of projects from each of these three thematic evaluations – one public sector, one private sector – reveals how the evaluation criteria and benchmarks were defined. The six projects were: Cohesion evaluation, public sector: Construction of new waste incinerator and a storage and transfer facility for organic waste. Financed by an EIB loan to a municipal government agency. Cohesion evaluation, private sector: Redevelopment of a former military airfield into a commercial airport and associated business and commercial development. Financed by an EIB loan to a private company. Energy Efficiency evaluation, public sector: regional investments in solar photovoltaic equipment on the roofs of public buildings; other renewable energy installations including solar water heaters and other renewable energy sources such as biomass or geothermal; equipment to improve energy efficiency in public buildings, as well as in street and traffic lighting. Financed by an EIB loan to a regional government and intermediated by participating financial institutions. Energy Efficiency evaluation, private sector: development of remote digital electricity metering infrastructure throughout the client country, replacing existing electromechanical equipment, thereby allowing remote reading and several interactive services. The project included investments in data network and management centers. Financed by an EIB loan to a private power distribution company. SME evaluation, public sector: credit to SMEs throughout the client country, in particular for energy efficiency and renewable energy sources and for restarting entrepreneurs after a period of inactivity. Financed by an EIB global loan to a majority state-owned bank. - 62 - SME evaluation, private sector: credit to SMEs and microenterprises. Financed by an EIB global loan to a private credit and leasing company. Table A1 shows the definitions of the evaluation criteria that were applied to both public and private sector operations. Table A1: EIB Evaluation Criteria in Six Project Evaluations Criterion Definition Relevance The extent to which the objectives of a project are consistent with EU policies; EIB priorities; national, regional, and local strategies; and beneficiaries’ needs. Relevance also takes account of the internal coherence of objectives and the relevance of the design. It also covers the identification of risks and measures to mitigate them. Effectiveness The extent to which the objectives of the project have been achieved, or are expected to be achieved, taking into account their relative importance, while recognizing any change introduced in the project since loan approval. Effectiveness considers the achievement of operational objectives (“outputs”), intermediate objectives (“results” or “outcomes”) and global objectives (direct and indirect effects following from the outcomes, corresponding to, often longerterm, “impacts”). For example, in the Energy Efficiency evaluation of a private sector project, the operational objectives were modernizing the company’s electricity monitoring; intermediate objectives were a reduction in operational costs and an improved competitive position; and global objectives were to develop a building block for smart grids, reduce greenhouse gas emissions, maximize the effects of market liberalization, and support regional development. Efficiency The extent to which project benefits/outputs are commensurate with resources/inputs. At appraisal, project efficiency is normally measured through the financial and economic rates of return. In public sector projects a financial rate of return is often not calculated at appraisal, in which case the efficiency of the project is estimated by cost effectiveness analysis. Efficiency also includes such aspects as the project’s physical implementation, schedule, and procurement; costs and financing; operation (management, use, and employment); and cooperation and coordination with other counterparts. Sustainability The likelihood of continued long-term benefits and the resilience to risk over the intended life of the project. The assessment covers physical and operational sustainability; organizational, institutional and regulatory sustainability; environmental and social sustainability; and financial and economic sustainability. Environmental, Economic, and Social Impact Environmental impact considers two aspects: Compliance with environmental safeguards, including EU and/or national as well as EIB guidelines. Environmental impact beyond safeguards compliance. - 63 - Criterion Definition Economic and social impact includes, for example, employment created outside of the project, the creation of new businesses, etc. EIB Contribution EIB Contribution considers two aspects: EIB Management of the Project Cycle (or “EIB Management”) EIB Financial Contribution -- the financial contribution provided in relation to the alternatives available, including facilitating co-financing from other sources (catalytic effect). Other EIB Contribution (optional) relates to any significant non-financial contribution to the operation provided by EIB – e.g., improvements in the technical, institutional, economic, or other aspects of the project. EIB’s handling of the operation, from project identification and selection to postcompletion monitoring. An overall project performance rating combines the ratings on Relevance, Effectiveness, Efficiency, and Sustainability. The Environmental, Economic, and Social Impact criterion is rated “below the line”. All criteria are rated on a four-point scale: Relevance, Effectiveness, Efficiency, Sustainability, are rated as Excellent, Satisfactory, Partly Unsatisfactory, and Unsatisfactory. The overall assessment reflects the individual assessments within the same scale. Individual assessments on EIB Contribution are rated as High, Significant, Moderate, and Low. Individual assessments on EIB Management of the Project Cycle are rated Excellent, Satisfactory, Partly Unsatisfactory, and Unsatisfactory. Inter-American Development Bank IDB recently re-designed their approach to development effectiveness for IDB Non-Sovereign Guarantee (NSG) operations.7 The new approach is aligned with the broader discussion regarding the IDB Group’s private sector operations, where development effectiveness and the measurement of development impact are placed front and center. The envisioned changes are intended to “raise the bar” for NSG projects by requiring broader and deeper analysis of project logic, sector context, and other factors. 7 IDB (2013). Revised Development Effectiveness Framework for Non-Sovereign Guarantee Projects. Revised version. Policy and Evaluation Committee, 28 June. Private sector operations in the IDB Group are also provided by the Inter-American Investment Corporation (IIC) and the Multilateral Investment Facility (MIF). - 64 - The previous Development Effectiveness Matrix (DEM) was designed to improve project evaluability at entry, monitoring during implementation, and evaluation at completion. Experience with the DEM revealed some shortcomings. The DEM scoring system created incentives for project teams to try to score as many sub-sections as possible, regardless of their importance for the project. This lessened the focus and relevance of the DEM. In addition, the DEM’s information requirements placed a burden on IDB staff, suggesting that a more streamlined approach was needed. The new Development Effectiveness (DE) Toolkit attempts to harmonize the evaluability approach between IDB’s NSG and Sovereign Guarantee (SG) projects. The DE Toolkit has four components: (i) a DEM Summary; (ii) a DEM Worksheet; (iii) an Evaluability checklist; and (iv) a Results Matrix. The DEM Worksheet has three main sections that correspond to the evaluation criteria in the ECG Private Sector GPS, fourth edition: Development Outcome, based on (i) Project/Company Business Performance; (ii) Economic Sustainability; (iii) Private Sector Development; and (iv) Environmental and Social Policy Assessment. Additionality, consisting of (i) Financial Additionality and (ii) Non-Financial Additionality. Strategic Alignment, against IDB Strategic Development Objectives and Country Strategy Development Objectives. Instead of performance area scores being calculated as a weighted average of all possible subsections, the DEM performance areas are to be scored based only on factors or sub-sections that are most relevant within each performance area. This is intended to provide a clearer picture of each project’s intended development results. The Evaluability Checklist has three sections: the justification for IDB intervention, the project’s developmental objectives, how the project addresses the identified market failure or development problem, and the intended beneficiaries. It assesses the quality of the empirical evidence provided to support the intervention, and a verification that the vertical logic of the project holds. an assessment of the quality of the project’s financial and economic analysis, in accordance with the standards of the ECG GPS. For real sector projects, ex ante costbenefit analysis is carried out to calculate FRR and ERR (or ROIC and EROIC); for financial intermediary projects, the analysis includes the financial performance of financial institutions as well as the economic sustainability of their portfolio. an assessment of the monitoring and evaluation plan, in order to ensure meaningful ex post evaluation as well as tracking and reporting on development results. The list of indicators includes not only those in the DEM and Results Matrix, but also the tracking indicators useful for monitoring the achievement of IDB corporate goals. - 65 - The Results Matrix includes output and outcome indicators with baseline and target values. The Results Matrix allows for a sharpened focus on those indicators that are most relevant for measuring the project’s achievements of their stated development objectives. The DEM Summary contains summary scores from the DEM Worksheet and the Evaluability Checklist. The DE Toolkit is to be supported by IT solutions to increase efficiency. All information and data points are to be entered only once, and then are automatically fed into different tables including the DEM Summary and the Results Matrix. DEMs and DEM scores are to be updated annually as part of NSG Project Supervision Reports. Monitoring activities will focus mainly on the tracking of performance indicators included in the Results Matrix, which will be reflected mainly in the updated DEM scores mostly for Project/Company Business Performance and Economic Sustainability. For other areas, any major events or milestones will be tracked, and if there is any difference between actual and expected project performance, the corresponding DEM scores will be updated. European Bank for Reconstruction and Development EBRD’s Evaluation Department (EvD) is currently preparing a guidance note on EBRD’s evaluation performance rating system. The objective is to strengthen the existing project performance rating system and extend it to a wider range of evaluation products. At the time of the writing of the present paper, the guidance note was still being written. However, some of its features can be gleaned from the Approach Paper8 and from conversations with EvD management. Among the principles underlying the new evaluation performance rating system are the following: The rating system should provide a robust measure of performance. This means that the criteria and sub-criteria included in the overall performance assessment are those that are directly attributable (or largely attributable) to EBRD. Those criteria to which the operation contributes but is not solely or largely responsible, or which are the consequence of success rather than a determinant of it, may be assessed but not included in the overall performance rating. To the extent possible, a common framework should apply to all types of evaluation that rate performance (including self evaluation and independent evaluation) as well as all objects of evaluation – investments, frameworks, technical cooperation (TC), non-TC grants, strategies, and policies. There may be some differentiation and customization at lower levels (sub-criteria and benchmarks) and to the weights used in aggregating criteria ratings to the overall performance rating. 8 EBRD (2013). Guidance Note: Evaluation Performance Rating System. Approach Paper. Evaluation Department, March. - 66 - The rating methodology should follow internationally accepted good practice standards, principally those of ECG and the Evaluation Network of OECD-DAC. Guidance should indicate the type of evidence that should be provided. The evaluation criteria should reflect EBRD’s unique mandate EvD currently uses seven evaluation criteria for the performance rating of operations, two of which have sub-criteria: Transition Impact Realized TI at the time of evaluation TI that can still be achieved Risk attached to achieving remaining TI Environmental and social performance and change Environmental and social performance of the project and the sponsor Extent of environmental and social change Additionality Project and Company Financial Performance Fulfillment of Project Objectives EBRD Investment Performance Bank Handling of the Project The Approach Paper suggests that a mapping of the existing criteria to the standard OECD-DAC criteria could be a useful starting point, and gives an example of how to do this: Relevance Effectiveness Efficiency Sustainability Impact Additionality Fulfillment of Project Objectives Outputs and outcomes associated with transition, environmental, and social impact that are largely attributable to the project (mostly client-level transition impact, and extent of environmental and social change, and environmental and social performance of the project). Project and company financial performance FIRR and/or other financial ratios demonstrating efficient funds use Bank investment performance Bank handling in terms of efficiency of process Environmental and social sustainability (rather than the sustainability of results). However, it would be possible to consider company financial performance at the time of evaluation under “effectiveness” and projected future financial performance as evidence of the sustainability of results. Remaining transition impact and the risk to achieving this could be included under Sustainability. Transition Impact at the sector and economy-wide levels Environmental and social performance of the sponsor - 67 -