SYNTHESIS REPORT Learning from IDA Experience Lessons from IEG Evaluations, with a Focus on IDA Special Themes and Development Effectiveness © 2019 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2019. Learning from IDA accuracy of the data included in this work. The Any queries on rights and licenses, including Experience: Lessons from IEG Evaluations, with boundaries, colors, denominations, and other subsidiary rights, should be addressed to a Focus on IDA Special Themes and information shown on any map in this work do World Bank Publications, The World Bank Development Effectiveness. Synthesis Report. not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC Independent Evaluation Group. Washington, World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: DC: World Bank. territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Learning from IDA Experience Lessons from IEG Evaluations, with a Focus on IDA Special Themes and Development Effectiveness April 9, 2019 Contents Abbreviations ....................................................................................................................................................vi Acknowledgments ......................................................................................................................................... vii Key Messages ................................................................................................................................................. viii Management Response ................................................................................................................................ 1 1. Introduction .................................................................................................................................................. 1 IDA18 and Its Special Themes................................................................................................................... 1 Purpose, Scope, and Limitations of the Report ..................................................................................3 2. Development Effectiveness of IDA Operations and Country Programs................................. 5 Development Effectiveness of IDA Operations ......................................................................................5 Development Effectiveness of IDA Country Programs ........................................................................8 Enhancing IDA’s Country Engagement: Lessons from Recent Country Program Evaluations .....................................................................................................................................................11 Lessons from the Small States Cluster Country Program Evaluations (FY05–15) ................................... 11 Lessons from the Rwanda CPE (FY09–17) ........................................................................................................ 13 3. Evaluation Findings Relevant to the IDA18 Special Themes .................................................... 14 Jobs and Economic Transformation ..................................................................................................... 14 Competitiveness and Jobs .................................................................................................................................... 14 Inclusive Business ..................................................................................................................................................... 17 Gender and Development ....................................................................................................................... 18 Gender in IDA Country Strategies ...................................................................................................................... 18 Gender in IDA Operations..................................................................................................................................... 21 Climate Change.......................................................................................................................................... 23 Climate Change in the World Bank Group Country Strategies and Project Portfolio ..................... 23 Pollution Management and Carbon Finance ................................................................................................ 25 Fragility, Conflict, and Violence............................................................................................................. 26 Governance and Institutions .................................................................................................................. 29 Governance in SCDs and CPFs ........................................................................................................................... 29 Tax Revenue Mobilization .................................................................................................................................... 29 iii Enhancing Institutions for Infrastructure Development and Service Delivery ................................... 30 Citizen Engagement ................................................................................................................................................ 31 4. Monitoring, Evaluation, and Results .................................................................................................. 33 Monitoring and Evaluation ..................................................................................................................... 33 Bank’s Performance in Ensuring Quality at Entry and Quality of Supervision ....................... 35 Data ................................................................................................................................................................ 36 Bibliography..................................................................................................................................................... 37 Boxes Box 2.1. Achieving a Climate Change-Related Objective: The Case of Bolivia ............................ 11 Box 3.1. Corporate Commitments on Gender ........................................................................................ 19 Figures Figure 2.1. World Bank Project Outcome Ratings, by Instruments and FCV Status .....................7 Figure 3.1. Share of IDA Country Strategies with Climate Change Objectives or Outcomes 23 Figure 3.2. Share of Climate Change Adaptation and Mitigation Activities in the World Bank’s Project Portfolio ................................................................................................................................. 24 Figure 4.1. IDA Projects: Quality of Monitoring and Evaluation Ratings, FY06–17 .................... 34 Tables Table 1.1. Special Themes under IDA16, IDA17, and IDA18 .................................................................. 3 Table 2.1. Distribution of IEG CLR Outcome Ratings, FY16–18 ........................................................... 9 Table 2.2. Extent of IDA Special Theme Coverage in IDA Country Program, FY16–18 ............. 10 Table 2.3. IEG Efficacy Ratings by Objectives under IDA Special Themes (percent).................. 10 Appendixes Appendix A. IDA18 Special Themes and Their Focuses ......................................................................41 Appendix B. World Bank Project Performance ..................................................................................... 43 Appendix C. Additional Findings from Reviews of Country Strategies ......................................... 48 Appendix D. The Incorporation of Gender in Projects ....................................................................... 53 Appendix E. Methodology for Climate Change Analyses ................................................................. 56 Appendix F. Bank Performance and Monitoring and Evaluation Ratings .................................... 58 Appendix G. Monitoring and Evaluation Issues in Project Performance Assessment Reports .... 64 iv Appendix H. List of Independent Evaluation Group Evaluations .................................................... 68 v Abbreviations CDD community-driven development CLRR Completion and Learning Report Review CPE Country Program Evaluation CPF Country Partnership Framework DPF development policy financing FCV fragility, conflict and violence GP Global Practice ICR Implementation Completion and Results Report IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation LIC low-income country M&E monitoring and evaluation MIGA Multilateral Investment Guarantee Agency PDO project development objective RAP Results and Performance of the World Bank Group SCD Systematic Country Diagnostic vi Acknowledgments This report was prepared by an Independent Evaluation Group (IEG) team composed of Yumeka Hirano (task team leader), Ali Khadr, and Stephen Hutton under the guidance of Jeffrey Chelsky (manager) and Auguste Tano Kouame (director) and the overall direction of Alison Evans (Director-General, Evaluation). Franz M. Loyola, Swizen Rubbani, Yoshine Uchimura, and Disha Zaidi provided additional data analyses and research. Carla Fabiola Coles provided administrative support. Peer reviewers were Gaiv M. Tata (former manager, Resource Mobilization, World Bank; director, Growth Solutions for Development LLC) and Erika A. Jorgensen (former economic adviser, Macroeconomics and Financial Management Global Practice, World Bank). The team wishes to thank many IEG and World Bank Group management colleagues for their comments on earlier drafts of this document. vii Key Messages Development Effectiveness, Monitoring, and Evaluation The development effectiveness of International Development Association (IDA) operations is improving but lagging in fragility, conflict, and violence (FCV) countries.1 The share of exiting IDA operations rated moderately satisfactory or better (MS+) on outcome rose from 68 percent in FY12–14 to 76 percent in FY15–17. But the share of projects with MS+ outcome ratings in IDA FCV countries declined between the two periods, with the decline driven by the portfolios in Afghanistan and, to a more limited extent, the Republic of Yemen. Despite the improvement in outcome ratings at the project level, IDA country program outcomes remain weak. Outcome ratings for IDA country programs completed over FY16–18 were 41 percent MS+, substantially below the corporate target of 70 percent. A disconnect between project and country strategy ratings can reflect incomplete alignment of the project portfolio and the priorities identified in the associated Country Partnership Framework (CPF). Strengthening IDA’s results orientation for maximum development impact requires better monitoring and evaluation (M&E). M&E quality in IDA projects is improving, but M&E design and country data gaps require greater attention. M&E quality was rated substantial or above in 36 percent of projects exiting in FY15–17, up by 6 percentage points over FY12–14 (41 percent of projects weighted by commitment volumes, an increase of 11 percentage points). These favorable trends reflect efforts at the corporate level, including more training on results reporting by Operations Policy and Country Services and the Independent Evaluation Group (IEG). However, a review of Project Performance Assessment Reports covering 57 projects in 22 countries and eight regional projects—all in IDA countries—found shortcomings in M&E design, particularly in choice of results indicators, which in turn affected M&E implementation and use. A special effort is required in the context of FCV where an IEG evaluation found that only a third of programs assessed recorded fragility-specific outcomes, and only a quarter 1 Because each Independent Evaluation Group evaluation looks at operations and activities approved over a 5- to 10-year period leading up to it, their findings are derived mostly from experience before the 18th International Development Association (IDA18), and as such may not remain relevant in the IDA18 period. Moreover, many of these evaluations did not focus solely on the experience of IDA countries. Nevertheless, many of the findings have relevance to the IDA18 special themes and can inform IDA decision-making. To the extent that data relate to the performance of projects that were approved/financed under previous IDA cycles, conclusions do not reflect performance under IDA18. viii Key Messages had M&E systems specifically designed to track FCV issues (World Bank 2016g). Lack of data at the country level has been found to be a constraint to quality M&E in IDA- funded projects. Data gaps deserve more attention in IDA countries to improve M&E. In this regard, IEG’s evaluation of data for development found that systemwide approaches to building statistical capacity are most promising. IDA performance in ensuring quality of supervision have improved marginally in recent years, but continued effort is needed if larger IDA commitment volumes are to be translated into greater development impact. The share of IDA projects for which quality at entry was rated moderately satisfactory or higher (MS+) improved from 58 percent in FY11–12 to 61 percent in FY15–17, although it declined when weighted by commitment volumes (from 68 percent in FY12–14 to 65 percent in FY15–17). Key factors contributing to poor quality at entry include complexity of project design and lack of realism in proposed implementation schedules. Though higher, quality of supervision ratings exhibited a similar pattern. The share of IDA projects with MS+ ratings increased from 79 percent in FY12–14 to 80 percent in FY15–17 (by number of projects), while the share of MS+ ratings weighted by commitment volumes decreased from 85 to 80 percent during the same period. Factors affecting quality of supervision range from weak project management, weak fiduciary management, low safeguards compliance, inadequate attention to technical issues and M&E (World Bank 2016e). These issues require attention from management. Special Themes The IDA18 special themes have been reflected in country strategies and operations, albeit with variation across themes. Among the special themes, jobs and economic transformation, climate change, and governance and institutions were more frequently reflected in the objectives of country strategies relative to other themes. There has been notable progress in addressing climate change and gender in both IDA country strategies and operations. For example, between FY08–10 and FY15–17, there was an increase of roughly 50 percent in the share of project components in the IDA portfolio with potential climate change benefits. Similarly, there has been growing integration of gender in IDA country strategies. However, the achievement of objectives related to the special themes has varied across themes. The World Bank Group achieved substantial results with respect to objectives related to gender and jobs and economic transformation in IDA country programs, with more than half the objectives receiving a rating of mostly achieved or higher. Results under climate change-related objectives and governance and institutions-related were more limited, with mostly achieved or higher ratings of 44 percent and 24 percent, respectively. ix Key Messages The jobs and economic transformation special theme has been well integrated into country strategy objectives, backed by increased analytical work, but there is scope for stronger integration into operations. Almost all Completion and Learning Reviews reviewed by IEG noted the inclusion of the jobs and economic transformation theme in country strategy objectives. The new country engagement model appears to have facilitated this. The intensity of analytical work on job diagnostics and measurement has increased significantly in Systematic Country Diagnostics and CPFs. However, only a small proportion of the Bank Group portfolio supporting this agenda explicitly references jobs in objectives, interventions, or indicators. IEG’s evaluation on industry competitiveness and jobs suggests that pursuit of job creation in individual operations should to be approached from a multisector perspective (World Bank 2016b). Although uptake of gender in Systematic Country Diagnostics for IDA countries has been good, carryover into CPFs has lagged and progress in integrating gender into operations varies. CPFs have tended to reflect Systematic Country Diagnostics in identifying key gender issues, but they have not necessarily provided concrete actions or entry points for addressing them. Gender integration in IDA operations has also been uneven across Global Practices. Community-Driven Development operations and those related to the development of the rural nonfarm economy have been relatively successful in addressing gender, though closer tracking of access to opportunities for women is needed. Increasing IDA attention to climate change is a positive sign; other environmental challenges in IDA countries also deserve attention. There was a modest increase in the share of country strategies containing an explicit objective with potential climate change benefits for both mitigation and adaptation over FY08–17. The data also show roughly a 50 percent increase in the share of project components with potential climate change benefits for both climate change mitigation and adaptation between FY08–10 and FY15– 17. However, IEG’s Results and Performance of the World Bank Group 2017 found that, at the World Bank–wide level, increases in support for climate change adaptation and mitigation have been accompanied by declining support for other environmental challenges, especially in LICs (World Bank 2018b). For example, indoor air pollution, which is a major concern in IDA countries, has received little attention. Also, damage from toxic substances including lead, electric waste, and pesticides may be severe in LICs, but an absence of data means that these problems might not be noticed or addressed. In FCV-affected countries, country program priorities should pay closer attention to the drivers of fragility, more forcefully incorporating the Risk and Resilience Assessments that are prepared for an increasing number of countries and investing in the pathways to sustaining the recovery from crisis and conflict. IEG’s evaluations found that country x Key Messages strategies lacked tailoring to fragility and conflict drivers and the flexibility to adjust if risks materialize. A recent IEG review of seven Systematic Country Diagnostics / Country Partnership Frameworks (CPFs) in FCV countries found that the Bank Group has enhanced its analytical capacity to engage in situations of FCV through Risk and Resilience Assessments. However, CPFs did not always cover the fragility drivers identified by the Risk and Resilience Assessments. There is also a need for a more holistic approach to private sector development and greater attention to capacity building, which are two key pathways to sustaining the recovery from crisis and conflict. More holistic approaches to private sector development that go beyond support for livelihoods and short-term employment schemes are needed and can help improve prospects for longer-term sustainability of program results. Also, IEG’s evidence indicates that capacity building is critical to the sustainability of public service delivery in FCV-affected countries. As part of the governance and institutions special theme, the increasing engagement of citizens in projects, CPF preparation, and policy formulation needs to be continued. Corporate commitment to mainstream citizen engagement generated awareness and buy-in among senior management and staff. The corporate commitment helped increase the percentage of projects using citizen engagement mechanisms, especially those that were not safeguard related. More CPFs explicitly identified the objective of engaging citizens. Engaging citizens in IDA operations can be broadened and deepened with concrete steps, including building capacity, strengthening monitoring and reporting, and regular outreach. Support to Small States The increase in the IDA’s envelope during the IDA18 replenishment will particularly benefit small states. The increase in the base allocation and a policy framework for providing small states with substantial concessional resources and other provisions has boosted their access to IDA financing. Benefits could be further enhanced if risk preparedness and resilience building extend beyond infrastructure to incorporate policy and regulatory changes. Past IDA financing has enabled Bank Group programs in small states to contribute significantly to building disaster and climate change resilience by helping to improve preparedness capacity and making infrastructure more resilient to disasters. However, addressing small states’ disaster vulnerabilities more effectively requires focus on policy and regulatory changes to shift incentives toward resilience building. xi Management Response Management welcomes the Independent Evaluation Groups (IEG)’s three synthesis reports related to the International Development Association (IDA): (i) Learning from IDA Experience: Lessons from IEG Evaluations, with a Focus on IDA Special Themes and Development Effectiveness; (ii) Synthesis Report on IDA Regional Window Program, 2003–17; and (iii) IDA’s Crisis Response Window: Lessons from IEG Evaluations. Together, the reports provide a useful summary of the existing evaluative evidence and provide valuable inputs to the IDA19 replenishment discussions. While this report is informative, covering areas related to IDA18’s five special themes, we note that it synthesizes IEG work on subjects or countries for which IEG completed evaluations since FY16, and so it is not, in itself, a comprehensive assessment of IDA’s performance and effectiveness. For example, regarding the Governance and Institutions theme, evidence is drawn from a review of Systematic Country Diagnostics (SCDs) and Country Partnership Frameworks (CPFs), reports on domestic resource mobilization and on citizen engagement, and a few other sector evaluations with institutional development issues. These represent a small subset of IDA’s governance-related work and do not cover important topics under IDA18, such as public financial management or state-owned enterprise reforms. That said, IEG itself acknowledges the limitations of this review, and we commend and value the IEG team’s effort to synthesize evidence and recommendations from different sources and extract relevant lessons. Management is pleased to see that outcome ratings for IDA-financed operations have improved in most areas, except for fragility, conflict and violence (FCV)-affected countries. During FY15–17, the share of non-FCV IDA operations with satisfactory outcomes has increased by 11 percentage points to 79 percent, compared to FY12–14. The exception is in the IEG-validated ratings for FCV-affected countries, where the ratings have essentially stagnated. As the report points out, the stagnation in the ratings for FCV countries (from 69 percent to 65 percent by project count) is likely to reflect “the challenges of operating in an increasingly volatile and conflict-ridden environment,” including most recently in Afghanistan and Yemen. Management is aware of the challenges to enhancing effectiveness in FCV situations, and successive IDA replenishments have responded by refining the policy agenda on FCV. As IDA attempts further to expand its support in country conditions of increasing uncertainty and volatility, the risks to achieving intended objectives will increase, requiring IDA to show greater agility and adaptability for course corrections during implementation. Management also welcomes the finding that recently closed CPFs increasingly reflected the priorities of the IDA18 special themes—in particular, jobs and economic transformation, climate change, and governance and institutions. In this context, it is not surprising that fewer gender-related or FCV-related CPF objectives have been found. As gender efforts are mainstreamed, CPFs have opted to include gender as a cross-cutting 1 Management Response theme integrated into all CPF objectives. Similarly, CPFs prepared for fragile and conflict-affected situations may not have narrowly-defined “FCV-related objectives,” as CPFs often pursue economic or social objectives in an FCV context, reflecting FCV sensitivity in the choice of sectors and themes for the CPFs, geographic targeting of programs, and implementation modalities within operations (for instance, by consistently addressing exclusion). Management notes that IDA country program outcomes during FY16–18 were modest, calling for intensified attention and actions. Nonetheless, it is premature to argue, without further analysis, that the disconnect between project and country program ratings may indicate “incomplete alignment of project portfolio objectives” with the CPF priorities (paragraph 2.8). Significant volatility has been observed in annual averages in these outcomes, as the number of Completions and Learning Reviews (CLRs) differs significantly from year to year and the samples are small (for example, only six IDA CLRs in FY18). The FY16–18 period is affected, for instance, by low ratings observed in FY16–17 (45 percent and 46 percent) compared to FY15 (67 percent) or FY19 (57 percent). The overall trend since FY16 seems to be positive in this regard. The lower average in some years could reflect the challenges and risks we face in achieving CPF-level outcomes that are higher in results chains, such as inclusive growth or improved governance, which are influenced by a country’s overall economic performance, government leadership, and political economy. The valuable insights drawn from recent evaluation findings related to the special themes should be interpreted with caution due to the information gaps and time lapse since these evaluations were conducted. For instance, the statements on climate change based on the Results and Performance 2017 report (paragraph 3.18) do not take into account the important roles played by non-lending activities such as carbon finance and Advisory Services and Analytics. On gender, the report does not cover the contributions of the Regional Gender Action Plans (an IDA17 commitment) in generating awareness of gender dimensions in country programs and operations, or the new approach through the gender tag that focuses on more context-specific actions. Approaches evolve rapidly and continuous efforts to address the issues raised in recent evaluations have shown notable progress over the past few years. The findings and recommendations cited in the report need to be triangulated with other available, updated information. On FCV, the IDA18 scale-up and differentiated approach are changing the way the Bank Group works in diverse and challenging FCV settings. Integration of FCV issues into SCDs and CPFs has been improving as experience from the increasing number of Risk and Resilience Assessments (RRAs) accumulates and enhanced support and guidance are provided to operations staff, including through IDA18’s Risk Mitigation Regime and Refugee Sub-Window. Some of the recent CPFs include explicit approaches and actions arising from the RRA findings (for example, Guinea, Niger, Tajikistan). Similarly, on Governance, the finding from the IEG 2017 report that little attention was paid and limited support provided to tax policy and 2 Management Response administration in low-income countries has been overridden by the enhanced support and tools provided to staff since 2017 and fivefold increases in annual average lending during FY16–19 compared to FY11–15 (see footnote 40). Management appreciates the report’s mention of the slow but steady improvements in recent years in the exit ratings of quality of monitoring and evaluation (M&E) for IDA- financed operations and acknowledges the need to continue to strengthen M&E design and implementation, including particularly in the choice of results indicators. A number of initiatives have been undertaken over the past few years to improve M&E quality, including: a renewed emphasis on the intervention logic or theory of change to articulate the links between specific interventions and expected outcomes at the CPF or operational levels; development of new learning modules, or updates of existing ones, on M&E and results frameworks; and expanded provision of training and learning opportunities on M&E tools, good practice examples or innovative approaches (for example, Results Academy, CPF academy, various M&E-related operations clinics, and learning events organized by the Results Measurement and Evidence Stream). Management will continue its effort to ensure the operational staff receive support and guidance on M&E when needed, improve guidance and resources available to staff, and encourage greater use of M&E data and necessary mid-course corrections. Finally, regarding IDA interventions intended to help promote employment (paragraph 3.4), Management notes that the limited explicit reference to jobs in project objectives or indicators is a result of a deliberate effort to define intended outcomes that are reasonably attributable and achievable by the end of projects. World Bank operations are typically oriented to have indirect effects on jobs, and thus the creation of jobs is rarely directly attributable to World Bank interventions. For instance, a project may train youths to make them more employable or extend credits to entrepreneurs to start businesses, which should help beneficiaries find employment or create additional jobs. However, job creation is neither guaranteed (due to external factors such as job market conditions or the overall business environment) nor necessarily observable by the end of the project (due to the time lag). 3 1. Introduction The 18th replenishment of the International Development Association (IDA18), one of the world’s major providers of financial resources to the poorest countries, was the largest in the institution’s 56-year history. Together with significant changes in its policy and financing framework, IDA’s enhanced commitment authority was expected to enable faster progress toward the international community’s far-reaching and ambitious 2030 agenda,1 which aligns closely with the World Bank Group’s twin goals of eradicating extreme poverty and boosting shared prosperity in a sustainable manner. IDA18 and Its Special Themes Under IDA18, IDA deputies, borrower representatives, and Bank Group management agreed on an ambitious and transformational policy and financial package, highlights of which include the following: • Leveraging donor contributions by raising capital market financing to expand commitment authority under IDA18 to $75 billion over FY18–20 compared with $52 billion under IDA17 (the previous record); • Enhancing support to fragility, conflict, and violence (FCV) and small states to accelerate progress toward the 2030 Agenda; • Mobilizing private sector investment through the Private Sector Window; • Promoting regional interventions and crisis preparedness and response; and • Strengthening IDA’s results orientation, monitoring and evaluation (M&E), and corporate effectiveness. The IDA18 special themes reflect substantial continuity over the past decade but also address key emerging issues (table 1.1). Toward 2030: Investing in Growth, Resilience and Opportunity was chosen as the overarching theme for IDA18.2 Five special themes were 1This agenda, agreed in 2015 (which includes agreement on the Sustainable Development Goals, the Conference of Parties 21 agreement on climate mitigation, adaptation and finance, the Addis Ababa Action Agenda, and the Sendai Framework for disaster risk management) represents a paradigm shift in the development dialogue. 2 The overarching theme for 17th replenishment of the International Development Association (IDA17) was maximizing development impact and the one for IDA16 was delivering development results. 1 Chapter 1 Introduction selected to orient IDA support: 3 (i) jobs and economic transformation; (ii) gender and development; (iii) climate change; (iv) FCV;4 and (v) governance and institutions. Aligning IDA support with these special themes was intended to help more effectively address frontier issues confronting IDA countries while furthering the twin goals, ensuring that support is tailored to country contexts, and delivering it through a combination of regular IDA country allocations and special windows—all mutually compatible and reinforcing. It also served to sharpen the focus on results across IDA countries in critical areas.5 Three of the five themes—gender and development, climate change, and FCV—were retained from IDA16 and IDA17 in view of the persistent challenges that they pose. The new special theme on jobs and economic transformation in IDA18 replaces (but overlaps with) IDA17’s special theme on inclusive growth.6 The special theme on governance and institutions is new. Appendix A elaborates on areas of focus under each of the special themes. The IDA18 midterm review Implementation and Results Progress Report reported robust progress on the policy commitments under IDA18 (World Bank 2018c).7 Key highlights under the special themes included better support for FCV countries, stronger integration of climate change in all country planning products (Systematic Country Diagnostics [SCDs] and Country Partnership Frameworks [CPFs]) and operations, and strengthened support for legal reforms that enhance gender equality.8 Nevertheless, the midterm review report also pointed to constraints relating to Bank Group 3IDA first introduced special themes in 2005 under IDA14 to “reinforce the effectiveness of IDA assistance in ensuring lasting reduction in poverty and making progress toward the [Millennium Development Goals]” (World Bank 2005). 4Fragility, conflict, and violence (FCV) refers to the challenge of these conditions regardless of whether a country is classified as being in a fragile and conflict-affected situation (FCS). Some of the findings referred to in this report relate to FCS. For ease of presentation, however, the report will consistently use the term FCV. 5At a time of limited resources, results are at the core of IDA’s business model and are an area of continued management attention to ensure that the results culture is mainstreamed throughout IDA’s work. Participants asked to further improve monitoring and evaluation (M&E) to ensure IDA is doing all it can to deliver for its clients (World Bank 2017h, section D). 6Addressing country-specific impediments to productive jobs was identified as one of four important channels for inclusive growth under IDA17. 7 The details of implementation related to the IDA18 special themes, its Windows for dedicated financing, and Financial Framework are provided in accompanying progress reports. 8The midterm review reports that more than three-quarters of 46 policy commitments under the special themes are solidly on track for delivery, with some already delivered. 2 Chapter 1 Introduction implementation and client capacity and it underscored the need to enhance the quality of M&E in IDA-financed operations and in tracking results. Table 1.1. Special Themes under IDA16, IDA17, and IDA18 IDA16a (FY12–14) IDA17a (FY15–17) IDA18a (FY18–20) Inclusive growth Jobs and economic transformation Gender mainstreaming and Gender equality Gender and development gender-related Millennium Development Goals Climate-resilient development climate change Climate change FCS FCS FCV Governance and institutions Sources: World Bank 2011, 2014c, 2017h. Note: Crisis response was one of the four special themes under IDA16. The Crisis Response Window was established under IDA16 to augment financing for IDA countries as they address the effects of severe economic crises and natural disasters. FCS = fragile and conflict-affected situations; FCV = fragility, conflict, and violence; IDA = International Development Association. Purpose, Scope, and Limitations of the Report The objective of this synthesis report is to draw on findings and lessons from recent IEG evaluations (that is, those completed since FY16) and databases that are pertinent to IDA18 special themes and IDA support more generally to inform forthcoming IDA19 discussions. The report seeks to answer the following questions: • What have been the recent trends in development effectiveness for IDA operations and country programs? • To what extent have Bank Group country programs and individual operations addressed the IDA18 special themes and has this changed over time? • What lessons emerge to help improve the development effectiveness of IDA country programs and individual operations in the areas of IDA18 special themes? The synthesis report focuses on learning from IDA experience over the last 10 years in relation to areas covered by the IDA18 special themes, drawing on relevant IEG evaluations completed since FY16.9 Because each of these evaluations looks at operations 9The period covered by individual evaluations varies. For example, the selected Country Program Evaluations cover fiscal year (FY)05–18, the selected Completion and Learning Report 3 Chapter 1 Introduction and activities approved over a five to 10-year period leading up to it, their findings are derived mostly from experience before IDA18, and as such may not remain relevant in the IDA18 period. Moreover, many of these evaluations did not focus solely on the experience of IDA countries. Nevertheless, many of the findings have relevance to the IDA18 special themes and can inform IDA decision-making. The IEG evaluations reviewed for this report include country-focused, thematic, corporate, and project evaluations, as well as learning products and activities (appendix H).10 To the extent that many IEG evaluations were not conducted with an explicit focus on IDA, the relevance of evaluative evidence will be uneven across the special themes. The report does not assess progress toward IDA’s replenishment commitments or present any associated recommendations. The main audience for this report is members of the World Bank Executive Board (including the Committee on Development Effectiveness), IDA Deputies, and Bank Group management. Additional audiences include client country stakeholders, individual program task teams, and other development partners. The report comprises four chapters. Chapter 2 reviews quality and development effectiveness of IDA country operations and programs using IEG’s Results and Performance of the World Bank Group (RAP) database, IEG Completion and Learning Report Reviews (CLRRs) of country strategies, and Country Program Evaluations (CPEs) in IDA and Blend countries. Chapter 3 summarizes the key findings and lessons from thematic evaluations and learning products. It also assesses the extent to which Bank Group country programs and individual operations have addressed IDA18 special themes based on findings from IEG evaluations and additional analysis. Chapter 4 highlights findings and lessons relating to monitoring and evaluation (M&E), including the results orientation of IDA operations and the adequacy of country data, that cut across the special themes and IDA support more generally. Reviews cover FY07–18, and the selected major evaluations and learning products cover a period of approximately 10 years. The report drew on the Management Action Record as needed (for example, in the section on 10 FCV). 4 2. Development Effectiveness of IDA Operations and Country Programs This chapter draws on IEG’s RAP database and CLRR database to analyze recent trends in the development effectiveness of IDA operations and country programs.1, 2 To the extent that evidence is available, it also analyzes how effectively IDA support at the country level has addressed objectives related to the IDA18 special themes. Finally, based on recent CPEs, it presents lessons that may be relevant to future IDA support. As indicated above, to the extent that data relate to the performance of projects that were approved/financed under previous IDA cycles, conclusions do not reflect performance under IDA18. Development Effectiveness of IDA Operations Outcome ratings of recently closed IDA projects show an improvement over the previous period. The share of IDA-funded operations for which the overall outcome was rated moderately satisfactory or better (MS+) rose from 68 percent of projects closed in FY12–14 to 76 percent of projects closed in FY15–17,3 exceeding the corporate target of 75 percent.4 Weighted by the size of projects (that is, net commitment volume), the share rated MS+ increased from 75 percent to 83 percent in the respective periods, exceeding the corporate target of 75 percent (figure 2.1; appendix B, table B.1). In FCV-affected IDA countries, project outcome ratings continue to lag non-FCV countries. MS+ outcome ratings in IDA FCV countries declined from 69 percent of the number of projects exiting in FY12–14 to 65 percent of FY15–17 exits (appendix B, table B.1). Weighted by the value of projects, the share rated MS+ declined marginally from 76 percent to 75 percent in the respective periods. The share of MS+ outcome 1Database for the Results and Performance of the World Bank Group (RAP) 2018, whose data cutoff point is October 3, 2018. 2The development effectiveness of operations solely focuses on World Bank, not on the International Finance Corporation (IFC) or the Multilateral Investment Guarantee Agency (MIGA), which only financed by IDA in a few cases. 3More precisely, the operations are those for which completion reporting was submitted to IEG in the fiscal year in question. 4The World Bank Group’s FY17 Corporate Scorecard sets a target of 75 percent of the number of projects (IDA and IBRD) with outcome ratings of moderately satisfactory or higher (MS+) and 80 percent of projects having outcome ratings of MS+ when weighted by project volume. 5 Chapter 2 Development Effectiveness of IDA Operations and Country Programs ratings for projects in IDA FCV countries was substantially lower than for IDA projects in non-FCV countries exiting in FY15–17. This downward shift can mostly be attributed to projects in Afghanistan and, to a lesser extent, the Republic of Yemen. The decline likely reflects the challenges of operating in an increasingly volatile and conflict-ridden environment. Projects in FCV countries also face challenges, which stem in part from more limited administrative resources and the availability of local staff, smaller project size,5 and more difficult country conditions (for example, on security and political divisions).6, 7 IDA development policy financing (DPF) operation outcome ratings have shown marked improvements, catching up with investment project financing operation outcome ratings. The share of IDA DPF operations with MS+ outcome ratings rose by 15 percentage points (to 75 percent) by number of projects and by 24 percentage points (to 83 percent) when weighted by commitment volumes between FY12–14 and FY15–17, respectively (appendix B, table B.1). Outcome ratings for IDA investment project financing operations saw an increase, with the share of MS+ outcome ratings rising from 71 percent in FY12–14 to 76 percent in FY15–17 by number of projects and 79 to 83 percent by commitment volumes.8 The improvement of IDA DPF operation was driven by changes in the middle of the ratings scale, with a substantial increase in moderately satisfactory ratings and a substantial decline in moderately unsatisfactory ratings (appendix B, table B.3). This also reflects the improving outcome ratings trend for operations under the Macroeconomics, Trade, and Investment GP, which had a relatively high share of DPF (more than 90 percent by both number of projects and commitment volume). 5Independent Evaluation Group (IEG) analysis of RAP data shows that smaller projects tend to have lower ratings than larger projects, consistent with patterns reported in previous RAPs. The analysis suggests country characteristics (for example, IDA versus the International Bank for Reconstruction and Development or FCV) are underlying factors. 6IEG RAP evaluation team conducted structured interviews and workshops with multiple stakeholders to identify factors affecting quality at entry. 7The result was driven by a few countries (including Afghanistan, the Republic of Yemen, Togo, and Madagascar). However, based on a decomposition analysis, Afghanistan and the Republic of Yemen contributed most to the shift. 8 It should be noted, however, that ratings between the two instruments are not comparable — investment project financing and development policy financing (DPF) operations differ in structure and function. In addition, the project rating methodology differs —investment project financing is assessed for efficiency, while DPF is not. 6 Chapter 2 Development Effectiveness of IDA Operations and Country Programs Figure 2.1. World Bank Project Outcome Ratings, by Instruments and FCV Status a. Outcome ratings, share of projects b. Outcome ratings, weighted by volume FY12-14 FY15-17 FY12-14 FY15-17 Corporate Target Corporate Target (75%) (75%) Project Type Project Type IDA IDA IBRD IBRD Other* Other* IPF-IDA IPF-IDA IPF-IBRD IPF-IBRD DPF-IDA DPF-IDA DPF-IBRD DPF-IBRD Country Type Country Type All FCV All FCV All Non-FCV All Non-FCV IDA FCV IDA FCV IDA non-FCV IDA non-FCV 0 50 100 0 50 100 Percent Rated MS+ Percent Rated MS+ Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year; MS+ = moderately satisfactory or better. *Other includes agreement types such as Carbon Initiative, Global Environment Fund, Montreal Protocol, Recipient Executed Trust Funds, Special Funds, and not identified. IDA FCV and IDA non-FCV refer to project ratings in IDA FCV/non- FCV countries. Relevance of design and systematic analytical work—two elements of quality at entry– are key determinants of success in IDA DPF operations. IEG’s evaluation on DPF in IDA countries shows that improving relevance of design is key to achieving better development outcomes (World Bank 2018g).9 This requires congruence between policies supported (that is, prior actions) and project development objectives. The evaluation also found that strong analytical underpinnings are critical to successful DPF outcomes. In addition, operations that include actions to sustain the adequacy of the macroeconomic framework during implementation tend to be more successful. 9The evaluation followed a multipronged methodology to investigate the causal relationship between DPF inputs and the achievement of their expected development outcomes on projects closed over FY09–17. It includes empirical pattern identification (structured, protocol-based portfolio analysis) to identify trends of DPF in IDA countries and empirical analysis to identify empirical patterns in terms of correlates of DPF success, focusing on intervention and contextual factors whose importance was identified through literature review. 7 Chapter 2 Development Effectiveness of IDA Operations and Country Programs IDA project outcomes and trends vary significantly across Global Practices (GPs), in part reflecting differences in instrument composition. The Macroeconomics, Trade, and Investment GP saw a significant increase in the share of MS+ outcome ratings—from 59 to 75 percent by number of projects and from 59 to 81 percent weighted by commitment volumes—between FY12–14 and FY15–17. The Environment and Natural Resources GP also saw a notable increase in MS+ outcome ratings, from 54 to 77 percent by number of projects but a decline from 71 to 63 percent by commitment volumes. The poorest- performing GP on IDA project outcome ratings was Governance, which in FY12–14 recorded an MS+ share of 52 percent by number of projects. By FY15–17, the share had increased to 58 percent. Weighted by commitment volumes, however, MS+ outcome ratings for this GP stood at 74 percent in FY12–14, increasing slightly to 76 percent by FY15–17. MS+ outcome ratings in the Energy GP saw a significant decline—of 7 percentage points by number of projects and 15 percentage points by commitment volumes (appendix B, table B.5). Development Effectiveness of IDA Country Programs The section presents findings on trends in the development effectiveness and quality of IDA country programs. It also examines the extent to which country programs evaluated by IEG reflected the five IDA18 special themes and the extent to which the associated objectives have been achieved. The findings are based on IEG’s CLRRs of CPFs.10 IEG reviewed 32 IDA country Completion and Learning Reviews completed during FY16–18. Details of the methodology are provided in appendix C.11 Positive project outcome ratings do not necessarily translate into positive country program outcomes. IDA country program outcomes have been well below the corporate target. The share of IDA country programs with development outcome ratings of moderately satisfactory or higher (MS+) was 41 percent in FY16–18, well below the corporate target of 70 percent (for IDA and International Bank for Reconstruction and Development; table 2.1). Additionally, about half the IDA CLRRs rated program outcomes as moderately unsatisfactory or worse. This contrasts with the project-level ratings. A disconnect between project and country strategy ratings can reflect incomplete alignment of project portfolio objectives and the priorities identified in the associated CPF. 10Before the Bank Group’s introduction of a new country engagement model in 2014, Country Partnership Frameworks (CPFs) were known as Country Partnership Strategies or Country Assistance Strategies. 11 Based on country classification in the Completion and Learning Report Review year. 8 Chapter 2 Development Effectiveness of IDA Operations and Country Programs Table 2.1. Distribution of IEG CLR Outcome Ratings, FY16–18 IDA World Bank Total IEG Outcome Rating (no.) (%) (no.) (%) Highly satisfactory 0 0 0 0 Satisfactory 1 3 4 7 Moderately satisfactory 12 38 27 48 Moderately unsatisfactory 10 31 12 21 Unsatisfactory 5 16 9 16 Highly unsatisfactory 1 3 1 2 Not rateda 3 9 3 5 Total 32 100 56 100 Source: Independent Evaluation Group. Note: CLR = Completion and Learning Review; IDA = International Development Association; IEG = Independent Evaluation Group. a. In 2016, IEG and Bank Group jointly agreed that, in a few exceptional cases, no ratings are provided for CLRs that have data challenges caused by considerable amount of time elapsed since the formal completion of the Country Assistance Strategy cycle or by other factors. IDA18 special themes have been substantially incorporated into the objectives of country strategies even before IDA18, although the extent varies across countries and themes. All CLRRs reviewed included at least one objective related to the five IDA18 themes. Among the special themes, jobs and economic transformation, climate change, and governance and institutions had the largest uptake (97, 88, and 72 percent of country objectives, respectively) in IDA country programs (table 2.2). Twenty-three of the 32 country programs (72 percent) had objectives relating to governance and institutions. Out of 360 country program objectives in the CLRRs reviewed, 206 could be mapped to at least one of the five IDA18 special themes (appendix C). More than half (53 percent) of the 206 objectives that could be mapped to a special theme were mapped to the jobs and economic transformation theme,12 followed by governance and institutions (21 percent) and climate change (21 percent; appendix C, table C. 2). 12The substance of objectives mapped to this special theme includes building quality infrastructure, raising productivity within the agricultural sector, facilitating connectivity to market opportunities and building the capabilities of firms and people to take advantage of these opportunities, and enhancing private sector competitiveness and integration with global markets. 9 Chapter 2 Development Effectiveness of IDA Operations and Country Programs Table 2.2. Extent of IDA Special Theme Coverage in IDA Country Program, FY16–18 Country Programsa IDA18 Special Theme (no.) (percent) Jobs and economic transformation 31 97 Climate change 28 88 Governance and institutions 23 72 Gender and development 5 16 Fragility, conflict and violence 2 (one of 17 FCV countries and one 6 (with 6% of countries non-FCV country) designated FCV) Source: Independent Evaluation Group. Note: IDA = International Development Association. a. There are 32 total programs. Efficacy in achieving objectives related to the special themes has varied.13 The Bank Group achieved substantial results under gender and jobs and economic transformation, with about more than half the objectives receiving a rating of mostly achieved or higher (table 2.3). As for climate change-related objectives, 44 percent (29 objectives) were rated as mostly achieved or higher (see box 2.1 on Bolivia). Similar to project outcomes, the efficacy of Bank Group–supported programs in achieving objectives related to governance and institutions was more limited, with a quarter (24 percent) of the related objectives rated as mostly achieved or higher. Table 2.3. IEG Efficacy Ratings by Objectives under IDA Special Themes (percent) J&E Governance Climate Gender FCV IEG Efficacy Ratings (N = 110) (N = 44) (N = 44) (N = 5) (N = 3) Achieved 23 8 21 60 0 Mostly achieved 28 16 23 0 33 Partially achieved 36 42 31 20 33 Not achieved 13 34 26 20 33 Total 100 100 100 100 100 Source: Independent Evaluation Group. Note: IEG = Independent Evaluation Group; FCV = fragility, conflict, and violence; IDA = International Development Association; J&E = jobs and economic transformation. 13Efficacy (or effectiveness) refers to the degree to which an objective is achieved; typically, this is captured by the extent to which associated outcome targets in CPFs are realized. However, CPFs often target higher-level outcomes (for example, region-wide or country-wide), or at least outcomes that are influenced by more than Bank Group interventions (World Bank 2018k). 10 Chapter 2 Development Effectiveness of IDA Operations and Country Programs Box 2.1. Achieving a Climate Change-Related Objective: The Case of Bolivia Under the FY12–15 Country Partnership Strategy, World Bank Group support to Bolivia (IDA graduate in FY17) achieved its climate change-related objective of streamlining disaster risk management across sector institutions and levels of government. The Bank Group supported the reconstruction of infrastructure destroyed by natural disasters, while assisting with the establishment of Disaster Risk Management Units in nine sectors by training their staff, preparing recommendations for a more comprehensive National Risk Management System, and providing a policy loan that helped strengthen Bolivia’s legal and institutional framework for comprehensive management of disaster and climate risks. Source: World Bank 2015. Enhancing IDA’s Country Engagement: Lessons from Recent Country Program Evaluations Lessons from the Small States Cluster Country Program Evaluations (FY05–15) Findings from recent IEG CPEs can help inform IDA support under the IDA18 special themes. Two recent CPEs covered IDA countries—the Small States Cluster Country Program Evaluation and the Rwanda CPE (World Bank 2016a, 2016d, 2018j).14 In addition to its special themes, IDA18 specifically sought to enhance support to small states (defined as those with 1.5 million or fewer inhabitants). The cluster Country Program Evaluation focused on Bank Group programs in small Caribbean and Pacific Island countries, many of which are IDA-eligible by virtue of the IDA provision establishing favorable lending terms for small island economies, which IDA18 has since extended to all small states. More significant enhancements in resilience to climate and disaster risks require greater attention to policy and incentive changes. Climate change represents a particular threat to small Caribbean and Pacific Island states as rising temperatures increase the frequency of extreme weather events and rising sea levels mean more serious effects on populated coastal areas. Bank Group programs in affected countries have contributed significantly to building disaster and climate change resilience by helping to improve preparedness capacity, making infrastructure disaster more resilient, and increasing the focus on resilience in government planning. The World Bank played a pivotal role in 14IDA countries encompass both IDA-only and blend countries, which include many small island states. 11 Chapter 2 Development Effectiveness of IDA Operations and Country Programs establishing the Caribbean Catastrophe Risk Insurance Facility as well as a similar pilot facility in the Pacific. These provide funding for emergency response to major disasters, thereby containing their fiscal fallout. Nevertheless, direct Bank Group support to build resilience in small states (notably with respect to infrastructure), while helpful, has been costly and has only reduced vulnerability to a limited extent.15 Real reductions in vulnerability require efforts on a vastly increased scale through policy and regulatory interventions to generate meaningful changes in public and private incentives and behavior. Greater focus on the long-term risks of climate change and facilitating access to climate financing is also needed, especially in countries with limited capacity. Enhanced tools under IDA18 help fill unmet needs for Bank Group support, especially for disaster and climate change risk mitigation. The cluster Country Program Evaluation raised the possibility of linking IDA allocations to vulnerability to better respond to the risks small states face. Although the IDA18 allocation formula does not differentiate based on specific measures of vulnerability, the virtual quadrupling of the base allocation has, and other adjustments, 16 paved the way for a ramped-up Bank Group role in helping to mitigate disaster and climate change risks (and engage on the IDA18 special themes more generally). This represents an important recognition of small states’ intrinsic vulnerabilities. IDA18’s enhancement of the IDA Crisis Response Window also boosts ex post response capacity to disasters (often helping to enhance resilience in the process through the “build back better” feature of projects receiving Crisis Response Window financing),17 as does the provision to adjust IDA financing terms for countries suffering natural disaster losses exceeding one-third of gross domestic product (if warranted based on a debt sustainability analysis). Greater focus on specific leading sectors and on employment opportunities outside the countries can help advance the jobs and economic transformation agenda. Bank Group programs in small Caribbean and Pacific Island countries sought to enhance the policy 15This finding in the Small States Cluster Country Program Evaluation was based on an in-depth assessment of Bank Group support to the Organisation of Eastern Caribbean States countries and the Pacific Island countries, with more cursory and less comprehensive assessments of Bank Group support to Mauritius, the Seychelles, Djibouti, and Cabo Verde. 16IDA18 has put in place a policy framework to provide substantial concessional resources to small states. It has also made the catastrophe deferred drawdown option, which offers immediate liquidity to countries after a catastrophe, available to IDA countries (this was previously not the case). 17Enhancements to the Crisis Response Window include increasing the funding allocated to it and aligning the governance arrangements for responding to economic shocks with the two-step process that is already in place for natural disasters and health emergencies. 12 Chapter 2 Development Effectiveness of IDA Operations and Country Programs and regulatory framework for market-led growth (notably facilitating trade and improving availability of skilled labor), strengthening the financial sector, expanding infrastructure and improving its management, and supporting leading sectors (notably tourism, fisheries, and agriculture). However, while engagement based on the Doing Business framework has led to useful reforms, better addressing the most binding constraints to growth and job creation will also require diagnosing and addressing specific constraints in dominant sectors, as several programs are already doing. Regarding job opportunities outside of these countries, World Bank Advisory Services and Analytics in the Pacific Island countries highlighted the essential role of overseas employment and remittances in economic viability. This body of Advisory Services and Analytics, and crucially World Bank policy dialogue with Australia and New Zealand, helped influence their decisions to expand employment opportunities for temporary migrants from the Pacific Island countries. An impact evaluation of New Zealand’s Recognized Seasonal Employer program found significant positive effects on participant worker households in Tonga and Vanuatu, including per capita income increases of more than 30 percent (World Bank 2014a). The Small States Cluster Country Program Evaluation recommends that similar initiatives in other small states and IDA countries may warrant consideration by the Bank Group. Lessons from the Rwanda CPE (FY09–17) The Rwanda CPE pointed to issues for Bank Group program attention in a country that has decisively emerged from FCV status and seeks to sustain high, inclusive growth. Three broad lessons relating to four of the five IDA special themes emerge from the CPE. First, Rwanda vividly illustrates that, alongside other partners, the Bank Group can effectively support a successful path out of FCV status when the country has strong, stable leadership committed to development under clear and detailed plans and programs that it owns. Second, while the Bank Group has provided crucial support for agricultural modernization, energy development, and other areas to advance Rwanda’s ambition to transform the economy from one based on subsistence agriculture to a knowledge-based regional service hub, given the country’s high inequality, careful management is needed to ensure that agricultural intensification does not exacerbate landlessness, and thereby poverty and inequality. And third, systemic, cross-cutting solutions to governance and institutions and gender challenges require comprehensive integration of sector-specific Bank Group experiences in Rwanda—for example, ensuring that under public financial management, public procurement reform is broad enough to encompass power purchase agreements. While gender-related objectives were pursued in certain areas, notably agriculture and social protection, the CPE highlighted the importance of mainstreaming gender perspectives both at the overall strategy level as well as in specific areas of support. 13 3. Evaluation Findings Relevant to the IDA18 Special Themes This chapter summarizes findings and lessons relevant to the IDA18 special themes based on longer-term IDA experience.1 These are drawn mainly from IEG thematic evaluations and learning products as well as recent RAP reports. The chapter also highlights findings and lessons from a recent IEG corporate evaluation of the Bank Group’s new country engagement model (World Bank 2017m),2 which includes the SCD and the CPF. Given the interconnectedness among the special themes, findings and lessons often relate to more than one of them, and there is an element of discretion in precisely where (under which theme) they appear. Finally, some IEG evaluations classified countries by income group rather than by lending terms. In these cases, it is assumed that findings and lessons for low-income countries (LICs) apply to IDA countries, minor caveats in equivalence notwithstanding.3 Jobs and Economic Transformation Competitiveness and Jobs Policy-based lending can help promote competitiveness and jobs through, for example, support for improvements to the regulatory environment in agriculture (where Bank 1In general, three criteria were used to assess whether findings and lessons from evaluations were relevant for inclusion in this synthesis report: (i) the finding or lesson applied directly to IDA operations and programs; (ii) the finding or lesson applied to operations and programs in low-income countries (LICs); and (iii) the finding or lessons was extracted from an evaluation where the concentration of country case studies was in IDA countries. 2 Under the world bank group strategy released in 2013, country programs will focus on accelerating progress toward sustainably reducing poverty and building shared prosperity (the “twin goals”). To this end, the World Bank Group established a new model for country engagement consisting of two instruments. The SCD uses data and analytic methods to help country clients and World Bank Group teams identify the most critical constraints to, and opportunities for, reducing poverty and building shared prosperity. The CPF determines focus areas for World Bank Group support that align with the country’s development agenda, address the key constraints and opportunities identified in the SCD, and reflect the comparative advantages of the World Bank Group (World Bank 2017m). SCDs are Bank Group products, not joint products with client governments. 3Based on the country classification for FY19, all LICs (except one, which was not rated) are IDA countries (including one blend country). 14 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes Group support to boost competitiveness has been prominent). The IEG evaluation on competitiveness and jobs (World Bank 2016b) assessed Bank Group activities embodying explicit objectives supporting competitiveness in four sectors (“industries”)—manufacturing (including agribusiness), information and communication technology, tourism, and agriculture.4 World Bank lending with a focus on industry-specific competitiveness was concentrated in the agricultural sector (60 percent), mainly in LICs and the Africa Region.5 One specific line of World Bank lending, which sought to help improve the sector regulatory environment,6 had a relatively high success rate of 70 percent MS+ outcome ratings when delivered through DPF (well above the success rate for investment project financing).7 Of these DPF operations, those that focused on agriculture were concentrated in LICs, and supported agricultural policy reforms, development and implementation of action plans and strategies (for example, on fertilizer), adoption of commodity-specific strategies (for example, for cotton), and adoption of frameworks for agricultural extension, research, and education. In FCV countries, stronger frameworks are needed to help promote inclusive growth and jobs. In many cases, when operating in FCV countries, the Bank Group lacked 4Over 2008–14, there were 881 such activities containing some elements of industry-specific support: 463 World Bank lending operations, 165 IFC investment projects, 190 IFC Advisory Services engagements, and 63 MIGA guarantee projects—a total value of $21.6 billion. 5 By contrast, 70 percent of IFC investment and advisory projects and 80 percent of MIGA guarantee projects supporting competitiveness were in manufacturing, including agribusiness. IFC investments in manufacturing were mainly in economies that are more developed, whereas MIGA guarantees were in low-income countries. According to IFC, a major challenge in the manufacturing sector in LICs is finding strong project sponsors. Potential investments are too small in many developing economies (especially in many small markets). As a result, there is little incentive for IFC investment officers to conclude these deals. 6The evaluation grouped World Bank Group interventions under seven lines: specialized infrastructure, industry-specific institutions, industry-specific innovation, specialized skills, industry-specific regulatory environment, specialized finance, and specialized trade and links. 7The evaluation also tested for a correlation between World Bank support to industry competitiveness and job creation at the aggregate level, comparing two groups of countries: those with limited World Bank support (with no more than two interventions categories) and countries with more extensive World Bank support (with three or more intervention categories). The result (all industries with available data—agriculture, agribusiness, manufacturing, and tourism) show a positive, significant association between the intensity of World Bank industry competitiveness support and job creation (World Bank 2016b). 15 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes realistic frameworks to address inclusive growth and jobs that take account of economic constraints in the country and provide for effective coordination and synergies across Bank Group institutions (World Bank 2014). Focused medium- to long-term strategies backed by clearly prioritized and sequenced programs of support were typically missing,8 as were adequate analyses of the drivers of conflict and fragility in the country and of opportunities and binding constraints for the private sector (World Bank 2014, 2016f, and 2016g). Bank Group interventions aimed at improving productivity and competitiveness in specific sectors need sharper focus on jobs with a multisector perspective. Even though employment is a central aspect of the productivity and competitiveness agenda, only a small proportion of the Bank Group portfolio supporting this agenda explicitly references jobs in objectives, interventions, or indicators (World Bank 2016b).9 Reference to job quality is even less common. Improvements in productivity and competitiveness have the potential to both create and destroy jobs and to improve or worsen working conditions for workers.10 In a context of structural transformation, the relative size of some industries tends to decline with economic development while for others it tends to expand. In applying normative judgments regarding the effects of Bank Group interventions on jobs in particular sectors, taking a multisector perspective is thus essential, notably in IDA countries. The new country engagement model should foster progress in systematically integrating the jobs agenda into country diagnostic work and CPFs. The intensity of analytical work on jobs diagnostics and measurement has increased significantly (World Bank 2017d and 2017m). Many SCDs identified and analyzed opportunities for economic growth (in fact, this was the most consistently explored dimension in SCDs). Of these, the majority 8This said, fragility and uncertainty in FCV countries often make it impossible to plan with a longer-term perspective. In fact, in many FCV countries, the Bank Group finds it necessary to adopt a shorter-term approach through a Country Engagement Note than a full CPF. 9 Most Bank Group operations tend to have more indirect effects on jobs and job creation, making attribution to the interventions challenging, as stated in the Management Response (World Bank 2016b). Work is currently under way to broaden the menu of available jobs and economic transformation-related indicators in addition to having “job creation” as an explicit metric. 10The immediate effects of improved labor productivity on the demand for labor (other things equal) tend to be negative. However, the longer-term effects (both direct and indirect, the latter through the impact of productivity improvements on overall market size and market share) can be positive or negative. 16 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes analyzed the drivers of growth and the impact on poverty—for instance, through job creation and labor income channels. CPFs then took up the agenda. For example, CPFs in Côte d’Ivoire, Uganda, and Haiti, included objectives aimed at helping to develop micro, small, and medium enterprises—a specific area of focus under IDA18 (appendix A)—as a means of reducing extreme poverty, creating jobs, and promoting shared prosperity, inclusion, and growth (World Bank 2017m). Inclusive Business In low-income and higher-risk economies, inclusive business models offer the International Finance Corporation (IFC) a promising vehicle to support private sector solutions that can reach people at the base of the economic pyramid.11, 12 IFC’s inclusive business portfolio figures more prominently in IDA and FCV countries, which tend to involve higher country-, sponsor-, and market-related risks than in other client countries (World Bank 2018d). As a share of commitment volumes for projects with inclusive business components, IDA countries account for 36 percent of the portfolio; for the rest of IFC projects, IDA countries account for 24 percent. FCV countries account for 8 percent of inclusive business commitment volumes, more than double the share for the rest of IFC projects (3 percent). IFC inclusive business projects in agribusiness and forestry—the focus of case studies in that evaluation—are distributed evenly across regions but are concentrated in IDA countries. In terms of outcomes, inclusive business projects perform similarly to the rest of the IFC portfolio in IDA countries (53 percent versus 51 percent, respectively, rated mostly successfully or better). Notably, no trade- off between profitability and inclusion objectives is apparent based on available evidence. The performance of inclusive agribusiness projects is also comparable with 11The special theme on jobs and economic transformation focuses on facilitating connectivity to markets through support for global value chains and building capabilities for increased productivity and earnings through support for facilitating access to finance and building management capacity in micro, small, and medium enterprises (appendix A). 12Inclusive business is defined as a business that provides goods, services, and livelihoods on a commercially viable basis, either at scale or scalable, to people living at the base of the economic pyramid, making them part of the value chain of companies’ core business as suppliers, distributors, retailers, or customers. IFC’s definition of inclusive business excludes base of the economic pyramid employment. 17 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes that of other IFC agribusiness projects, except for their environmental and social ratings.13 IFC can help clients who have already established inclusive businesses to develop value chains and replicate or scale up where conditions are favorable. In inclusive agribusiness projects, for example, IFC financing often helps clients who already use an inclusive business model to sustain or expand operations. Inclusive business activities also enhance opportunities for the poor in the value chains that relate to the business. Drivers of success in these agribusiness projects include: (i) sustainability and scalability of clients’ business model, notably strong results and growth of the business and high returns on invested capital; (ii) favorable movements in global commodity prices; and (iii) in a few cases, tariff protection. In Nigeria,14 one of the key factors that enabled a company to expand inclusive business operations to a new foreign market was support from the parent company to help the company operate in the new market despite a weak macroeconomic situation. Gender and Development Gender in IDA Country Strategies The Bank Group has increasingly integrated gender into IDA country strategies. Discussion of gender issues has increased significantly over time, featuring in virtually all country strategies by 2016 (World Bank 2018e) compared with 64 percent in 2005. A systematic review of gender integration in RAP 2015 found that among the 58 country strategies completed in FY12–14,15 50 (that is, 86 percent) incorporated gender along at least one of three dimensions (diagnostic, actions, and M&E indicators). Twenty-three country strategies (40 percent) explicitly reflected gender in an objective or a pillar, and 18 addressed it in a cross-cutting way (World Bank 2016e).16 All five of the country 13Environmental and social ratings are weaker, mostly owing to challenges involving land issues and waste processing unrelated to the inclusive nature of their business models. The evaluation conducted 16 country case studies including 10 IDA countries: Ghana, Liberia, 14 Nicaragua, Bangladesh, Honduras, Pakistan, Zambia, Mozambique, Nigeria, and Central African Republic. All those that closed during FY12–14, 35 of which were joint World Bank–IFC strategies. Joint 15 World Bank–IFC strategies did not show a higher level of gender integration. 16This review focused on more recent CPF, while the review of Completion and Learning Report Reviews in chapter 2 covered the earlier Country Partnership Strategies or CPF which were already completed. 18 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes strategy documents that included an explicit gender pillar were for IDA countries (IDA countries represented 65 percent of the country strategies reviewed). Under the new country engagement model, in line with corporate commitments, gender perspectives were well integrated into IDA country SCDs. IEG’s early implementation assessment of the SCD and CPF process noted progress on corporate commitments on gender (box 3.1; World Bank 2017m). The assessment found that all nine IDA country SCDs (out of 22 SCDs reviewed) discussed gender issues and five identified key gender gaps to be addressed (World Bank 2017m). These discussions reflected an evolution beyond human development toward economic empowerment, especially in relation to labor force participation, skills development, and jobs. Several SCDs offered good examples of gender integration. The Chad SCD stands out because it discusses in detail several constraints to women’s agricultural productivity as well as the impact of high fertility and early childbearing on women’s economic activity and maternal health. Limited access to land by women and lack of labor time figured among the most binding constraints. However, in some cases (for example, Bangladesh and Mali), when SCDs identified priorities to reduce poverty, they did not explicitly spell out their gender linkages.17 Box 3.1. Corporate Commitments on Gender In December 2015, the World Bank Group approved a new strategy, reinvigorating its commitment to gender, which was already spelled out in World Bank Operational Policy 4. 20 requiring country strategies to be informed by a gender assessment. As part of the 17th replenishment of the International Development Association (IDA), the Bank Group committed to integrating gender considerations into the analyses, program contents, and results frameworks in all IDA Country Partnership Framework and achieved the target in FY17 (World Bank 2017a). The Bank Group’s Corporate Scorecard monitors the integration of gender considerations in all IDA and International Bank for Reconstruction and Development CPFs, with a target of 100 percent satisfactory attention to gender by FY17. Starting in FY16, SCDs were required to incorporate gender in their analytical frameworks. Source: World Bank 2017a. Strong integration of gender issues in the SCD needs to carry over into the CPF. CPFs have tended to align with SCDs in identifying key gender issues, but they have not 17In the case of Bangladesh, the Systematic Country Diagnostic (SCD) states that specific linkages were intended to be determined once a Gender Assessment was completed to better understand core issues of gender inequality and social inclusion. Linkages continue to be identified at the individual project level based on development objectives and the specific circumstances in the field. 19 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes necessarily identified specific strategic priorities and entry points for addressing them (as was done for instance with the school-to-work transition in Bolivia, an FY17 IDA graduate; World Bank 2017m). Quality of gender integration varies across CPFs. In some cases (Haiti and Uganda), CPFs have outlined clear areas of action but have not necessarily identified how they would address key gender gaps. In other cases (Maldives and Mali), CPFs have simply referred to mainstreaming gender in the Bank Group portfolio and analytical work with little discussion of gender in CPF program objectives and areas of focus. Most CPFs have made a generic reference to gender mainstreaming in the portfolio, regardless of whether gender is identified as a cross- cutting issue. Overall, compared with the previous incarnation of country strategy documents (Country Assistance Strategies and CPSs), CPFs have better articulated Bank Group interventions to address specific gender inequalities and have made more comprehensive use of gender-disaggregated indicators. Nevertheless, the alignment between the actions proposed and indicators in the results frameworks was often weak.18 More work is needed to integrate gender objectives in conflict-affected areas. Few programs in FCV countries were designed or implemented taking into consideration gender disparities, and the little analytical work that was done was not translated into operations (World Bank 2016i). With few exceptions, Bank Group country strategies did not focus on gender-based challenges in conflict contexts until the latter half the evaluation period (2010 onward).19 FCV-specific gender-based challenges include that: (i) women (children) are often targeted deliberately in conflicts; and (ii) postconflict reconstruction may require special economic empowerment programs due to the high number of women-headed households. In addition, most World Bank–supported nationwide projects were not gender-responsive in design or implementation in regions of subnational conflict. Community-driven development (CDD) approaches have dominated work on gender issues in conflict-affected areas, where women’s roles in decision-making remain limited. 18Bank Group gender strategy (FY16–23): Gender Equality, Poverty Reduction and Inclusive Growth , finalized after the current guidance for the SCDs and CPFs, provides more specific direction on how gender can be incorporated in SCDs and CPFs. The SCD and CPF guidance will need to be updated to incorporate this direction. 19The World Bank has been increasingly emphasizing gender-based violence issues, particularly in FCS. The recent incidence in World Bank–financed projects may produce positive results. 20 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes Gender in IDA Operations Integration of gender in IDA operations needs continuing attention.20 IEG conducts a systematic assessment of gender dimensions in individual World Bank projects as part of the Implementation Completion and Results Report (ICR) review process (appendix D). The latest IEG analysis, on projects that closed during FY14–18, found that gender was an explicit part of a project development objective (PDO) or at least one project component in 64 out of 202 IDA projects, or 32 percent (appendix D, table D.1). In addition, 73 percent of these 64 IDA project ICRs presented sex-disaggregated or male- or female-specific indicators, with a further 13 percent of ICRs reporting on the share of female beneficiaries.21 Fourteen percent did not have any gender-specific reporting (table D. 2). The share of IDA projects incorporating gender in a PDO or at least one project component has remained stable over time. However, there has been a shift in composition between the two aspects of gender integration, with the share of projects explicitly embodying gender in a PDO increasing from 7 percent in FY15 to 13 percent in FY17 (figure D.1).22 Gender integration in IDA operations originating in some GPs needs particular attention. Only half the GPs had exiting IDA projects that explicitly integrated gender in a PDO, with the Health, Nutrition, and Population GP posting the highest share (38 percent). Exiting IDA projects where gender was an explicit part of at least one project component appeared more across GPs, albeit with variations in their share. Such projects were found in 13 out of 15 GPs, with the Health, Nutrition, and Population GP again posting the largest share, followed by Agriculture (table C.3). In the Environment and Natural Resources GP, the Energy and Extractives GP, and the Transport and Digital Development GP, no exiting projects explicitly integrated gender into a PDO, and the share of exiting projects with gender as an explicit part of at least one project component was lower than in other GPs. 20IEG assessed IFC’s approach to gender integration in RAP 2015 (World Bank 2016e). While the report does not provide information specific to IDA countri es, it indicates that IFC’s approach to gender has become more focused, prioritizing three of the five private sector dimensions — entrepreneurship, employment, and corporate leadership—and has introduced some specific initiatives. 21The “share of female beneficiaries” indicator has some serious pitfalls, as highlighted in IEG evaluation, as it often simply refers to project beneficiaries or residents in the project area (World Bank 2016e). 22 The share of project with at least one component that incorporates gender has slightly declined (figure D.1). 21 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes CDD operations and those relating to rural nonfarm economy development have been relatively effective in addressing gender issues. An IEG learning product on Gender in CDD found that CDD programs tended to increase female participation in village committees (World Bank 2017l). CDD interventions have the potential to address some of the constraints to women’s economic, political, and social empowerment that exist in the rural areas where such interventions operate. Their activities can have both direct impacts (especially on economic empowerment) and indirect impacts (mostly on political and social empowerment). However, the learning product also points out that the impacts of increased female participation are less clear because most CDD programs do not measure the quality and results of participation. Progress on gender integration was also observed in Bank Group support for rural nonfarm economy. Sex- disaggregated data across the rural nonfarm economy portfolio increasingly record women’s participation in Bank Group-financed projects, although few projects record women’s access to economic opportunities, and none record the distributional benefits by gender (World Bank 2017d). Other sectors need to be better gender-informed, in line with the example of higher education. Urban transport operations paid little attention to the special needs of women and disabled persons among disadvantaged groups (the poor, women, disabled persons, and the elderly), although the poor received more attention (World Bank 2017f). In response to this evaluation finding, the Bank Group committed that at least two-thirds of all urban passenger transport financing operations delivered under IDA18 would specifically cater to the different mobility and personal security needs of women and men.23 In its support for higher education, the World Bank has explicitly targeted vulnerable and disadvantaged groups—defined mainly on the basis of income, gender, and ethnicity—in about one-third of its investment projects. Targeting women has been more prominent in LICs and lower-middle-income countries, where IDA operations are concentrated (World Bank 2017e). 23IDA18 midterm review reported that implementation of this measure is on track (World Bank 2018c). 22 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes Climate Change Climate Change in the World Bank Group Country Strategies and Project Portfolio There has been a moderate increase in attention to climate change in Bank Group country strategies in IDA countries. Over the period of FY08–17, there was a modest increase in the share of country strategies containing an explicit objective with potential climate change benefits for both mitigation and adaptation (figure 3.1).24 Greater attention was paid to climate change adaptation than to mitigation. As compared with International Bank for Reconstruction and Development countries, IDA countries more often included objectives supporting access to energy, through renewable and clean technologies. These figures should be seen against the backdrop of the corporate mandate established under the 2016 Bank Group climate change action plan, which requires all CPFs to “take climate change into account.”25 Figure 3.1. Share of IDA Country Strategies with Climate Change Objectives or Outcomes 50% 43% 38% 40% 30% 30% 27% 20% 10% 0% FY08-12 FY13-17 FY08-12 FY13-17 CC Mitigation CC Adaptation/DRM Source: Independent Evaluation Group analysis of all 101 country strategies for IDA-eligible countries approved FY08–17. Note: CC = climate change; IDA = International Development Association. Corporate prioritization of climate change has been successful in increasing IDA operations supporting both mitigation and adaptation. IEG analyzed the potential climate change benefits from the project portfolio in IDA countries between FY08–10 and IEG carried out an assessment of all country strategies approved between FY08–17 using its 24 RAP 2017 database (World Bank 2018i). 25The mandate does not require that CPFs include objectives or outcome targets related to climate change. Figure 3.1 covers the period before the corporate action plan was drawn up. 23 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes FY15–17 using a database created for IEG’s RAP 2017 (see appendix E for methodology).26 The data show roughly a 50 percent increase in the share of project components with potential climate change benefits for both climate change mitigation and adaptation between the two periods (figure 3. 2). There has been a large increase in the share of financing commitment volumes in IDA countries with potential climate change cobenefits, from 17 to 28 percent, driven by climate change adaptation.27 This reflects increases in financing for disaster risk reduction works, climate-resilient infrastructure, and climate change-related budget support, while support for disaster risk management policy and institutional frameworks have been consistently present. Support for renewable energy has increased, as has support for sector-specific mitigation issues (for example, in transport, agriculture, forests, or urban), while there has been a substantial decline in support for energy efficiency. Figure 3.2. Share of Climate Change Adaptation and Mitigation Activities in the World Bank’s Project Portfolio Share of Components Share of Commitments 25 25 21 20 20 18 17 17 15 15 15 14 15 15 Percent Percent 12 9 9 8 10 10 5 6 5 5 5 3 0 0 IDA Non-IDA IDA Non-IDA IDA Non-IDA IDA Non-IDA CC Adaptation CC Mitigation CC Adaptation CC Mitigation 2008-2010 2015-2017 Source: Independent Evaluation Group analysis of a sample of one-third of newly approved project financing operations, FY08–10 and FY15–17. Notes: IDA includes both IDA and IDA-International Bank for Reconstruction and Development blend countries. Components can include both mitigation and adaptation benefits but only a small number provided both. CC = climate change; IDA = International Development Association. Large increases in support for climate change adaptation and mitigation had been accompanied by declining support for other environmental challenges, especially in LICs. IEG’s RAP 2017 (World Bank 2018b) found that, while the Bank Group had 26The main analysis reported here focuses on environmental effects of projects, not other World Bank support such as Advisory Services and Analytics or carbon finance. However, analysis on those product lines produced similar conclusions to the finding regarding projects. 27 Commitment amounts are calculated by project component. 24 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes increased project support for environmental sustainability by 4 percentage points over a decade, it had decreased support for other important environmental challenges, including biodiversity conservation and water resource management. Moreover, World Bank environmental support was highest in upper-middle-income countries, followed by lower-middle-income countries, with the lowest share in LICs and a widening gap over time, which suggests that such support is very modest (and increasingly so) in IDA countries.28 Efforts to explicitly support environmental cobenefits in climate change work could help support both goals. Pollution Management and Carbon Finance Intensified focus on climate change under IDA18 should not result in neglect of pollution concerns. The most severe pollution problems are seen in middle-income countries, which are mostly not IDA countries; IDA countries are more likely to encounter higher pollution problems after graduation. Yet certain local pollution problems are major concerns for IDA countries, while most of these countries are not major greenhouse gas emitters. Indoor air pollution causes roughly 49 percent of pollution-related deaths in developing countries, at roughly 2.9 million deaths per year, including in many IDA countries. Yet only 9 percent of Bank Group pollution abatement interventions addressed indoor air pollution. Damage from toxic substances including lead, electric waste, and pesticides may also be severe in LICs, but an absence of data means that these problems might not be noticed or addressed. This leads to four major conclusions relevant for IDA countries (World Bank 2017k). First, climate change mitigation projects should be explicitly designed to also address local air pollution. Second, greater attention should be paid to indoor air pollution. Third, countries approaching middle-income status could benefit by building up their pollution management frameworks in advance, in the hopes of avoiding the most severe pollution costs suffered by many lower-middle-income countries. Fourth, greater efforts are needed to collect data on the severity of pollution from toxic substances. Efforts to extend benefits of carbon market mechanisms to IDA countries need to learn from experience. Though the World Bank was very successful in supporting development of carbon finance markets before their collapse, it was less successful in extending their benefits to LICs (World Bank 2018a). The World Bank was the main provider of technical and financial assistance to bring LICs into the Clean Development 28The data show that the share of project components with potential climate change benefits in LICs increased by 1 percentage point from 25 to 26 percent between FY08–10 and FY15–17, while the share in upper-middle-income countries increased by 8 percentage points, from 42 to 50 percent. 25 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes Mechanism process, and it was more successful than other Clean Development Mechanism participants in supporting LICs. However, emission reduction credit issuances in LICs remained less than 1 percent of the global market. Capacity constraints in client countries and a shortage of quality projects were important factors, as most LIC projects were less competitive and higher-risk, with limited scale, in a knowledge-driven market with high transaction costs. The Clean Development Mechanism also excluded agricultural land use and deforestation/degradation abatement projects, which could have been entry points for many LICs. Carbon markets collapsed in 2012 just as many LICs were beginning to engage in them, especially through Program of Activities approaches which helped to reduce transaction costs for small-scale projects. In any future carbon market mechanisms, IDA and LICs would benefit from (i) a clear prioritization of development cobenefits in any carbon finance projects, to avoid the perception of trade-offs between climate change mitigation and development; and ii) the inclusion of agriculture, land use, and forestry in any market mechanism, which would offer more potential opportunities. Fragility, Conflict, and Violence Earlier evaluative evidence highlighted the need to “do things differently” in FCV contexts and pointed to several shortcomings. IEG found that country strategies lacked tailoring to fragility and conflict drivers, realism, and flexibility to adjust if risks materialize (World Bank 2014). 29 While the World Bank had been relatively successful in mainstreaming gender in its human development portfolio, it has paid insufficient attention to conflict related violence against women and economic empowerment of women in FCV. Regarding modes of engagement, CDD has been a useful vehicle for short-term assistance, but the long-term sustainability of programs remained questionable. The Bank Group has strengthened its capacity and approach to engaging in FCV, but progress in addressing IEG recommendations has been uneven. The Bank Group has strengthened its analytic capacity and knowledge, increased financing for FCV, and adjusted policies and human resources systems. But the degree of implementation of 29The IEG evaluation concluded that fragile and conflict-affected IDA countries receive less per capita assistance than nonfragile IDA countries (World Bank 2014b). This stood in contrast to other development finance institutions which had significantly higher allocations to IDA fragile countries. About one-sixth of IDA support went to fragile and conflicted-affected states compared with about 40 percent on average for development finance institution. 26 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes changes and uptake in operations is less clear. Annual Management Action Record updates indicate insufficient operationalization of fragility metrics to track FCV status of a country and efforts to build state capacity (World Bank 2018h).30 More focused and systematic discussion of fragility in FCV country strategies needs to translate more directly into country program priorities. Earlier IEG evaluations on FCV found that Bank Group strategies/programs in FCV countries were not tailored to FCV drivers (World Bank 2014b and 2016g). A recent IEG review of seven SCD/CPFs in FCV countries (six of which were IDA countries) found that they consistently discussed fragility (World Bank 2018h). It found that the Bank Group has enhanced its analytical capacity to engage in situations of FCV through Risk and Resilience Assessments, which can inform SCDs and CPFs. However, CPFs did not always cover the fragility drivers identified by the Risk and Resilience Assessments. The CPFs often lacked a narrative to show whether and how country program priorities were responsive or tailored to the FCV contexts, and provisions for tracking fragility indicators at both the country and project level. More holistic approaches can help address challenges to public service delivery in FCV- affected countries. The Bank Group often did not have a sufficiently holistic approach to private sector development in FCV countries. IEG’s evaluation of Bank Group support for fragile situations in nonfragile and Conflict States had similar findings (World Bank 2014b and 2016i).31 The principal channel of support for the private sector development agenda in FCV situations was through CDD-type projects. The primary focus of World Bank efforts was on direct support to livelihood and short-term employment schemes, with questionable prospects for longer-term sustainability. Public service delivery initiatives in IDA FCV situations need to integrate capacity building to assure sustainability of results. The IEG evaluation on health services 30The evaluation also argues that results frameworks could include (and report on) fragility and resilience indicators to monitor FCV trends at the country level. State capacity measures are still not addressing fragility drivers except for decentralization which is on the agenda in several FCV countries. 31 Findings and lessons from the IEG evaluation of World Bank Group assistance to Low-Income Fragile and Conflict-Affected States (World Bank 2014b) remain relevant to current FCV situations in IDA countries. The IEG evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence (World Bank 2016g) focused primarily on middle-income countries (it had lesser coverage of IDA countries, although half the country case studies were IDA countries). 27 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes reported that the World Bank has provided substantial support in this area, addressing capacity limitations in FCV countries (World Bank 2018l).32 The World Bank supports health services in about 70 percent of countries in FCV situations. Support to FCV situations represents about 23 percent of projects and 11 percent of commitments approved during FY05–16. Capacity building was provided in nearly all operations (and the subsample of FCV countries; 97 percent and 100 percent, respectively) according to appraisal documents reviewed.33 The evaluation findings also highlight the importance of strengthening country health systems’ pandemic risk management capacity, which has been incorporated as a policy commitment under IDA18 and closely monitored.34 IEG’s evaluation on water supply and sanitation assessed 14 World Bank water supply and sanitation projects in FCV countries, including 10 projects in five IDA countries,35 which were completed during FY07–16 (World Bank 2017c). Positive results can be attributed to World Bank engagement and dialogue with governments and to fielding multidisciplinary teams for project preparation and implementation. However, the results faced varying levels of risk because of limited capacity.36 In Afghanistan’s cohort of five projects that addressed urban and rural water supply and sanitation, limited capacity and turnover of qualified staff were critical issues. The Bank Group may benefit from greater use of alternative consultation models and techniques for more inclusive engagement in preparing IDA FCV country strategies. In engaging stakeholders in the development of country strategies, Bank Group teams are encouraged to use a variety of tools. Nontraditional tools can be of particular value in IDA FCV countries, given the difficult operational environments, including security challenges, often encountered. However, only rarely do they tap into this range of The country case studies include three IDA countries (of which two IDA FCV countries), 32 Bangladesh, Liberia, the Republic of Yemen (out of six country case studies). 33Capacity building typically focused on health sector management, health finance, surveillance, and provision of basic health care and treatment. Capacity building activities for frontline service providers were largely carried out as planned (World Bank 2018l). 34The policy commitment is to improve institutional capacity to respond to pandemics through support at least 25 IDA countries in developing pandemic preparedness plans. The IDA18 midterm review reported that the progress is on track (delivered in 8 countries and ongoing in 15 countries; World Bank 2018c). 35 Afghanistan, Burundi, Haiti, Lebanon, Sierra Leone, and South Sudan. 36The risks the evaluation identified include inadequate tariff reform and cost recovery, which led to a lack of sustainable financing for operations and maintenance (World Bank 2017c). 28 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes options, typically falling back on traditional in-person consultation methods. Only a few of the CPFs reviewed had used alternative models of engagement, such as social media or online consultation, to reach a broader audience (World Bank 2018b).37 For instance, the Bank Group strategically used social media in Myanmar, where it was reengaging after a long hiatus, and in the Republic of Yemen, where it is striving to engage citizens despite the ongoing conflict. In the latter case, the Bank Group team held a series of online consultations with approximately 20,000 respondents drawn from different segments of the Republic of Yemen civil society and envisages using online consultations throughout the Country Engagement Note period to keep a pulse on stakeholders’ views. Yet, there are difficulties in reaching out to citizens in FCV countries where outreach is more apt to create tensions with the government. Consultations regarding Country Engagement Notes are therefore not as open, nor as openly reported on, given security or political complications. Governance and Institutions Governance in SCDs and CPFs The focus on governance in SCDs and CPFs needs to be continued while ensuring that IDA-supported measures address incentives and capacity. IEG’s early implementation assessment of the new country engagement model (World Bank 2017m) found that governance was featured in the SCDs and CPFs for all three IDA countries (Bangladesh, Chad, and Uganda) that featured in the sample of seven countries selected for the assessment.38 However, an in-depth analysis of, and clear approach to, governance was missing in some cases. Bank Group country engagement would have benefited from political economy analysis and more explicit discussion of capacity constraints and plans to build institutional capacity in line with country-specific governance challenges. Tax Revenue Mobilization Past evaluations concluded that little attention was paid in IDA countries to tax policy and administration reform, an important component of domestic resource mobilization. Over FY05–15, Bank Group support for tax policy and administration reform was provided through World Bank DPF and investment project financing operations (205 37In the evaluation of citizen engagement (World Bank 2018b), half the country case studies were IDA. 38 Governance and institutions had a large uptake in IDA country program reviewed. 29 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes projects in 107 countries for a total commitment of $28.4 billion) as well as IFC Advisory Services on business taxation. Bank lending operations that support tax reform were highly concentrated in middle-income countries (78 percent) with much less engagement in LICs (5 percent).39 In addressing tax policy and administration reform, careful consideration needs to be given to factors such as government ownership and capabilities. An IEG learning product found that effective tax revenue mobilization hinges among other things on strong government ownership and calibration of the tax system to the country’s tax administration capabilities (World Bank 2017j). In Pakistan, a lack of government ownership compromised the success of a stand-alone DPF operation.40 In Cambodia, modernization of revenue administration comprised one of two components of a World Bank project; achievement of the associated objective was substantial, in part due to careful design that recognized limited public sector capabilities in a postconflict country. Implementation was based on a series of relatively simple activities such as issuance of administrative circulars and promoting use of the banking sector for tax payments.41 Enhancing Institutions for Infrastructure Development and Service Delivery More effective support for infrastructure development and service delivery hinges on more comprehensively integrating institutional development. In the urban transport sector, weak institutional capacity and coordination remains a critical challenge (World Bank 2017f). Support for institutional development figures in 80 percent of Bank Group projects, yet it is often narrowly focused and not sustained beyond a single operation. Longer-term and more ambitious institutional reform engagement have been confined to only a few cases. In housing finance, IFC Advisory Services have supported capacity enhancement of microfinance institutions to develop innovative housing microfinance mechanisms in FCV countries, such as Afghanistan and Haiti, although market impact 39Bank Group management cautions that the situation has changed fundamentally since the review. The Global Tax Program was initiated in 2017 and is now funded by eight donors who have committed over $55 million to support a variety of domestic resource mobilization activities with a focus on country-level activities. Annual new loan commitments to LICs averaged $26 million during FY11–15. This jumped fivefold to reach an average of $131 million during FY16–19 and $268 million in FY19. 40 Pakistan Poverty Reduction and Economic Support Operation. 41Consistent with the lessons, the World Bank has developed tools for systematic and comprehensive assessment of country-specific domestic resource mobilization challenges, including tools that place political economy and the social contract between taxpayers and the state at the center of domestic resource mobilization engagements. 30 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes remains limited (World Bank 2016h). In water and sanitation, one of the two main sources of risk to development outcomes has been inadequate institutional capacity, especially in rural areas (World Bank 2017c). In supporting development of off-grid electricity services for the poor, implementation has often been retarded by one or more sector readiness conditions relating to governance, including the institutional framework, weak capacity, and poor financial accountability (World Bank 2017g). Citizen Engagement The increased engagement in recent years of citizens in projects, CPF preparation, and policy formulation needs to be continued. Corporate commitment to mainstream citizen engagement, based on the 2014 strategic framework, generated awareness and buy-in among senior management and staff. The corporate commitment helped increase the percentage of projects using citizen engagement mechanisms, especially those that are not safeguard related. Based on a portfolio of 299 randomly selected World Bank investment projects, the share of projects using nonsafeguards-related citizen engagement mechanisms increased from 67 percent to 76 percent between FY11–13 and FY14–16. Country strategies in which governance is highlighted as an important cross- cutting theme tend to promote reforms for greater transparency and accountability and reinforce systems for citizen engagement. Thirty-seven percent of the country strategies (17 CPFs) explicitly identified the objectives of engaging citizens. The evaluation on Engaging Citizens for Better Development Results (World Bank 2018b) reviewed all 46 CPF approved between FY15 and FY17, including those in IDA countries, and found that almost all refer to having consulted with civil society and, increasingly, with the private sector. Nevertheless, the World Bank has not sufficiently monitored key elements identified in the strategic framework as essential to ensure successful integration of citizen engagement in projects such as quality of engagement and closing the feedback loop with citizens.42 Concrete steps can be taken to broaden and deepen citizen engagement in IDA operations. Engaging citizens in IDA operations can have a tangible impact on the quality of services and on development outcomes. The citizen engagement evaluation, which included country case studies (half them in IDA countries) identified key issues to 42Bank Group management has pointed out that there are mechanisms in place for monitoring citizen engagement, including Operations Policy and Country Services guidance on citizen- oriented design and beneficiary feedback indicators, a stringent independent process by which the Citizen Engagement Secretariat, regional and GP focal points, and the data team review projects to ensure that design and indicators meet the definition, and progress recorded in the citizen engagement monitoring database. 31 Chapter 3 Evaluation Findings Relevant to the IDA18 Special Themes be addressed to expand and deepen citizen engagement, including building capacity, strengthening monitoring and reporting, and regular outreach using tools embedded in country systems. For example, in one of the IDA case countries, Kyrgyz Republic, a citizen engagement country road map outlined a set of priority areas and projects where citizen engagement could be strengthened. 32 4. Monitoring, Evaluation, and Results This chapter discusses key determinants of the development effectiveness of IDA support, including quality of M&E, the Bank’s performance in ensuring quality at entry and quality of supervision and data.1 Relevant findings are drawn mainly from IEG corporate evaluations, the RAP 2016 report: Managing for Results, and the latest RAP databases. Monitoring and Evaluation Strengthening IDA’s results orientation for maximum development impact requires better M&E. At a time of limited resources, results are at the core of IDA’s business model and are an area of continued management attention to ensure that the results culture and M&E are mainstreamed throughout IDA’s work. An assessment of the quality of M&E in IDA-financed operations was featured in the Results Measurement System under IDA17 and was retained under IDA18.2 The RAP 2016: Managing for Results (World Bank 2017i) indicated progress under the new country engagement model but cautioned that strengthening the foundations of results measurement and instilling a culture of evidence-based adaptive management and learning needs a stronger, more systematic, and holistic push.3 With respect to private sector interventions, even though IFC and the Multilateral Investment Guarantee Agency also made progress on M&E,4 limited evidence regarding beneficiary outcomes hindered the articulation and aggregation of results across the Bank Group. 1“Quality at entry” and “quality of supervision” ratings are sub-ratings for overall Bank Performance, which is presented in Appendix B (table B.2). 2 Under the Tier III—IDA organizational and operational effectiveness. 3The World Bank Group focused relatively less on helping clients strengthen capacity for their own use of data and evidence for adaptive management and learning. Weak client capacity is a constraint to managing for results. The World Bank Group will need a more strategic approach to address client M4R capacity and the demand for improved M4R, based on systematic diagnostics (World Bank 2017m). 4In 2018, IFC introduced the Anticipated Impact Measurement and Monitoring system to evaluate ex ante all IFC investment projects including in IDA and FCS countries. The approach provides IFC with a tool to set ambitious yet achievable targets, select projects with the greatest potential for development impact and financial sustainability, and then optimize project design. 33 Chapter 4 Monitoring and Evaluation and Results Figure 4.1. IDA Projects: Quality of Monitoring and Evaluation Ratings, FY06–17 M&E Quality, 3-Year Rolling By Number 100 80 60 32 30 31 33 36 40 31 27 25 28 24 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=300) (N=340) (N=349) (N=387) (N=457) (N=488) (N=485) (N=430) (N=367) (N=318) Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: IDA = International Development Association. IEG ratings of M&E quality in IDA projects have improved substantially in recent years. The share of IDA projects (by number of projects) for which M&E Quality was rated substantial or above (3-year rolling average) increased by 6 percentage points to 36 percent between FY12–14 and FY15–17 (figure 4.1). The improvement is more significant when weighted by commitment volumes: M&E quality was rated substantial or above in 41 percent of cases in FY15–17, an increase of 11 percentage points from FY12–14 (appendix F, figure F. 2). Moreover, the share of projects (weighted by commitment volumes) receiving a Negligible rating on M&E quality decreased from 9 percent in FY12–14 to just 4 percent in FY15–17. These favorable trends reflect efforts— in some cases involving IEG—at the corporate level.5 The choice of results indicators was a leading weakness in M&E, albeit some improvement in use of gender-disaggregated indicators was noted. A review of Project Performance Assessment Reports covering 57 projects in 22 countries and eight regional projects—all in IDA countries—found that shortcomings in implementation of M&E and use of M&E data were often due to poor M&E design (appendix G). In several cases, project monitoring indicators were found to have little relevance to the project objectives and outcomes sought. Other issues affecting the design of M&E frameworks included: (i) use of too many indicators; (ii) use of process- and output-focused indicators; and (iii) use of indicators lacking quantitative baselines and targets. Despite the shortcomings, 73 percent of these 64 IDA project ICRs presented sex-disaggregated or male- or female- 5 The IEG RAP evaluation team noted several factors behind these improvements, including more training on results reporting by Operations Policy and Country Services and IEG and outreach by IEG to the Practice Manager for Strategy and Operations teams. 34 Chapter 4 Monitoring and Evaluation and Results specific indicators, with a further 13 percent of ICRs reporting on the share of female beneficiaries. M&E frameworks need improvement to foster better assessment of results and encourage learning in relation to the FCV theme in IDA projects. IEG’s FCV evaluation (World Bank 2016g) found that only a third of programs assessed recorded fragility- specific outcomes, and only a quarter had M&E systems specifically designed to track FCV issues. The evaluation offered recommendations to strengthen M&E frameworks in FCV—for instance, including FCV-related outcome indicators and developing a new publicly disclosed system of markers and flags for monitoring and measuring fragility. Bank’s Performance in Ensuring Quality at Entry and Quality of Supervision There was a modest improvement in quality at entry and quality of supervision ratings (two sub-ratings for Bank’s Performance) for IDA projects in recent years. The share of IDA projects for which the Bank’s performance in ensuring quality at entry was rated moderately satisfactory or higher (MS+) improved modestly from 59 percent in FY12–14 to 61 percent in FY15–17 (appendix F, figure F.3). At the same time, when weighted by commitment volume, there was a deterioration of quality at entry ratings from 68 percent in FY12–14 to 65 percent in FY15–17. Key factors contributing to poor quality at entry include complexity of project design and lack of realism in proposed implementation schedules. The Bank’s performance in quality of supervision ratings exhibited a similar pattern and remained relatively high. The share of IDA projects with MS+ ratings increased from 79 percent in FY12–14 to 80 percent in FY15–17 (by number of projects), while the share of MS+ ratings weighted by commitment volumes decreased from 85 to 80 percent during the same period. The quality at entry and quality of supervision ratings for IDA FCV projects were lower than the average for IDA (appendix F, Figures F.5 and F.6). Looking at the quality at entry and quality of supervision ratings weighted by commitment size, IDA FCV experienced a sharper decline as compared with the IDA average. Where quality of supervision was weak, weak project management was identified as a key driving factor. RAP 2015 conducted a detailed analysis of factors affecting quality of supervision ratings (World Bank 2016e). These include weak project management, weak fiduciary management, low safeguards compliance, inadequate attention to technical issues, and M&E. The analysis found that project teams in these cases were not proactive in revising PDOs or restructuring the project. Weak project management included lack of timely implementation, inadequate and untimely advice to the client, delays in processing documents, and lack of timely follow-up on issues. 35 Chapter 4 Monitoring and Evaluation and Results Data A systemwide approach to building statistical capacity is key to providing country-level data support to IDA countries. IEG’s evaluation on data and statistical capacity found Bank Group support mostly effective at the country level in supporting data production, promoting open data, encouraging country clients to share data, and building the capacity of national statistical organizations in countries where it adopted a systemwide approach (World Bank 2017b). The evaluation reported statistical capacity improvements in IDA case study countries such as Afghanistan, Rwanda,6 and Tanzania. Bank Group support tended to be less effective in encouraging data sharing and use by governments and citizens.7 Data gaps deserve more attention in IDA countries to ensure quality of analytical work and improve M&E capacity. IEG’s SCD/CPF evaluation found that 6 IDA countries— Botswana, Chad, Côte d’Ivoire, Haiti, Mali, and Myanmar—had significant data gaps or out-of-date data (World Bank 2017m). The data inadequacies adversely affected the quality and relevance of the analysis of poverty and shared prosperity, particularly over the three- to five-year period leading to the preparation of countries’ SCDs. Most, but not all, SCDs reviewed identified the need for additional work by country on data issues. 6 In Rwanda, the U.K. Department for International Development, the United Nations Children’s Fund, and the World Bank (through its DPF series) supported the Ministry of Local Government in its ambitious attempt to create an integrated management and evaluation information system linking more than eight social protection programs. 7Data use extends to informing and enhancing policy making, service delivery, monitoring and evaluation, research, advocacy, and citizen engagement. 36 Bibliography World Bank. 2005. Additions to IDA Resources: Fourteenth Replenishment (IDA14)—Working Together to Achieve the Millennium Development Goals. Washington, DC: World Bank. ———. 2011. Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Sixteenth Replenishment, IDA16: Delivering Development Results. Washington, DC: World Bank. ———. 2014a. “The Development Impact of New Zealand’s RSE Seasonal Worker Policy.” Policy Research Working Paper 5488, World Bank, Washington, DC. ———. 2014b. Evaluation of World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States. Washington, DC: World Bank. ———. 2014c. Report from the Executive Directors of the International Development Association to the Board of Governors, Additions to IDA Resources: Seventeenth Replenishment, IDA17: Maximizing Development Impact. Washington, DC: World Bank. ———. 2015. IEG Review of the Completion and Learning Review (CLRR): Bolivia FY12-15. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016a. Cluster Country Program Evaluation on Small States: Pacific Regional Program Evaluation (FY05–15). Independent Evaluation Group. Washington, DC: World Bank. ———. 2016b. Industry Competitiveness and Jobs. An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016c. International Development Association (IDA) 18: Special Theme—Fragility, Conflict and Violence. Washington, DC: World Bank. ———. 2016d. Organization of Eastern Caribbean States (OECS) Regional Program Evaluation (Antigua and Barbuda, Dominica, Grenada, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines) (FY06–14). Independent Evaluation Group. Washington, DC: World Bank. ———. 2016e. Results and Performance of the World Bank Group 2015. Independent Evaluation Group. Washington, DC: World Bank. 37 Bibliography ———. 2016f. Supporting Transformational Change for Poverty Reduction and Shared Prosperity Lessons from World Bank Group Experiences. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016g. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016h. World Bank Group Support for Housing Finance. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017a. Corporate Scorecards: October 2017. Washington, DC: World Bank. ———. 2017b. An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017c. A Thirst for Change the World Bank Group’s Support for Water Supply and Data for Development. Washington, DC: World Bank. ———. 2017d. Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017e. Higher Education for Development. An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017f. Mobile Metropolises: Urban Transport Matters. An IEG Evaluation of the World Bank Group’s Support for Urban Transport. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017g. Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from the World Bank Group Experience. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017h. Report from the Executive Directors of the International Development Association to the Board of Governors. Additions to IDA Resources: Eighteenth Replenishment. Towards 2030: Investing in Growth, Resilience and Opportunity. Washington, DC: World Bank. ———. 2017i. Results and Performance of the World Bank Group 2016. An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017j. Tax Revenue Mobilization: Lessons from World Bank Group Support for Tax Reform. Independent Evaluation Group. Washington, DC: World Bank. 38 Bibliography ———. 2017k. Toward a Clean World for All: An Evaluation of the World Bank Group’s Support to Pollution Management. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017l. Women’s Empowerment in Rural Community-Driven Development Projects. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017m. World Bank Group Country Engagement an Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018a. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World: An Evaluation of the World Bank Group’s Support to Carbon Finance. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018b. Engaging Citizens for Better Development Results. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018c. IDA18 Mid-Term Review: Implementation and Results Progress Report. Washington, DC: World Bank. ———. 2018d. IFC’s Experience with Inclusive Business: An Assessment of IFC’s Role, Outcomes, and Potential Scenarios. Washington, DC: World Bank. ———. 2018e. Inclusive Growth: A Synthesis of Findings from Recent IEG Evaluations. Washington, DC: World Bank. ———. 2018f. Knowledge Flow and Collaboration Under the World Bank’s New Operating Model. Washington, DC: World Bank. ———. 2018g. Maximizing the Impact of Development Policy Financing in IDA: A Stocktaking of Success Factors and Risks. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018h. Recent Word Bank Experience with Risk and Resilience Assessments and Operational Programming in FCV Countries: Stocktaking of Implementation of Recommendations from IEG’s FCS/FCV Evaluations. Washington, DC: World Bank. ———. 2018i. Results and Performance of the World Bank Group 2017. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018j. Rwanda Country Program Evaluation FY09–17. Independent Evaluation Group. Washington, DC: World Bank. 39 Bibliography ———. 2018k. Update to Guidance on World Bank Group Program Outcome Ratings in Independent Evaluation Group Country Program Evaluations. Washington, DC: World Bank. ———. 2018l. World Bank Group Support to Health Services: Achievements and Challenges. Washington, DC: World Bank. 40 Appendix A. IDA18 Special Themes and Their Focuses Jobs and economic transformation. Drawing on lessons learned from IDA17, the Bank Group’s approach under IDA18 focuses on: (i) facilitating connectivity to markets through support for investments in quality infrastructure, regional integration, and global value chains; (ii) building capabilities for increased productivity and earnings through support for facilitating access to finance and building management capacity in small and medium enterprises and microenterprises; and (iii) establishing an economywide incentive framework that facilitates private investment. Gender and development. Under IDA18, the focus is on policy actions in five areas: (i) tackling remaining first-generation gaps in endowments, particularly maternal mortality and gaps in secondary educational attainment; (ii) removing existing constraints to more and better jobs for women; (iii) supporting women’s increased access to financial services and control over assets; (iv) enabling country-level action for more and better data and evidence; and (v) increasing operations that address deprivations of voice and agency, especially in FCV. Climate change. Building on IDA17, IDA18 climate commitments aim to (i) deepen and mainstream climate considerations; (ii) support efforts to achieve the Sustainable Energy for All objectives, including by enhancing electricity access and promoting renewable energy; and (iii) monitor and report on IDA resources used to address climate change. Fragility, conflict, and violence (FCV). IDA FCV account for 21 percent of global poverty and are expected to be home to more than 50 percent of the global poor by 2030 due to their high population growth rates and lagging development (World Bank 2016d). Support for Fragile and Conflict Situations (FCS) was one of the Bank Group’s six strategic themes announced in 2008 and has featured as a special theme since IDA 15.1 Under IDA 18, the Bank Group committed to effectively respond to FCV by (i) responding to FCV across the spectrum of fragility; (ii) mitigating fragility risks in non-FCV; (iii) addressing subnational FCV in higher capacity countries; and (iv) addressing the regional dimension of fragility and mitigating the impact of forced displacement. Governance and institutions. Under IDA18, areas of focus under this theme include domestic resource mobilization, public financial management, public administration reform, 1The World Bank Group replaced the term low-income countries under stress, which was introduced in 2003, with fragile and conflict-affected situations. Support for fragile and conflict- affected situations was one of the World Bank Group’s six strategic themes announced in 2008. 41 citizen engagement, and illicit financial flows. The theme facilitates an integrated, multisectoral approach to public sector reform that serves as a foundation for IDA’s development effectiveness. It also places continued emphasis on data and analysis as well as impact and results. 42 Appendix B. World Bank Project Performance Table B.1. Project Outcome Ratings a. Outcome ratings of MS+, by number of projects Share Projects (percent) (no.) Project or Country Type FY12–14 FY15–17 FY12–14 FY15–17 Project type IDA 68 76 492 318 IBRD 71 78 288 255 Othera 66 75 181 131 IPF-IDA 71 76 406 298 IPF-IBRD 66 77 209 218 DPF-IDA 60 75 90 20 DPF-IBRD 86 84 77 37 Country type All FCV 68 65 164 101 All Non-FCV 69 79 747 558 IDA FCV 69 65 150 91 IDA non-FCV 68 79 404 233 b. Outcome ratings of MS+, weighted by volume Share Volume (percent) ($, millions) Project or Country Type FY12–14 FY15–17 FY12–14 FY15–17 Project type IDA 75 83 31,613 24,606 IBRD 82 87 52,314 44,930 Othera 74 81 3,402 2,087 IPF-IDA 79 83 26,209 23,459 IPF-IBRD 80 87 30,549 30,151 DPF-IDA 59 83 5,354 1,147 DPF-IBRD 86 88 21,765 14,779 Country type All FCV 73 74 6,588 3,833 All Non-FCV 80 86 79,591 64,981 IDA FCV 76 75 5,973 3,292 IDA non-FCV 75 84 29,208 16,795 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCV = fragility, conflict, and violence; FY = fiscal year of project closing; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. a. Other includes agreement types such as the Carbon Initiative, Global Environment Fund, Montreal Protocol, Recipient Executed Trust Fund, Special Fund, and not identified. 43 Appendix B World Bank Project Performance Table B.2. Bank Performance Ratings a. Bank performance ratings of MS+, by number of projects Share Projects (percent) (no.) Project or Country Type FY12–14 FY15–17 FY12–14 FY15–17 Project type IDA 70 76 497 318 IBRD 72 78 292 255 Othera 66 72 182 131 IPF-IDA 71 76 406 298 IPF-IBRD 65 78 212 218 DPF-IDA 69 85 90 20 DPF-IBRD 88 84 78 37 Country type All FCV 71 65 165 101 All Non-FCV 70 79 755 558 IDA FCV 72 65 151 91 IDA non-FCV 70 78 407 233 b. Bank performance ratings of MS+, weighted by volume Share Volume (percent) ($, millions) Project or Country Type FY12–14 FY15–17 FY12–14 FY15–17 Project type IDA 79 82 33,065 24,606 IBRD 84 88 52,617 44,930 Othera 78 80 3,403 2,087 IPF-IDA 81 82 27,622 23,459 IPF-IBRD 80 88 30,551 30,151 DPF-IDA 71 88 5,354 1,147 DPF-IBRD 89 88 22,065 14,779 Country type All FCV 76 72 6,588 3,833 All Non-FCV 82 87 81,344 64,981 IDA FCV 78 72 5,973 3,292 IDA non-FCV 80 84 30,658 16,795 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCV = fragility, conflict, and violence; FY = fiscal year of project closing; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. a. Other includes agreement types such as Carbon Initiative, Global Environment Fund, Montreal Protocol, Recipient Executed Trust Fund, Special Fund, and not identified. 44 Appendix B World Bank Project Performance Table B.3. IDA Development Policy Financing Outcome Ratings Projects Share Volume Share (no.) (percent) ($, millions) (percent) Rating FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 HS 1 0 1.1 0 417.6 0.0 7.8 0 S 13 3 14.4 15 530.5 72.0 9.9 6.3 MS 40 12 44.4 60 2,222.0 875.8 41.5 76.3 MU 26 2 28.9 10 1,586.4 70.0 29.6 6.1 U 10 3 11.1 15 597.1 129.5 11. 2 11.3 Total 90 20 100 100 5,353.6 1,147.3 100 100 Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: FY = fiscal year of project closing; HA = highly satisfactory; IDA = International Development Association; MS = moderately satisfactory; MU = moderately unsatisfactory; S= satisfactory; U = unsatisfactory. 45 Appendix B World Bank Project Performance Table B.4. Project Outcome Ratings, by Global Practice a. Outcome ratings of MS+, by number of projects Share Projects (percent) (no.) Global Practice FY12–14 FY15–17 FY12–14 FY15–17 World Bank–wide 69 76 961 704 Poverty and Equity 50 100 4 5 Social Protection and Labor 89 93 44 29 Education 65 89 89 53 Transport and Digital Development 73 84 84 83 Social, Urban, Rural, and Resilience 77 80 121 90 Agriculture 70 75 74 73 Finance, Competitiveness, and Innovation 71 75 56 24 Health, Nutrition, and Population 78 75 74 52 Environment and Natural Resources 59 74 63 61 Macroeconomics, Trade, and Investment 68 73 106 41 Energy and Extractives 65 70 92 61 Water 61 67 69 67 Governance 54 64 68 44 Trade and Competitiveness 69 62 13 21 b. Outcome ratings of MS+, weighted by volume Share Volume (percent) ($, millions) Global Practice FY12–14 FY15–17 FY12–14 FY15–17 World Bank–wide 80 86 87,329 71,623 Poverty and Equity 97 100 557 738 Social Protection and Labor 95 100 6,816 4,054 Education 81 94 7,188 4,047 Transport and Digital Development 93 94 6,207 4,550 Social, Urban, Rural, and Resilience 63 91 4,508 2,876 Agriculture 82 90 9,930 7,977 Finance, Competitiveness, and Innovation 81 88 14,195 10,720 Health, Nutrition, and Population 53 86 986 475 Environment and Natural Resources 81 83 8,861 6,264 Macroeconomics, Trade, and Investment 85 82 2,609 1,831 Energy and Extractives 69 82 10,605 13,287 Water 82 81 3,328 5,143 Governance 82 80 5,875 3,915 Trade and Competitiveness 63 67 5,552 5,745 Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: World Bank–wide includes projects not tagged to a Region; FY = Fiscal Year; MS+ = moderately satisfactory or above. 46 Appendix B World Bank Project Performance Table B.5. Project Outcome Ratings, by Global Practice and Institution a. Outcome ratings of MS+, by number of projects IDA IBRD (no.) (no.) Global Practice FY12–14 FY15–17 FY12–14 FY15–17 World Bank–wide 69 82 65 73 Poverty and Equity 59 85 70 100 Social Protection and Labor 64 57 71 80 Education 54 77 67 64 Transport and Digital Development 59 57 94 92 Social, Urban, Rural, and Resilience 52 58 50 63 Agriculture 81 76 73 69 Finance, Competitiveness, and Innovation 59 75 90 82 Health, Nutrition, and Population 100 100 Environment and Natural Resources 91 88 87 100 Macroeconomics, Trade, and Investment 84 79 69 80 Energy and Extractives 67 64 67 75 Water 70 83 77 85 Governance 80 78 44 61 Trade and Competitiveness 68 76 71 78 b. Outcome ratings of MS+, weighted by volume IDA IBRD ($, millions) ($, millions) Global Practice FY12–14 FY15–17 FY12–14 FY15–17 World Bank–wide 83 83 83 75 Poverty and Equity 77 94 90 100 Social Protection and Labor 77 62 85 91 Education 71 63 90 89 Transport and Digital Development 75 84 99 95 Social, Urban, Rural, and Resilience 74 76 57 97 Agriculture 76 88 89 55 Finance, Competitiveness, and Innovation 59 81 90 90 Health, Nutrition, and Population — — 100 100 Environment and Natural Resources 90 99 98 100 Macroeconomics, Trade, and Investment 82 92 82 88 Energy and Extractives 81 86 40 94 Water 62 81 72 83 Governance 84 71 50 65 Trade and Competitiveness 75 83 82 87 Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = moderately satisfactory and above. 47 Appendix C. Additional Findings from Reviews of Country Strategies Drawing on Independent Evaluation Group (IEG) Completion and Learning Report Reviews (CLRRs), this section assesses whether the thematic content of the objectives was consistent with any of the five special themes from the 18th replenishment of the International Development Association (IDA18).1, 2 Objectives found to be aligned with one of the five special themes were mapped to the relevant special theme. Country Partnership Framework (CPF) objectives that did not fit any of the special themes were not mapped. Of 360 objectives in the 32 country programs under reviewed CLRRs, 206 objectives are mapped to at least one of the five special themes. Efficacy ratings, where IEG assesses the extent to which objectives were met, were used (table C.2). 1Independent Evaluation Group (IEG) review of Completion and Learning Reviews (CLRs) of country strategies. At completion of each country strategy—Country Assistance Strategies, Country Partnership Strategies, and Country Partnership Frameworks (CPFs)—the World Bank Group team prepares a CLR, which is a critical input into the design and implementation of the next CPF. IEG conducts a desk-based review of the CLR and provides an independent validation of the document, including its rating of the outcome of the country program. IEG then submits the CLR Review to the Board of Executive Directors as an input into its discussion of the new CPF. IEG’s validation of the CLR provides an independent rating on two key dimensions: the development outcome of the country program and the Bank Group’s performance. In addition to the overall development outcome rating, the extent of achievement of each CPF objective is independently assessed and rated by IEG. 2CLR Reviews examine achievement of individual Country Assistance Strategy / CPF objectives, as modified or adjusted in some cases by Country Assistance Strategy Progress Reports or Performance and Learning Reviews. 48 Appendix C Additional Findings from Reviews of Country Strategies Table C.1. IEG Completion and Learning Report Reviews, by Region and Institution, FY16–18 IBRD IDA Total Region (no.) (%) (no.) (%) (no.) (%) Africa 2 8 13 41 15 27 East Asia and Pacific 2 8 8 25 10 18 Europe and Central Asia 10 42 3 9 13 23 Latin America and the Caribbean 6 25 5 16 11 20 Middle East and North Africa 4 17 1 3 5 9 South Asia 0 0 2 6 2 4 Total 24 100 32 100 56 100 Source: Independent Evaluation Group. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IEG = Independent Evaluation Group. Table C.2. IEG Efficacy Ratings of Objectives under IDA Special Themes IEG Efficacy Rating J&E Governance Climate Gender FCV Achieved (no.) 23 3 8 3 0 Mostly achieved (no.) 28 6 9 0 1 Partially achieved (no.) 36 16 12 1 1 Not achieved (no.) 13 13 10 1 1 Not verified or rated (no.) 10 6 5 0 0 Total (n = 206) (no.) 110 44 44 5 3 Share (percent) 53 21 21 2 1 Source: Independent Evaluation Group. Note: IDA = International Development Association; IEG = Independent Evaluation Group; FCV = fragility, conflict, and violence; J&E = jobs and economic transformation. Jobs and Economic Transformation The substance of World Bank Group objectives mapped to this special theme were broadly in line with the IDA18 areas of action and included building quality infrastructure, raising productivity within the agricultural sector, facilitating connectivity to market opportunities and building the capabilities of firms and people to take advantage of these opportunities, and enhancing private sector competitiveness and integration with global markets. Objectives aimed at improving the quality of infrastructure focused mainly on improvements in transportation (Uganda, Mali), power (Kosovo, Laos) and water (Haiti and Mozambique) infrastructure. The objective of increasing connectivity was pursued through increasing coverage and lowering costs of telecommunications services (Kiribati, Bolivia, The Gambia) and by supporting measures to exploit the potential for regional integration (Uganda and Ethiopia). World Bank support under this theme also sought to increase productivity in agriculture by 49 Appendix C Additional Findings from Reviews of Country Strategies supporting equitable access to secure land tenure for men and women, credit and insurance, extension services, and electricity and irrigation. In Mali, for example, seven out of the 21 objectives were related to jobs and competitiveness and included increasing access to electricity, improving the business environment, increasing agricultural productivity and competitiveness, and improving integration of Mali with regional and global markets. The efficacy of Bank Group country programs on the jobs and competitiveness–related objectives was mixed, with about half the objectives receiving a rating of mostly achieved or higher. Of the 110 jobs and competitiveness–related objectives, 100 were rated by IEG. About half (51) of these were rated as mostly achieved or higher, 36 as partially achieved, and 13 as not achieved (table C.2). Climate Change The theme of climate change featured prominently and had the second highest number of objectives mapped to it. Out of the 32 IDA CLRRs reviewed, 28 had objectives related to mitigating the effects of and building resilience to climate change (see table 2. 2 in chapter 2). For example, the objective of enhancing disaster preparation and response capacity featured in the CLRRs of disaster-prone countries such as Haiti, Tuvalu, Bangladesh, and Madagascar. Protection of natural resources was an objective in the CLRRs of Côte d'Ivoire (forests), Sri Lanka (IDA graduate in FY17) (productive land), and the Federated States of Micronesia (fisheries). Less than half the IEG-rated objectives related to climate were mostly achieved or higher. Thirty-nine out of the 44 climate objectives were rated by IEG. Of these, 44 percent (29 objectives) were rated as mostly achieved or higher while 31 percent (10 objectives) were rated as not achieved. Successful cases include the country program in Bolivia (IDA graduate in FY17), as highlighted in the main text. Governance and Institutions The theme of governance was an important part of many of the CLRRs for country strategies in IDA countries and focus mainly on public finance. Twenty-three of the 32 countries considered (72 percent), had governance related objectives. Specifically, 44 of the 206 objectives were governance objectives. Twenty-six of the 44 governance objectives had to do with public finance, including improving tax administration and revenue management, quality of internal and external budget control, procurement, and public expenditure management. Improving transparency also featured prominently in the CLRRs of 10 countries and accounted for 11 of the 44 governance objectives. In IDA countries such as Niger and Mozambique, Bank support for improving transparency 50 Appendix C Additional Findings from Reviews of Country Strategies was focused on the extractives sector, the main vehicle for which was the Extractive Industries Transparency Initiative.3 World Bank support for improving citizen and community participation in governance was present in the CLRRs of four countries (six objectives). Notably absent from the CLRRs reviewed were objectives specific to domestic resource mobilization. This can be explained however by the fact that the theme of building capacity for domestic resource mobilization started to be emphasized after the Addis Ababa meetings in 2015 by which point most of the underlying country strategy documents were already in existence (World Bank 2013). Almost a quarter of the governance related objectives were rated as mostly achieved or better while a little more than a third were rated as not achieved. For example, Mozambique had achieved one governance objective—improving transparency in extractive industries—with four grants that supported Mozambique to become a member and comply with the Extractive Industries Transparency Initiative. Gender and Development Gender-related objectives were rarely mentioned explicitly in strategy objectives, although gender was integrated into Bank Group activities. The review focused on identifying objectives that explicitly mentioned gender or gender-specific issues. The exercise considered the fact that monitoring indicators for some objectives related to jobs, education, and health specified different targets for men and women. Objectives, which are not specific to gender, but with gender-specific indicators in results framework, were not counted as gender objectives, despite their contributions to highlighting differences in results between men and women. As such, the review mapped only five out of 206 objectives (in five out of 32 countries) to the gender special theme. Three out of these five objectives were rated by IEG as having been achieved while one was rated as partially achieved. A possible explanation for the paucity of explicit gender objectives is that the World Bank’s approach was to mainstream gender across all its objectives and activities as opposed to having only a specific objective on gender in each CPF. 3The Extractive Industries Transparency Initiative is the global standard for the good governance of oil, gas, and mineral resources. When implemented by a country, the Extractive Industries Transparency Initiative ensures transparency and accountability about how a country's natural resources are governed. This ranges from how the rights are issued, to how the resources are monetized, to how they benefit the citizens and the economy. 51 Appendix C Additional Findings from Reviews of Country Strategies Fragility, Conflict, and Violence The review of IDA country program objectives cited in CLRRs found only a limited number of fragility, conflict, and violence (FCV)-related objectives despite the large number of FCV countries. Only three out of 206 objectives (in 2 of 32 countries) directly referenced FCV. This is in spite of the fact that 17 of the 32 countries reviewed were/are on the World Bank list of fragile and conflict-affected situations. The three objectives that focused explicitly on fragility and violence sought to strengthen regional approaches to combat crime and violence in Central America (Honduras) and increase postwar economic opportunities (Côte d'Ivoire). Honduras, as an example, partially achieved the objective of strengthening capacity of communities and local governments in social prevention of crime and violence. The Bank Group supported the flagship government Safer Municipalities Program aimed at improving citizen security across municipalities and focused its support on the poorest urban municipalities with high levels of crime. While the project achieved some outcomes, it did not achieve its development objectives and exited the portfolio as unsatisfactory. Reference World Bank. 2013. Financing for Development Post 2015. Washington, DC: World Bank. 52 Appendix D. The Incorporation of Gender in Projects In fiscal year (FY)16, the Independent Evaluation Group (IEG) piloted a systematic documentation of gender dimensions in individual World Bank projects as part of the Implementation Completion and Results Report (ICR) review process. ICR reviewers examine the following: • Whether gender is an explicit part of the project development objective (PDO) or one of the project components • Whether the ICR reports sex-disaggregated or female- or male-specific indicators. Building on the pilot analysis in the Results and Performance of the World Bank Group in 2015, 2016, and 2017, the latest analyses are presented below. Table D.1. Projects that Incorporated Gender in PDOs or Components, FY14–18 (number) Projects with Gender Explicit Part of Projects with Gender At Least One Component (but not Institution Explicit in PDO the PDO) IDA (N = 202) 16 48 IBRD (N = 156) 7 21 World Bank–wide (N = 440) 29 85 Source: Independent Evaluation Group analysis. Note: World Bank–wide include “other” categories. PDO = project development objective. Table D.2. Presence of Gender-Relevant Indicators among Projects that Incorporated Gender as Part of the PDO or at Least One Component (percent) ICR Contains Sex- Disaggregated or ICR Reports Only the Male- or Female- Share or Percentage of No Gender- Institution Specific Indicators Female Beneficiaries Specific Reporting IDA (N = 64) 73 13 14 IBRD (N = 28) 43 11 46 World Bank–wide 65 11 24 (N = 114) Source: Independent Evaluation Group analysis. Note: ICR = Implementation Completion and Results Report; PDO = project development objective. 53 Appendix D The Incorporation of Gender in Projects Table D.3. IDA Projects that Incorporated Gender in PDOs or Components, by Global Practices, FY14–18 (number) Projects with Gender Explicit Part of At Least Projects with Gender One Component (but not Practice Group Explicit in PDO the PDO) Agriculture (N = 8) 2 7 Education (N = 14) 3 4 Energy and Extractives (N = 12) 0 4 Environment and Natural Resources (N = 16) 0 1 Finance, Competitiveness and Innovation 0 0 (N = 3) Governance (N = 3) 1 1 Health, Nutrition and Population (N = 9) 6 8 Macroeconomics, Trade and Investment 0 3 (N = 1) Poverty and Equity (N = 0) 0 0 Social Protection and Labor (N = 2) 0 5 Social, Urban, Rural and Resilience Global 3 4 Practice (N = 8) Trade and Competitiveness (N = 3) 1 4 Transport and Digital Development (N = 8) 0 2 Water (N = 5) 0 5 World Bank–wide (N = 92) 16 48 Source: Independent Evaluation Group analysis. Note: N = number of projects; PDO = project development objective. FY refers to the year the project was closed. 54 Appendix D The Incorporation of Gender in Projects Figure D.1. IDA Projects that Incorporated Gender in PDOs or Components Gender was an explicit part of at least one component (but not the PDO) 50 Gender was an explicit part of the PDO 45 40 36 35 35 29 30 Percent 25 23 20 27 15 25 10 5 13 7 0 3 2015 (N=55) 2016 (N=63) 2017 (N=70) Fiscal year Source: Independent Evaluation Group analysis. Note: PDO = project development objective. 55 Appendix E. Methodology for Climate Change Analyses The assessment of the support for climate change in country strategies and in the project portfolio is based on methods and a database developed for the Results and Performance of the World Bank Group 2017 (RAP17; World Bank 2018i). Appendix C of that report gives the full methodology. For both country strategies and projects, the analysis is based on a classification of potential environmental benefits (including climate change mitigation and climate change adaptation) developed jointly with operational staff as part of RAP17. This framework identified 13 subsectors of environmental benefits, including climate change mitigation and climate change adaptation/disaster risk management. The database from RAP17 was used for this report on IDA, but additional analysis was carried out to separate IDA countries (including IDA/IBRD Blend countries) from IBRD countries, using IDA-eligibility definitions as of June 2018. The results are based on a comprehensive review of the components (or pillars for DPF) of a one-third random sample of World Bank projects approved in FY08–10 and FY15– 17, covering 1,684 components from 558 projects. Of these, 1,084 components are from IDA countries. The review was based solely on an expert review of appraisal documents, with oversight from a secondary expert reviewer. This review did not include analytical and advisory activities or “nonstandard” product lines such as carbon finance. Project components were identified as offering potential climate change mitigation benefits if they provided one or more of: supply side energy efficiency, demand side energy efficiency, extractive industry climate change mitigation, renewable energy, reduced emissions from fossil fuel combustion, sectoral climate change mitigation for transport, agriculture, forestry, or urban sectors, or climate change mitigation policy reform. Climate change adaptation was not meaningfully separable from (climate- related) disaster risk management. Project components were identified as offering potential climate change adaptation / domestic resource mobilization benefits if they included disaster-related or disaster-resistant infrastructure, relevant institutional reform or capacity building, proresilience policy reform, financial disaster risk management, resilience from ecosystem services, health-related climate change adaptation, disaster-related safety nets, and resilience-related climate smart agriculture. An individual component could potentially provide both climate change mitigation and climate change adaptation benefits. In assessing country strategies with climate change-related objectives or outcomes, the results are based on an assessment of all 208 strategies covering 120 countries prepared 56 Appendix E Methodology for Climate Change Analyses during FY08–17 carried out for RAP17. In this report on IDA, the focus was on the 101 strategies prepared for IDA-eligible countries. IEG analyzed the extent to which objectives or outcomes in these strategies supported climate change mitigation or climate change adaptation, using the same classification framework as for projects. 57 Appendix F. Bank Performance and Monitoring and Evaluation Ratings Figure F.1. World Bank Projects: Quality of M&E Ratings, FY06–17 a. M&E ratings, three-year rolling, by number of projects 100 80 60 36 36 33 35 40 40 31 29 28 30 30 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=571) (N=659) (N=704) (N=797) (N=914) (N=979) (N=950) (N=868) (N=768) (N=704) b. M&E ratings, three-year rolling, weighted by volume 60 48 45 37 49 48 39 51 39 41 37 40 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=44,170) (V=48,228) (V=57,943) (V=73,752) (V=84,972) (V=91,542) (V=85,829) (V=86,701) (V=77,365) (V=71,623) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; M&E = monitoring and evaluation; V = total volume of projects ($, millions). 58 Appendix F Bank Performance and Monitoring and Evaluation Ratings Figure F.2. IDA Projects: Quality of Monitoring and Evaluation Ratings, FY06–17 a. M&E ratings, three-year rolling, by number of projects M&E Quality, 3-Year Rolling By Number 100 80 60 32 30 31 33 36 40 31 27 25 28 24 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=300) (N=340) (N=349) (N=387) (N=457) (N=488) (N=485) (N=430) (N=367) (N=318) b. M&E ratings, three-year rolling, weighted by volume M&E Quality 3-Year Rolling By Volume 60 48 30 48 41 41 27 33 38 34 40 28 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=15,027) (V=17,064) (V=19,093) (V=20,106) (V=24,292) (V=27,681) (V=30,365) (V=28,233) (V=24,305) (V=24,606) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; IDA = Independent Evaluation Group; M&E = monitoring and evaluation; V = total volume of projects ($, millions). 59 Appendix F Bank Performance and Monitoring and Evaluation Ratings Figure F.3. IDA Projects: Quality at Entry Ratings, FY06–17 a. Quality at entry ratings, three-year rolling, by number of projects Quality at Entry, 3-Year Rolling By Number 100 80 72 73 70 64 60 61 58 59 59 59 60 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=383) (N=359) (N=362) (N=395) (N=463) (N=494) (N=490) (N=432) (N=367) (N=318) b. Quality at entry ratings, three-year rolling, weighted by volume Quality at Entry, 3-Year Rolling By Volume 100 77 77 80 71 71 70 67 68 70 66 65 60 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=18,997) (V=17,913) (V=19,882) (V=20,621) (V=26,219) (V=29,310) (V=31,982) (V=28,235) (V=24,305) (V=24,606) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; IDA = International Development Association; V = total volume of projects ($, millions). 60 Appendix F Bank Performance and Monitoring and Evaluation Ratings Figure F.4. IDA Projects: Quality of Supervision Ratings, FY06–17 a. Quality of supervision ratings, three-year rolling, by number of projects Quality of Supervision, 3-Year Rolling By Number 100 84 83 79 78 79 79 78 77 80 77 80 60 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=378) (N=358) (N=360) (N=392) (N=460) (N=492) (N=489) (N=431) (N=367) (N=317) b. Quality of supervision ratings, three-year rolling, weighted by volume Quality of Supervision, 3-Year Rolling By Volume 100 84 81 80 84 85 83 79 81 79 80 80 60 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=18,661) (V=17,913) (V=19,732) (V=20,459) (V=26,057) (V=29,048) (V=31,732) (V=27,985) (V=24,305) (V=24,551) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; IDA = International Development Association; V = total volume of projects ($, millions). 61 Appendix F Bank Performance and Monitoring and Evaluation Ratings Figure F.5. IDA FCV Projects: Quality at Entry Ratings, FY06–17 a. Quality at entry ratings, three-year rolling, by number of projects Quality at Entry, 3-Year Rolling By Number 100 80 74 70 66 66 62 56 56 53 54 57 60 40 20 0 20 40 FY06-08 (N=67) FY07-09 (N=81) FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=91) (N=105) (N=143) (N=167) (N=166) (N=151) (N=128) (N=109) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; N = number of projects. b. Quality at entry ratings, three-year rolling, weighted by volume Quality at Entry, 3-Year Rolling By Volume 100 87 85 80 75 63 65 66 57 59 54 60 49 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=1,909) (V=3,120) (V=4,286) (V=5,492) (V=5,579) (V=5,435) (V=5,973) (V=5,658) (V=4,801) (V=3,292) Source: Independent Evaluation Group data. Note: FY = project closing fiscal year; IDA = International Development Association; V = total volume of projects ($, millions). 62 Appendix F Bank Performance and Monitoring and Evaluation Ratings Figure F.6. IDA FCV Projects: Quality of Supervision Ratings, FY06–17 a. Quality of supervision ratings, three-year rolling, by number of projects Quality of Supervision, 3-Year Rolling By Number 100 79 79 82 78 73 75 77 74 80 72 69 60 40 20 0 20 40 FY06-08 (N=67) FY07-09 (N=81) FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=90) (N=103) (N=140) (N=164) (N=165) (N=151) (N=128) (N=109) b. Quality of supervision ratings, three-year rolling, weighted by volume Quality of Supervision, 3-Year Rolling By Volume 100 88 88 74 79 79 80 80 71 71 71 72 60 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=1,909) (V=3,120) (V=4,136) (V=5,330) (V=5,417) (V=5,423) (V=5,973) (V=5,658) (V=4,801) (V=3,237) Source: Independent Evaluation Group data. Note: FCV = fragility, conflict, and violence; FY = project closing fiscal year; IDA = International Development Association; V = total volume of projects ($, millions). 63 Appendix G. Monitoring and Evaluation Issues in Project Performance Assessment Reports Objective: The objective of the exercise is to identify the main monitoring and evaluation (M&E) issues discussed in recent Project Performance Assessment Reports (PPARs) and to the extent that they are available, extract the relevant lessons and recommendations made in the reports. Methodology: The exercise involved a systematic review and coding of the M&E issues/challenges discussed in each report.1 The list M&E challenges was categorized and for each report, a value of one was assigned if a specific challenge was discussed and zero if the report did not mention the issue. Where multiple problems were discussed, a value of one was assigned to each instance. In addition, the lessons section of the PPAR was reviewed to identify any lessons relevant to M&E. Scope: The exercise was limited to the PPARs of IDA and blend countries and considered only PPARs completed during the period FY16 up to end of Q2 FY19. Overall, 47 PPARs covering 57 projects in 22 countries and eight regional projects were included in the review. Findings Most frequently cited M&E challenges were due to weaknesses in the M&E design. Additionally, the upstream weaknesses in M&E design frequently affected the downstream M&E implementation and eventual use of the projects’ M&E data. The choice of results indicators was the leading cause of challenges in M&E design. In several cases, results indicators were found to have little relevance to the project objectives/outcomes while more relevant indicators were not included in the results framework. The scope of the indicators was sometimes limited and captured only partial aspects of the intended result, or too broad and measured results beyond the scope of project activities. For example, the results framework of the Sierra Leone Public Financial Management project (P108069; FY09) included some indicators that were not affected by project activities or were insignificant since the target was already achieved at baseline. Moreover, the remaining indicators, though relevant, did not fully measure the impact of the project activities on the project development objective. Other issues affecting the design of M&E frameworks included among others: (i) the use of more 1 Each Project Performance Assessment Report includes a rating for monitoring and evaluation. The rating is based on an assessment of three main elements: (i) design; (ii) implementation; and (iii) use of data. 64 Appendix G Monitoring and Evaluation Issues in Project Performance Assessment Reports numerous indicators than is considered good practice, (ii) use of process and output- focused indicators, (iii) use of indicators lacking quantitative baselines and targets, (iv) failure to include or undertake impact assessments or surveys critical to the quality of outcome indicators and their measurement, (v) inappropriate data collection instruments, (vi) lack of an adequate M&E plan, including training and allocation of roles and responsibilities between the different implementing/reporting layers of the project (table F.1). Challenges in implementation of M&E and use of M&E data were often due to legacy issues from poor M&E design. Implementation of M&E was affected by issues such as limited capacity, poor technology, inadequate training and resourcing, and weak M&E systems which would hamper the systematic collection of M&E data. For example, the design of the Cambodia demand for good governance project (P101156; FY09) called for each implementing agency to have its own M&E system, managed by dedicated internal units. However, different stakeholders had differing interpretations of indicators, and thus aggregation was problematic. Additionally, M&E requirements were not appropriate, given the low local capacity. Some indicators required surveys—in a country where there is limited expertise in survey methodology. Key concepts such as activities, output, outcome, and impact do not have words in Khmer, and were difficult to grasp for the Cambodians staffing M&E units. Lastly, the M&E training provided was too theoretical, and M&E budgets were inadequate. To address these constraints, the M&E systems should have been as simple as possible. The use of M&E data was affected by the lack of data, absence of baseline or target values. Moreover, even in cases where M&E data was properly collected, there was sometimes limited evidence that the M&E data was used in decision-making. Instances of successful M&E occurred where appropriate M&E design was coupled with effective implementation of M&E and where there was sufficient evidence of use of M&E data in decision-making. For example, in the Senegal Nutrition Enhancement Program (P070541), the M&E design was well embedded institutionally and emphasized stakeholder ownership and use for the purposes of learning, accountability and decision-making. The indicators were measurable, and some baselines were available and documented in the project appraisal document, largely drawn from the Demographic and Health Survey. Other baselines specific to intervention areas were established by Knowledge Practice and Coverage surveys and an impact evaluation. In addition, there was strong evidence of the use of data for decision-making. A PPAR for two projects implemented in two recent IDA graduates (Bolivia and Vietnam) found that they faced similar M&E challenges to IDA countries. The issues identified included weak M&E implementation, use of indicators that are too output 65 Appendix G Monitoring and Evaluation Issues in Project Performance Assessment Reports focused or that bear little relevance to the objective and no evidence of use of M&E data. For example, the Vietnam Forest Sector Development Project (P066051) experienced several M&E related challenges. First, the original key performance indicators were too output focused and had to be adjusted at the midterm review. Second, M&E implementation was weak in part due to the inadequate training provided to the M&E staff. Lastly, the commune working groups and extension agents did not help collect data and biodiversity tracking was limited, partly because most of the grants to management boards were too small to cover the expense of this activity. Table G.1. M&E Challenges Identified in IDA PPARs Projects Affected M&E Issue (no.) Indicators captured only partial aspects of the intended result 34 Inadequate relevance of indicators to objectives/outcomes 32 Focus on outputs rather than outcomes 21 Lack of M&E system/framework 15 Lack of or inaccurate quantitative targets/baselines 20 Insufficient incorrect or unavailable data 20 Limited evaluation capacity/training 8 No impact surveys 7 Inadequate resourcing of M&E 2 No evidence of M&E data use 21 IDA projects reviewed 65 Source: Independent Evaluation Group. Note: IDA = International Development Association; M&E = monitoring and evaluation; PPAR = Project Performance Assessment Report. Lessons The final outcomes in a results framework need to be specific and attributable to the project. For example, it is difficult to attribute the outcomes of relatively smaller projects to broadly defined outcomes at the country level. Efforts should be made to design project-level outcome indicators so that they are directly relevant to the specific activities of the project (Lao Trade Development Facility; P106165) Community-driven development projects require a solid and comprehensive M&E framework and management information system at the start of the project. M&E is typically a weak element in all projects, but it presents special challenges in community- driven development interventions, which require multidimensional and somewhat intangible outcomes such as community empowerment, participation, and decision- 66 Appendix G Monitoring and Evaluation Issues in Project Performance Assessment Reports making to be measured alongside more tangible technical outputs (creation of infrastructure, correct process implementation) and benefits generated by increased access to services. Yet, exactly because of the complexities and challenges of community- driven development interventions, it is important to have a complete and robust M&E system at the outset to guide project implementation and monitor the realization of hundreds or thousands of subprojects (Lao Poverty Reduction Fund Project; P077326). Programmatic approaches that involve multiple projects over a long period of time are more effective when they measure the extent to which overall programmatic objectives are being met, in addition to assessing project outputs. A results framework is needed that specifies not only the short-term objectives of each of the projects in the series, but also how these link to the overall program objectives. Monitoring and evaluation systems should track not only individual project accomplishments but should also be designed to assess the programmatic objectives, even if these can only be achieved over time (Ethiopia Pastoral Community Development Adaptable Program Loans 1 and 2; P075915/ P108932) When M&E is not adequately resourced or implemented, there can be delays in the identification of problems and the application of remedies. M&E weaknesses can be compensated by leveraging information available elsewhere, or through specific impact assessments and perceptions surveys (Burkina Faso Growth and Competitiveness Credits 1–4). M&E in conflict areas. In an emergency response project, in a conflict-affected setting, M&E needs to be designed in a manner that provides iterative access and reporting on physical assets, services, and social phenomena, and conducted by persons that can access areas that are frequently but intermittently inaccessible. Rapid assessment of implementation quality, feedback, and redress can reduce a project’s chance of contributing to or reigniting conflict (Central African Republic Emergency Food crisis and Agriculture relaunch project; P149512). 67 Appendix H. List of Independent Evaluation Group Evaluations Table H.1. Country Program Evaluations from IDA and Blend Countries Delivered FY17–18 Lending Delivery Evaluation Category No. FY Evaluation Country Period Region (FY19) 1 2018 CPE Rwanda FY09–18 AFR IDA 2 2017 Cluster CPE on Small States Dominica FY06–14 LAC Blend Vol. I OECS Regional Program Evaluation 3 2017 Cluster CPE on Small States St. Lucia FY06–14 LAC Blend Vol. I OECS Regional Program Evaluation 4 2017 Cluster CPE on Small States St. Vincent FY06–14 LAC Blend Vol. I OECS Regional Program Evaluation and the Grenadines 5 2017 Cluster CPE on Small States Grenada FY06–14 LAC Blend Vol. I OECS Regional Program Evaluation 6 2017 Cluster CPE on Small States Federated FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation States of Micronesia 7 2017 Cluster CPE on Small States Kiribati FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation 8 2017 Cluster CPE on Small States The FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation Marshall Islands 9 2017 Cluster CPE on Small States Samoa FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation 10 2017 Cluster CPE on Small States Tonga FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation 11 2017 Cluster CPE on Small States Tuvalu FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation 12 2017 Cluster CPE on Small States Vanuatu FY05–15 EAP IDA Vol. II Pacific Regional Program Evaluation Source: Independent Evaluation Group. Note: There is no CPE delivered in FY16. CPE = Country Program Evaluation; AFR = Africa; EAP = East Asia and Pacific; LAC = Latin America and the Caribbean; OECS = Organization of Eastern Caribbean States. 68 Appendix H List of Independent Evaluation Group Evaluations Table H.2. Completion and Learning Report Reviews of Country Strategies from IDA and Blend Countries Delivered FY15–18 CASCR Country Review Review Classification FY19 IDA # Country Period Region FY (Review Year) Classification 1 Bangladesh (FY11–15) SAR 2016 IDA IDA 2 Bolivia (FY12–15) LAC 2016 IDA IBRD 3 Cambodia (FY05–15) EAP 2016 IDA IDA 4 Cameroon (FY10–14) AFR 2017 Blend Blend 5 Côte d'Ivoire (FY10–14) AFR 2016 IDA IDA 6 Ethiopia (FY13–16) AFR 2017 IDA IDA 7 Gambia, The (FY13–16) AFR 2018 IDA IDA 8 Guinea (FY14–17) AFR 2018 IDA IDA 9 Guyana (FY09–12) LAC 2016 IDA IDA 10 Haiti (FY09–14) LAC 2016 IDA IDA 11 Honduras (FY12–15) LAC 2016 IDA IDA 12 Kiribati (FY11–14) EAP 2017 IDA IDA 13 Kosovo (FY12–16) ECA 2017 IDA IDA 14 Lao PDR (FY12–16) EAP 2017 IDA IDA 15 Lesotho (FY10–14) AFR 2016 IDA IDA 16 Madagascar (FY07–13) AFR 2017 IDA IDA 17 Mali FY08–15) AFR 2016 IDA IDA 18 Marshall Islands (FY13–16) EAP 2017 IDA IDA 19 Micronesia, Fed. Sts. (FY14–17) EAP 2017 IDA IDA 20 Moldova (FY14–17) ECA 2018 Blend Blend 21 Mozambique (FY12–15) AFR 2017 IDA IDA 22 Nicaragua (FY13–17) LAC 2018 IDA IDA 23 Niger (FY13–16) AFR 2018 IDA IDA 24 Samoa (FY12–16) EAP 2017 IDA IDA 25 Sri Lanka (FY13–16) SAR 2016 IDA IBRD 26 Tanzania ((FY12–16) AFR 2018 IDA IDA 27 Togo* (FY08–17) AFR 2017 IDA IDA 28 Tonga (FY11–14) EAP 2017 IDA IDA 29 Tuvalu (FY12–15) EAP 2017 IDA IDA 30 Uganda (FY11–15) AFR 2016 IDA IDA 31 Uzbekistan (FY12–15) ECA 2016 Blend Blend 32 Yemen, Rep. (FY10–15) MNA 2017 IDA IDA Source: Independent Evaluation Group. Note: CLRRs delivered and planned to be delivered in FY19 are not included, even though their review periods end in FY18 or before. AFR = Africa; CASCR = Country Assistance Strategy Completion Review; EAP = East Asia and Pacific; ECA = Europe and Central Asia; FY = fiscal year; IDA = International Development Association. LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 69 Appendix H List of Independent Evaluation Group Evaluations Table H.3. Thematic Evaluations and Learning Products No. Delivery Evaluation Evaluation Type FY 1 FY17 Bank Group support for shared prosperity Major evaluation 2 FY17 World Bank Group Country Engagement: An Early-Stage Assessment of Major evaluation the SCD and CPF Process and Implementation 3 FY16 Learning from IDA Experience Learning Product 4 FY18 DPO in IDA Countries Meso evaluation 5 FY19 Inclusive Growth Synthesis report 6 FY19 Bank Group's Approaches to Engaging Citizens Major evaluation 7 FY18 Essential Health Care Major evaluation 8 FY18 IFC’s Experience with Inclusive Business Meso evaluation 9 FY17 Data for Development Evaluation Major evaluation 10 FY17 Water Supply and Sanitation Major evaluation 11 FY17 World Bank Group Engagement in Health PPPs Synthesis report 12 FY17 Higher Education for Development Major evaluation 13 FY17 Growing the Rural Nonfarm Economy to Alleviate Poverty Major evaluation 14 FY17 Urban Transport Services Major evaluation 15 FY17 Financial Viability of the Electricity Sector in Developing Countries Learning Product 16 FY17 Reliable and Affordable Off-Grid Electricity Services for the Poor Learning Product 17 FY17 Results and Performance of the World Bank Group (RAP) 2016: Annual review managing for development results 18 FY16 Evaluation of the World Bank Group’s Support for Capital Market Major evaluation Development 19 FY16 World Bank Group Support for Housing Finance Learning Product 20 FY16 World Bank Group Support to Industrial Competitiveness and Its Major evaluation Implications for Jobs 21 FY16 Supporting Transformational Change for Poverty Reduction and Shared Learning product Prosperity 22 FY16 Bank Group Support Fragile Situations in Non-FCS Countries Major evaluation 23 FY16 Land Administration Learning Product 24 FY16 DPO: Poverty and Social Impact Analysis Learning Product 25 FY17 Gender in Community-Driven Development learning product Learning product 26 FY16 Results and Performance of the World Bank Group (RAP) 2015: gender Annual review integration 27 FY18 Clean World for All Major evaluation 28 FY18 Carbon Finance Major evaluation 29 FY18 Results and Performance of the World Bank Group (RAP) 2017: Annual review environmental sustainability 30 FY16 Lessons from Environmental Policy Lending Learning Product 31 FY15 DPO: Environment and Social Risk Management Learning Product Source: Independent Evaluation Group. Note: DPO = development policy operation; PPP = public-private partnerships. 70 Appendix H List of Independent Evaluation Group Evaluations Table H.4. PPARs on IDA and Blend Countries, Delivered in FY16–19 Lending Delivery Category No. FY Document Name Country (FY19) 1 2019 Bangladesh and Nepal: strengthening regional South Asia; IDA cooperation for wildlife protection in Asia Bangladesh 2 2018 Mali—Project to support grassroots initiatives to fight Mali IDA hunger and poverty 3 2018 Kyrgyz Republic: Village Investment Projects I and II Kyrgyz Republic; IDA World 4 2018 Sierra Leone: Integrated Public Financial Management Sierra Leone IDA Reform project 5 2018 Burkina Faso, Ghana and Mali: West Africa transport Western Africa IDA and transit facilitation project 6 2018 Lao People’s Democratic Republic: sustainable forestry Lao People's IDA for rural development project Democratic Republic 7 2018 Cambodia: trade facilitation and competitiveness Cambodia IDA 8 2018 Cameroon, Chad, Central African Republic, São Tomé Western Africa; IDA and Príncipe: internet and mobile connectivity: Central Central Africa African backbone program (APL 1A and APL 2) 9 2018 Mongolia: renewable energy for rural access project Mongolia Blend (REAP) 10 2018 Lao People’s Democratic Republic: trade development Lao People's IDA facility project Democratic Republic; World 11 2018 Central African Republic—Emergency Food Crisis Central African IDA Response and Agriculture Relaunch Project Republic 12 2018 Burkina Faso—First, Second, Third and Fourth Growth Burkina Faso IDA and Competitiveness Credits Projects 13 2018 Nicaragua: off-grid rural electrification (PERZA) project Nicaragua IDA 14 2018 Mali: rural community development project Mali IDA 15 2017 Ghana: agriculture development policy operations, Ghana IDA phase I—IV 16 2017 Lao People’s Democratic Republic: second education Lao People's IDA development project Democratic Republic 17 2017 Senegal: A Decade of World Bank Support to Senegal’s Senegal IDA Nutrition Program 18 2017 Nepal: second rural water supply and sanitation project Nepal IDA 19 2017 Nigeria: national water sector reform project Nigeria Blend 20 2017 Cambodia: demand for good governance project Cambodia IDA 21 2017 Kyrgyz Republic: rural education project Kyrgyz Republic IDA 22 2017 Nepal: poverty alleviation fund project Nepal IDA 71 Appendix H List of Independent Evaluation Group Evaluations Lending Delivery Category No. FY Document Name Country (FY19) 23 2017 Ethiopia, Mali, Morocco, South Africa, Tanzania, World; Africa; IDA/IBRD Tunisia—Africa Stockpiles Program Project Middle East and North Africa 24 2017 Senegal—Urban Mobility Improvement Project Senegal IDA 25 2017 Cambodia: Public Financial Management and Cambodia IDA Accountability Project 26 2017 Ghana: eGhana project Ghana IDA 27 2017 Solomon Islands: Rural Development Program Solomon Islands IDA 28 2017 Kenya: Development of the National Statistical System Kenya Blend Project 29 2017 Zambia: Water Sector Performance Improvement Zambia IDA Project 30 2017 Senegal: A Decade of World Bank Support to Senegal’s Senegal IDA Nutrition Program 31 2016 Tanzania—Local Government Support Project Tanzania IDA 32 2016 Malawi—Financial Management, Transparency, and Malawi; World IDA Accountability Project 33 2016 Mozambique—Market-Led Smallholder Development Mozambique IDA in the Zambezi Valley Project 34 2016 Ethiopia—First and Second Phase of the Pastoral Ethiopia IDA Community Development Project 35 2016 Uganda—Millennium Science Initiative Project Uganda IDA 36 2016 Ghana—Second Urban Environmental Sanitation and Ghana IDA Second Phase of Small Towns Water and Sanitation Projects 37 2016 Bangladesh—Social Investment Program Project Bangladesh; World IDA 38 2016 Malawi—Third Social Action Fund and Second Phase of Malawi IDA Adaptable Program Lending Project 39 2016 Ghana—Economic Governance and Poverty Credit, and Ghana IDA Seventh and Eighth Poverty Reduction Support Credits Projects 40 2016 Tanzania—Second Social Action Fund Project Tanzania IDA 41 2016 Niger—Institutional Strengthening and Health Sector Niger IDA Support Project 42 2016 Zambia—Public Sector Management Program Support Zambia IDA Project 43 2016 Ghana—Statistical Development Project Ghana IDA 44 2016 Lao People’s Democratic Republic—Poverty Reduction Lao People's IDA Fund Project Democratic Republic 72 Appendix H List of Independent Evaluation Group Evaluations Lending Delivery Category No. FY Document Name Country (FY19) 45 2016 Mozambique—Third, Fourth and Fifth Poverty Mozambique IDA Reduction Support Credit Project 46 2016 Bangladesh—Phase one of Rural Transport Bangladesh IDA Improvement Project 47 2016 Nigeria—Lagos Urban Transport Project Nigeria Blend Source: Independent Evaluation Group. Note: APL = adaptable program loan. 73