Results and Performance of the World Bank Group 2016 An Independent Evaluation Results and Performance of the World Bank Group 2016 An Independent Review © 2017 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Contents ABBREVIATIONS ................................................................................................................................. VI ACKNOWLEDGMENTS ........................................................................................................................ IX OVERVIEW............................................................................................................................................. X MANAGEMENT COMMENTS ............................................................................................................ XIV 1. MANAGING FOR RESULTS ..................................................................................................... 1 Why Address Managing for Results Now? ................................................................................................................ 1 What Is Managing for Development Results? ........................................................................................................... 2 Limitations of this Review .......................................................................................................................................... 3 M4R History—More than a Decade with Some Success .......................................................................................... 4 What Is the Evidence on the World Bank Group’s M4R?.......................................................................................... 7 The World Bank Group Support to Client M4R Capacity ........................................................................................ 16 Conclusions ............................................................................................................................................................. 22 2. RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS ............................... 25 World Bank Group Commitments ........................................................................................................................... 25 World Bank Project Performance ............................................................................................................................ 27 Trends in Outcome Ratings for IFC Investment and Advisory Projects .................................................................. 34 Trends in Outcome Ratings for MIGA Projects ....................................................................................................... 37 Country-Level Performance .................................................................................................................................... 38 3. MANAGEMENT ACTION RECORD ........................................................................................ 51 Tracking the Implementation of IEG Recommendations ......................................................................................... 51 Status of Bank Group Implementation of IEG Recommendations .......................................................................... 51 IEG Recommendations on M4R ............................................................................................................................. 54 Influence of IEG Recommendations ....................................................................................................................... 55 Using the MAR for Adaptive Management and Learning ........................................................................................ 56 Conclusion .............................................................................................................................................................. 56 Box Box 3.1. Using Impact Evaluations for Better Development Results: The Road Ahead ........................ 55 Figures Figure 1.1. Principles of Managing for Results ........................................................................................ 3 Figure 1.2. Operationalizing Principles of Managing for Results in World Bank Group Operations ...... 17 Figure 1.3. Monitoring Was the Most Common M4R Element Addressed for Clients ........................... 18 Figure 1.4. A Majority of Latin America Projects Addressed M4R Capacity .......................................... 18 Figure 2.1. World Bank Group Commitments Increased between FY13 and FY16 .............................. 26 Figure 2.2. Volume-Weighted Percentage of Projects Rated MS+ on Outcome and World Bank Performance Improved from FY10–12 to FY13–15............................................................................... 28 iii CONTENTS Figure 2.3. Notable Gains and Gaps Are Found in Outcome Ratings ...................................................30 Figure 2.4. World Bank Project Ratings Varied across Regions ............................................................31 Figure 2.5. Outcome Ratings Varied across Global Practices, with Increased Ratings Noted for Agriculture and Education......................................................................................................................32 Figure 2.6. Development Outcome Ratings for IFC Investment Projects Decreased from 2009 to 2015 ..............................................................................................................................................................34 Figure 2.7. Development Outcome Ratings for IFC Investment Projects Varied among Regions; Sub- Saharan Africa Development Results Trended Up ................................................................................35 Figure 2.8. Development Outcome Ratings for IFC Investment Projects Varied across Industries; FIG Performance Was Up but CTT Dropped Below 40 Percent ...................................................................35 Figure 2.9. MIGA Development Outcome Ratings for MIGA Projects Remained Steady over a Six-Year Period ....................................................................................................................................................38 Figure 2.10. Country Development Outcome Ratings Exceed Target in IBRD Countries, but Not in IDA and FCV Countries ................................................................................................................................39 Figure 2.11. Country Strategy Development Outcome Ratings in Three Regions Exceeded the Target during FY07–16 .....................................................................................................................................40 Figure 2.12. Countries with More Than One Country Strategy during FY07 –16 Show Varied Pattern of Change in Country Development Outcome Rating ................................................................................41 Figure 2.13. Percentage of Country Strategies with World Bank Group Performance Rated MS+/Good+ Varied ....................................................................................................................................................42 Figure 2.14. In Most Regions, the Percentage of CLRs with World Bank Performance Rated Good+ or MS+ Was Below Target in FY13-16 ......................................................................................................42 Figure 2.15. Thematic Objectives and Their Achievement Ratings, by CPIA ........................................45 Figure 3.1. Management Self-Ratings of Satisfactory Implementation Were Consistently Higher Than IEG Ratings between FY12 and FY16 ...................................................................................................52 Figure 3.2. The Most Common IEG M4R-Related Recommendation Addressed Monitoring and Evaluation ..............................................................................................................................................52 Figure 3.3 Recommendations on M&E Had the Largest Gap between IEG and Management in Year Four .......................................................................................................................................................53 Tables Table 2.1. Number of Countries with CPFs and CASs Reviewed by IEG, FY07 –16 .............................39 Table 2.2. Country Development Outcome Ratings, Project Outcome Ratings, and Project Risk Ratings, FY07–16 ..................................................................................................................................44 Appendixes APPENDIX A. IEG EVALUATIONS AND LEARNING PRODUCTS FY15–17 AND FINDINGS RELATED TO M4R ............................................................................................................................... 59 APPENDIX B. METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW ............................. 67 APPENDIX C. ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO ................................. 79 iv CONTENTS APPENDIX D. METHODOLOGY FOR REVIEW OF WORLD BANK PROJECTS WITH BELOW-THE- LINE RATINGS ..................................................................................................................................... 85 APPENDIX E. ANALYSIS OF EXTENT TO WHICH GENDER IS INCORPORATED IN PROJECT DEVELOPMENT OBJECTIVES, COMPONENTS, AND INDICATORS ............................................... 88 APPENDIX F. ADDITIONAL INFORMATION ON THE IFC PORTFOLIO ........................................... 92 APPENDIX G. ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS............................................................................................... 96 APPENDIX H. ADDITIONAL ANALYSIS OF COUNTRY COMPLETION AND LEARNING REVIEW DATA .................................................................................................................................................. 101 APPENDIX I. REGIONAL RESULTS AND PERFORMANCE ............................................................ 106 APPENDIX J. IEG NOTE ON GLOBAL PRACTICES ........................................................................ 206 BIBLIOGRAPHY................................................................................................................................. 262 v Abbreviations BDA Bangsamoro Development Agency (the Philippines) CAS country assistance strategy CASCR Country Assistance Strategy Completion Report CASPR Country Assistance Strategy Progress Report CLR completion and learning review CMU country management unit CPE country program evaluation CPF country partnership framework CPIA Country Policy and Institutional Assessment CPS country partnership strategy CRR credit risk rating CTT Telecom, Media, Technology, and Venture Investing CY calendar year DIME Development Impact Evaluation DOTS Development Outcome Tracking System DPF development policy financing DPL development policy loan DPO development policy operation E&S environmental and social ECD early childhood development EFI Equitable Growth, Finance, and Institutions EGPRC economic governance and poverty reduction credit ESAP Environmental and Social Action Plans EvNote Evaluative Note FCV fragility, conflict, and violence FIG Financial Institutions Group FMIS financial management information system FY fiscal year GAO Government Accountability Office GDP gross domestic product HFC housing finance company IAD International Association for Development IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report ICRR Implementation Completion and Results Report Review ICT information and communication technology IDA International Development Association vi ABBREVIATIONS IDA18 18th Replenishment of IDA IDG IFC Development Goal IEG Independent Evaluation Group IFC International Finance Corporation IICCR institutional investor country credit risk IMF International Monetary Fund INR Infrastructure and Natural Resources IPF investment project financing ISN interim strategy note LEAP Livelihood Empowerment against Poverty M&E monitoring and evaluation M4R Managing for Results MAR Management Action Record MAS Manufacturing, Agribusiness, and Services MIGA Multilateral Investment Guarantee Agency MOPAN Multilateral Organisation Performance Assessment Network MS moderately satisfactory (rating) MS+ moderately satisfactory or better (rating) MU moderately unsatisfactory (rating) NOP No Opinion Possible OECD Organisation for Economic Co-operation and Development OECS Organisation of Eastern Caribbean States OPCS Operations Policy and Country Services PAD project appraisal document PBG policy-based guarantee PCR project completion report PDO project development objective PforR Program-for-Results PPAR project performance assessment report PPP public-private partnership PRI political risk insurance PRSC poverty reduction support credit PSIA poverty and social impact assessment RAP Results and Performance of the World Bank Group RETF recipient-executed trust fund RMS results measurement system SCD systematic country diagnostic SDG Sustainable Development Goal SHF Sociedad Hipotecaria Federal SME small and medium enterprises vii ABBREVIATIONS SORT Systematic Operational Risk-Rating Tool UNDP United Nations Development Programme XPSR Expanded Project Supervision Report All dollar amounts are U.S. dollars unless otherwise indicated. viii Acknowledgments This report was prepared by a team led by Nidhi Khattri and Raghavan Narayanan. It was written under the guidance of Auguste Tano Kouame (Director), Nick York (former Director), and Marie Gaarder (Manager), as well as the overall direction of Caroline Heider (Director-General, Evaluation). The team comprised Joy Behrens, Nils Fostvedt, Surajit Goswami, Katsumasa Hamaguchi, Houqi Hong, Ursula Martinez, Santhadevi Meenakshy, Xiaoxiao Peng, Jakob Rusinek, Bahar Salimova, Anthony Martin Tyrrell, Yoshine Uchimura, and Abby Bo Wan. Additional analytic support was provided by Ana Belen Barbeito, Leonardo Bravo, Unurjargal Demberel, Hiroyuki Hatashima, Edwin Hernandez-Ayala, Yunsun Li, and Amanda Liu. Data entry was carried out by Noelle Pierce and Yezena Yimer. Brenda Barbour supervised the analyses related to the Management Action Record, which was led by Bahar Salimova and Yoshine Uchimura. Jakob Rusinek, Stephanie Evergreen, and Francis Gagnon provided ideas and support with tables, figures, and communication. Anthony Tyrrell led the regional and global practice updates, with inputs on regional updates from Juan-Jose Fernandez-Ansola, Nils Fostvedt, and Surajit Goswami. The global practice updates benefited from inputs from IEG staff including Arun Arya, Elena Bardasi, Erik A. Bloom, Zeljko Bogetic, Leonardo Bravo, Susan Ann Caceres, Ann Elizabeth Flanagan, Gisela M. Garcia, Antonio Giuffrida, Stephen Hutton, Beata Lenard, Lauren Kelly, Moritz Piatti, Bekele Ambay Shiferaw, Jeffery Tanner, Victor Vergara, and Fang Xu. IEG would also like to thank Judith Lewetchou Efouefack, Lauren Kelly, and Gurkan Kuntasal for timely and insightful work on assessing World Bank Group Performance Standards and safeguards in projects that closed in FY15, and Judith Hahn Gaubatz for supporting analysis related to gender projects and project ratings disconnects, also for projects closed in FY15. This report benefited from peer review comments provided by Oscar Garcia (Head, Evaluation, IFAD), Karen Jorgensen (Head, Division of Peer Review and Evaluation, OECD), and Margaret Kakande (Head, Budget Monitoring and Accountability, Ministry of Finance, Uganda). Administrative and editorial support was provided by Marie Charles and William Hurlbut, respectively. ix Overview Several developments during the past three Strengthening M4R years led to a renewed and qualitatively different emphasis on results within the The World Bank Group has a long history World Bank Group. In 2013 the World of M4R and has made good progress on Bank Group established a new strategy creating a structure and process for results focused on the twin goals of ending extreme measurement. It has also launched reforms poverty and promoting shared prosperity. and several initiatives throughout the years In 2016 the International Development to enhance evidence-based management and Association (IDA) received the largest learning. IEG evidence shows progress but replenishment in its history, with a greater indicates that strengthening the foundations emphasis on results. More recently, Forward of results measurement and instilling a Look: A Vision for the World Bank Group in culture of evidence-based adaptive 2030 detailed the organization’s role in management and learning needs a stronger, global economic development and more systematic, and holistic push. emphasized an innovation-learning-scaling approach and strengthening evaluation. The collective challenges of data availability, More broadly, as the Millennium an inconsistent focus on beneficiary-level Development Goals ended in 2015, the outcomes, and the overall quality of results development community committed to the frameworks impede results measurement. more ambitious Sustainable Development This situation warrants attention. While Goals. improved under the new country engagement model, country strategies need This year’s Results and Performance of the World to enhance links between interventions and Bank Group review report emphasizes the desired outcomes and incorporate indicators theme of managing for development results that reflect those outcomes. In private (M4R). The first chapter reviews the World sector interventions, limited evidence of Bank Group’s status regarding M4R and beneficiary outcomes hinders the summarizes the organization’s approach to articulation and aggregation of results across and degree of success with integrating the World Bank Group. International M4R’s key principles—measurement of Finance Corporation (IFC) investment outcomes and use of evidence for adaptive projects and Multilateral Investment management and learning—to drive results. Guarantee Agency (MIGA) guarantee The report draws on cumulative evidence projects have little or no distinction in from the Independent Evaluation Group monitoring and evaluation (M&E) (IEG) evaluations and learning products to frameworks for repeat clients compared highlight the World Bank Group progress with new clients. This represents a missed and scope for improvement. The report also opportunity to develop enhanced results explores the World Bank Group’s frameworks and potentially capture engagement on developing client M4R stakeholder and beneficiary results more capacity. The second chapter reviews the systematically. performance of the World Bank Group’s operations, and the third chapter is an Empirical testing of the validity of a project update on the Management Action Record or program change theory—to learn about (MAR). what factors might affect outcomes— x OVERVIEW should be further strengthened in World capacity through its operations. The Bank projects. Evidence and data are emphasis in lending operations on client fundamental inputs to making informed capacity for data systems, monitoring, and judgments that underpin adaptive evaluation should be lauded for building the management and learning. Results need not foundation for evidence-based decision be expected to be driven uniquely by the making. The World Bank Group focused World Bank Group’s investments, but comparatively less on helping clients managing for development results requires strengthen capacity for their own use of an expected level of verifiable contribution. data and evidence for adaptive management Learning within the World Bank Group is and learning. These findings are preliminary, equally important and was missing when and the effectiveness of these project-level introducing new products or instruments (for efforts is not assessed for this review. At the example, MIGA non-honoring guarantees country level, about half of 40 country and IFC subnational investment projects) or strategies since FY14 had an M4R objective in sequenced and joint projects. This set of related to evidence-based decision making. challenges undermines evidence-based However, IEG rated less than 40 percent of adaptive management and learning, those objectives as achieved or mostly potentially at the corporate level as well. achieved. Weak client capacity is a constraint to M4R. The World Bank Group is a Indicators in the World Bank Corporate demand-driven organization, but it will need Scorecard and at other levels provide a strategic approach to address client M4R portfolio information, but the question of capacity, based on systematic diagnostics. whether the volume of indicators makes sense and whether they support appropriate On balance, the World Bank Group has adaptive management and learning at made good progress on building a different organizational levels needs review. measurement system, though fundamental Corporate results reporting that implies that measurement issues remain. It also needs to only achievements are important can distort pay greater attention to adaptive incentives and lead staff to favor reporting management and learning based on robust positive outcomes (even though data might evidence. A World Bank Group M4R plan not be robust) or to become excessively risk that addresses both M4R principles averse. systematically and holistically at different levels could provide a useful benchmark for There continues to be an organizational future M4R and its outcomes. need for matching staff capacity to the requirements for different M4R aspects and World Bank Group Performance for strengthening staff skills, competencies, and career paths in areas with gaps. M4R- The World Bank’s project outcome ratings related corporate initiatives, reforms, and remained essentially stable for projects actions underway need to be assessed on closed during the FY13–15 period. The their contribution to evidence-based share of projects with an outcome rating of adaptive management, learning, and, moderately satisfactory or above (MS+) was ultimately, results. 72 percent for FY13–15 and 70 percent for the FY10–12 period, both below the The exploratory desk-based review of corporate target of 75 percent by FY17. World Bank Group support for building Weighted by volume, the share of MS+ client M4R capacity report found that the projects rose from 81 percent for FY10–12 World Bank Group mainstreams client to 87 percent in FY13–15, above the FY17 xi OVERVIEW corporate target of 80 percent. The MIGA development outcomes (six-year performance of a few projects with very rolling average, excluding projects rated No large volume drove the increase in the Opinion Possible from the total) remained volume-weighted percent MS+. However, nearly steady at 61 percent compared with as shown in the Results and Performance of the prior review period. Seven projects were World Bank Group 2015, project size itself rated No Opinion Possible for outcomes in does not lead to higher ratings. Performance this review period. The best performing of projects in South Asia (82 percent rated sectors were agribusiness, manufacturing MS+) was higher than that in other World and services. Middle East and North Africa Bank regions in FY13–15, and among was the best performing region in this global practices, Social Protection and review period. Labor again ranked the highest with 90 percent of projects receiving MS+ outcome IEG rated 66 percent of country programs ratings during FY13–15. A review of MS+ for country strategy development implementation completion report reviews outcomes during FY13–16, below the FY17 shows that more than half of the projects corporate target of 70 percent. The reasons receiving an outcome rating of moderately for below-par performance relate to poor unsatisfactory or below in FY15 had results frameworks and ambitiousness of inadequate evidence to substantiate the objectives not commensurately underpinned outcome. by operations and support for country capacity. The development outcome rating Outcome ratings for IFC investment improved from unsatisfactory to satisfactory projects continued to trend downward from for 17 countries over a period of two prior review period—project success rates country strategies in five of the six World were down by 3 percentage points to 54 Bank regions (except for East Asia and percent for 229 projects evaluated in 2013- Pacific). Conversely, the development 2015 period. Latin America performed the outcome rating for 12 countries—six in best on development outcome and IFC Africa, five in Europe and Central Asia, and work quality ratings. Profitability, measured one in Latin America and the Caribbean— by investment outcome ratings, improved in declined from satisfactory to unsatisfactory, four of six regions. Development outcomes and for eight countries in Africa, it stayed at of real sector projects dropped by 5 the unsatisfactory level. Reasons for poor percentage points, but projects in the performance during two strategy cycles financial sector did better compared with include overambitious strategies with too the previous year (up by 1 percentage many objectives or objectives that are too point). Greenfield projects were generally ambitious relative to the country’s less successful than expansion projects. implementation capacity and available Development outcome ratings of projects resources. However, these are with debt and equity were higher than loan- postcompletion observations, and going only or equity-only projects. Large projects forward ex-ante deeper assessments are (investment size > $35 million) performed warranted, together with robust better (at 67 percent MS+) compared with performance and learning reviews. small and medium-size investment projects at 35 and 52 percent MS+, respectively. In Country capacity, as measured by Country advisory business, performance dropped Policy and Institutional Assessment (CPIA), from 65 percent MS+ last year to 61 is also significantly and positively related to percent, and was below the scorecard target both country-level outcomes and project of 65 percent. outcomes. IEG’s review finds that countries xii OVERVIEW with lower CPIA scores tend to achieve a implementation ratings is with the lower percentage of their country strategy recommendations focused on M&E. IEG objectives compared with countries with has been reporting this recurring trend for higher CPIA scores. But the achievement several years through the MAR updates. gap varies across themes. Achievement of objectives related to economic policy, Among the 277 recommendations that were finance, and environment and natural produced since 2008, about 34 percent have resource management was relatively lower in one or more M4R dimensions. IEG rated countries with lower CPIA ratings, but 68 percent of these recommendations as achievement of objectives related to human substantial, high, or completed. These development and gender was higher. Public recommendations include a large proportion sector management–related objectives were (62 percent) of M&E-related especially difficult to achieve, regardless of recommendations, which contributes to the CPIA. lower progress on M4R recommendations. Management Action Update The Management Action Record (MAR) tracks the actions of the Management of the World Bank Group institutions in response to IEG recommendations in corporate, sector, and thematic evaluations. The MAR tracks only recommendations that IEG and management agree or partially agree on. Management creates an action plan for the accepted recommendations, and the World Bank Group and IEG assess the implementation of each recommendation for four years, after which a recommendation is retired from the MAR. IEG and the World Bank Group tracked actions for 150 recommendations across all three World Bank Group institutions in 2016, drawn from the 26 evaluations produced between FY12 and FY15. About 32 percent of the IEG recommendations active in this year’s update focus on capacity building, 28 percent on M&E, and 18 percent on the World Bank Group’s strategic directions and approaches. For evaluations in the fourth year of review, IEG rates this year’s implementation progress as substantial for 72 percent of the recommendations. This year’s implementation ratings are lower than those IEG has provided in the past three years. The largest disconnect in the xiii Management Comments The management of the World Bank Group institutions welcomes the report of the Independent Evaluation Group (IEG), Results and Performance of the World Bank Group 2016 (RAP), and the opportunity to respond with comments. The report presents useful analysis and insights on the World Bank Group’s status and progress in three areas: (i) the institution’s approach to and success with integrating the key principles of managing for results (M4R); (ii) the recent performance of World Bank Group operations; and (iii) the implementation status of the Management Action Record (MAR). Management is pleased that the report recognizes recent positive trends, and it notes the challenges in the results and performance of World Bank Group operations that the report highlights. Management finds that this year’s thematic focus, M4R, is timely and relevant in the context of the rollout of the new Forward Look and the record commitment made for the 18th Replenishment of the International Development Association as the global development community works to achieve the Sustainable Development Goals (SDGs). World Bank Management Comments MANAGING FOR RESULTS Management appreciates the report’s recognition that the World Bank Group’s past efforts have “clearly led to positive changes” in managing for and reporting on development results, and that the institution is considered to be “at the forefront” in emphasizing the importance of M4R. Management also appreciates the effort made in the report to document some of initiatives the World Bank Group has taken and its achievements in integrating the key principles of M4R in its operations. At the same time, management acknowledges that there is a need to continue efforts to strengthen M4R within the World Bank, especially evidence-based adaptive management and learning, as well as to enhance client capacity for M4R, including in the areas of data availability at the country level and the use of evidence for policy making. The following paragraphs provide some specific comments on each aspect of M4R analyzed in the report. Measuring Development Results. The report finds that the World Bank has made important progress in measuring development results by improving structures and systems for measurement and placing greater emphasis on gathering data on results. Management recognizes the need to continue focusing on the overall quality of xiv MANAGEMENT COMMENTS results frameworks, in particular to strengthen how outcomes are measured. Management fully agrees with the report’s view that the development results of operations or country strategies should be outcomes rather than outputs. At the same time, management believes that there are multiple levels of outcomes within a results chain, and that World Bank-financed operations (and country strategies) need to target the levels of outcomes that are realistic and reasonably attributable to the interventions, given their duration, resources, and context. While it would be ideal to measure outcomes as changes at the ultimate beneficiary level (individuals, households, and firms, as cited in the report), this may not always be feasible or appropriate for every project or intervention. For instance, support for decentralization of service delivery might begin with an operation that focuses on setting up local government structures, personnel, and systems, and putting in place a mechanism for fiscal transfers. In countries with weak capacity, particularly in fragile or post-conflict situations, this activity alone might take the lifetime of the project to accomplish. To hold this project accountable for improved services or welfare at the individual or household level might be more than the project could be reasonably expected to achieve, and a more appropriate outcome indicator might be the impact on local government capacity (e.g., functioning local government structures with qualified personnel; functioning systems for planning, budgeting, and funds flow). In stronger countries, a similar project might be able to achieve improved services or greater well-being of ultimate beneficiaries within the lifetime of the project, and for such countries, individual/household access to quality services might be the right outcome indicators. Management has discussed this topic with IEG on various occasions, and is eager to continue to work with IEG and to provide better guidance to task teams on setting outcome targets at the appropriate levels. Related to this point, the report notes “managing for results requires some expected level of verifiable contribution.” It is important to recognize that outcomes at the highest levels in the results chain (e.g., reduced poverty, increased incomes, reduced mortality, lowered unemployment) are the ones that are least attributable to specific project interventions, as they are usually influenced by myriad factors. The further up the results chain the outcomes are, the harder it becomes to show a “verifiable” contribution of the project to them. This measurement challenge is exacerbated by the fact that there may be a time lag in achieving higher-level outcomes (e.g., incomes might increase a few years after a road has been completed). Thus, it is important to set outcomes at levels that are reasonably attributable, verifiable, and feasible given the duration, resources, and approach of the intervention. It is also important, as the report pointed out, to analyze diverse factors that might have affected the project outcomes, and to enhance projects’ monitoring and evaluation xv MANAGEMENT COMMENTS (M&E) design to assess projects’ efficacy. The World Bank has several efforts under way to support teams in thinking through results chains, outcomes, and issues of attribution versus contribution more carefully from the design through the Implementation Completion and Results Report (ICR) stage (including the ICR Reform and Agile Pilots). Adaptive Management and Learning to Achieve Results. Management acknowledges that more work needs to be done on incorporating lessons and evidence into project design, making course corrections during implementation, and encouraging innovation and risk-taking. Over the years, the World Bank has introduced a number of initiatives to promote adaptive learning, including adaptable program lending instruments (Adaptable Program Loans and Learning and Innovation Loans), Additional Financing, and simplified processes for project restructuring. While some progress has been made on flexibility, management recognizes the need for further work on more effectively closing feedback loops and adopting more evidence-based approaches. With regard to adaptive management at the corporate level, the report suggests that the Corporate Scorecard (CSC), specifically the Tier II indicators, should be used more extensively to provide strategic direction to management. Tier II of the CSC is intended to provide a snapshot of the kinds of results the World Bank has achieved at a specific point in time. The indicators were selected to reflect a balance across different sector or thematic areas as well as the feasibility of aggregating across projects. Thus, most indicators are output-oriented, and they capture results in a limited number of areas and sectors. It would therefore be misleading to use the CSC’s Tier II results as a tool for adaptive management to set strategic directions for the World Bank, at either the corporate or the sectoral level. Supporting Client M4R Capacity. Management appreciates the report’s exploratory analysis of the World Bank’s support for M4R capacity. The report documents the products and instruments the World Bank is using to support M4R capacity in client countries and provides a useful benchmark to monitor efforts going forward. This is a critical area that needs continued attention and effort. At the same time, the challenges of supporting client capacity in M4R cannot be underestimated, as investment in M4R approaches, such as data systems or results-based budgeting and financing, will require sustained effort and resources and are heavily dependent on the client’s political will and prioritization of the use of scarce resources. xvi MANAGEMENT COMMENTS RECENT RESULTS AND PERFORMANCE OF WORLD BANK OPERATIONS Project Performance. Management is pleased to see an overall upward trend in project performance in terms of IEG’s outcome ratings and a similar trend in ratings of Bank performance. While outcome ratings by volume have exceeded targets, management recognizes the need to continue to strengthen results at the individual project level. Country-Level Performance. Management appreciates the detailed analysis of country development outcomes, including the regression analysis showing the positive correlation of country development outcomes with the Country Policy and Institutional Assessment (CPIA) and with portfolio outcome ratings. At the same time, management acknowledges that the results frameworks and realism of country strategies need to be further strengthened, and takes note of the report’s suggestion that the performance and learning reviews of country partnership frameworks (CPFs) could be used more deliberately to review results chains and CPF outcomes. MANAGEMENT ACTION RECORD Management appreciates the analysis of the MAR process and implementation progress. Management notes that about a third of the recommendations adopted since 2008 had an M4R dimension, and that progress has been slow in implementing recommendations related particularly to capacity building and M&E. Management acknowledges the critical role of capacity building and M&E on development results and will continue strengthening efforts to address these issues. The MAR reform of 2011 established a clear and transparent mechanism to follow up on IEG’s recommendations, and the MAR has become an important accountability tool. After more than five years of implementation, management feels that a review of the MAR process and implementation experience may be warranted to improve its effectiveness and to make it more adaptable to changing circumstances. In that context, management is looking forward to carrying out the pilots that have been agreed following the IEG External Review. Going forward, management would like to take stock of the lessons from the pilots and assess the effectiveness of the MAR process in influencing positive change in the World Bank Group. IFC Management Comments Management appreciates IEG’s review and analyses detailed in the World Bank Group Results and Performance 2016. It commends IEG for highlighting the theme of M4R as this edition’s focus given the increased attention to development results by xvii MANAGEMENT COMMENTS the International Finance Corporation (IFC), including through the new Economics and Private Sector Development Vice Presidency. IFC management and staff appreciate the IEG team’s effort to understand the facts and viewpoints that IFC shared, and to incorporate them where possible. Importance of Quality Results Measurement Frameworks. IFC management fully agrees with the importance of having quality results measurement frameworks for effective, evidence-based adaptive management and learning, which would ultimately lead to strong results and impact. Management is therefore pleased with the recognition that the World Bank Group has made good progress on creating a structure and process for results measurement, and that it has carried out reforms and several initiatives to enhance evidence-based management and learning. IFC’s efforts for M4R have come a long way since the establishment of OEU (the first IFC evaluation unit and part of today’s IEG) in 1987, the implementation of Expanded Project Supervision Reports (XPSRs) in 1994 and the Development Outcome Tracking System (DOTS) in 2005. IFC’s current Results, Monitoring, and Evaluation system is considered as best practice among private sector oriented development finance institutions. The framework, approved and monitored by the Committee on Development Effectiveness since 2014, aims to strike the complex balance between quality, practicality, and reality. Under the leadership of the new Vice President, Economics and Private Sector Development, IFC is further strengthening its results framework, particularly with respect to ex-ante analyses. In this regard, management takes note of the IEG concern that the World Bank Group needs a stronger, more systematic, and holistic push to build a foundation and culture for evidence-based adaptive management and learning. Beneficiary-Level Results. Verifiable, auditable data that are, systematically available at the beneficiary level have been the standard that both IFC and IEG use for evaluation and validation. As the report notes, DOTS data benefit from an external assurance provider. With respect to the suggestion that repeat clients could be a starting point to explore a new approach that would aim to capture incremental, verifiable, and auditable stakeholder and beneficiary results more systematically, management appreciates the suggestion and would like to explore while keeping realistic parameters in place. Adaptive Management. As the report states, IFC continues to focus on adaptive learning and management. IFC has over 40 lesson databases, including one mentioned in the report (LessonFinder, managed by IEG). IFC’s Global Knowledge and Learning Office is consolidating them for more effective and efficient staff access. In addition, IFC has mainstreamed corporate programs to promote learning, especially since the 2014 survey undertaken by IEG. These programs, include: xviii MANAGEMENT COMMENTS Knowbel Award, the annual Knovember Lessons Sharing event, SmartLessons, and technical master classes. These are complemented by other ongoing efforts, such as the design of a new lessons’ framework and the mainstreaming of after-action reviews to capture project lessons and recommendations for improvements. In addition to Special Operations, all investment departments and some corporate departments, such as the Credit Department and the Environment, Social, and Governance Department, organize annual Knowledge and Learning weeks with a focus on sharing lessons learned. Many IFC departments have knowledge management staff dedicated to such lessons activities as MAS Talks, FIG Knowledge Bytes, and Infra PostVivems—all of these are described in the IEG–IFC joint internal report “Lessons in IFC” issued last year. Volume Targets. The report notes that according to an FY16 survey, IFC staff considered volume targets and project profitability to be the most important metrics for management. Management believes that this finding mirrors two of our business priorities but does not necessarily preclude other priorities, such as development impact and client satisfaction. In addition, while volume targets remain critical, IFC is reducing the importance of volume targets for equity investments, focusing on quality deals and quasi-equity products. This will also help reduce pressures on the deployable strategic capital created due to capital constraints. With respect to adaptive management, management provides signals and incentives for lessons focused on diverse aspects of our business, including project profitability, development impact, and client engagement. Feedback Loop to New Business. Management agrees that results and knowledge gained from M4R activities should be looped back to future operations. Management encourages lesson learning in all available platforms. With the various initiatives mentioned above underway, we believe that IFC is on the right trajectory. Self-evaluations by operational teams and IFC management aims to find constructive ways to better utilize XPSRs for lessons and knowledge. In relation to the formulation of management’s action plan in response to the Report on Self- Evaluation Systems (ROSES), one contemplated scenario is to intentionally separate or sequence the assessment related to accountability and learning in an investment evaluation. Additional mechanisms to systematically require lessons identification (similar to suggestions in the report) have also been under consideration, including mentioning past lessons in the Concept Review Note discussing the relevant lessons as part of the Immediate Response Mechanism. Additional Advisory-Specific Concerns. IFC management reiterates the point that it had shared with IEG earlier that the client capacity support analysis of the M4R review would have been more meaningful if the criteria or approach had been xix MANAGEMENT COMMENTS chosen more carefully and inclusively, factoring in the nature of various business areas. The chosen approach, focusing on clients’ enhanced management information, data, risk control, and financial reporting systems, inevitably excludes IFC Advisory Services work other than with financial institutions. As a result, the report found the most meaningful M4R for IFC Advisory Services to be in microfinance and insurance, where these are naturally the types of solutions that may be recommended to the client. However, it would be important to recognize that six other IFC Advisory Services Business Areas also contributed to strengthening M4R, for instance, by improving the understanding of issues and challenges and by prompting implementation of potential solutions. Management notes the concern raised about IFC Advisory Services not consistently adjusting project log frames to reflect changes in project circumstances, thus preventing the monitoring of accurate outcome achievement. IFC management wishes to clarify that the governance system, which is uniform across all Advisory Services and includes the controls for log frame changes, was implemented several years ago, benefitting from IEG’s strong inputs. IFC will aim to more consistently update the log frames now that IEG has clearly confirmed that it would rate the outcome against the adjusted framework, and not the original set at the time of approval. Clarifications. IFC management wishes to clarify two things that may not be clear in the report. First, regarding the statement that IEG finds that the investment and advisory M&E indicators are not always tracked consistently, management reiterates that it tracks a series of standard M&E indicators systematically and consistently through project maturity. This tracking is undertaken once a year for investments through DOTS and twice a year for advisory projects through the Advisory Services Operational Portal. IFC suggests that IEG contact IFC about its data validation process. Second, regarding the statement that the absolute numbers in the reach indicators are not reflected in relative terms for country or regional context, IFC’s practice is to present project-level reach indicators against the targets agreed with IFC management. Data users can apply the absolute numbers for various analytic purposes, such as country, region, instrument sector, and theme. Development Results. IFC management acknowledges that IFC’s investment development outcome success rate for the cohort CY13–15 was 54 percent on an unweighted basis, as measured by the binary share of positively rated investments in the evaluated sample. According to the IEG data posted on the intranet tableau, the success rate on a weighted basis was 65 percent.1 In both, the trend is 1 Ninety-nine percent CY15 validation completion rate footnoted in figure 2.6. xx MANAGEMENT COMMENTS downward, which also agrees with the recent trend in DOTS. The report attributes this primarily to the decline in IFC work quality (defined as appraisal, structuring, screening and supervision) and net commitment (project size) of projects. IFC management has been monitoring the downward trend in the past several years, particularly this year since the last RAP Board discussion. The three-year rolling sample (CY13-15) includes investment projects approved and executed since CY08, primarily during the major financial crises. During the years that followed the global financial crisis, emerging markets were hit by one of the largest slowdowns in recent history, as well as by volatility, currency depreciation, low commodity prices, and local and regional instability. These external factors were important in the declining performance. The IEG-IFC joint study (discussed below) revealed that the effects of the difficult business environment on IFC’s development performance were likely to have been compounded by IFC’s internal changes which, however, were being implemented with the goal of increasing IFC’s impact in the long run. As for the issue of net commitment to projects being the reason for the declining investment development outcome, management understands that this points to the smaller size of the net commitments in the cohort. This appears to be in line with IFC’s general approach to support smaller investments strategically through partnering with financial institutions. IFC takes note of IEG’s analysis. IEG-IFC Joint Study on IFC Investment Development Outcomes. Immediately after the last RAP Board discussion, during which IFC management committed to a close examination of IFC investments’ declining outcomes, IFC undertook a joint study with IEG on the subject with a focus on work quality. The study found that both external and internal factors were responsible for the trend over the last few years, in line with IEG’s analysis presented last year. While it was clear to IEG and IFC that external factors, notably the global financial crisis, had affected the outcomes of the investment cohorts in CY12–14 XPSRs, the study purposely focused on IFC’s internal factors—how IFC developed, screened, structured, processed, and managed projects in the current operating environment. The study broadly confirmed the IEG observation that shortcomings in work quality ultimately aggravated impacts from the difficult market environment. Internal factors, including an ambitious IFC growth strategy with higher risk taking, combined with a significant decentralization process, seemed to have led to reduced work quality, thereby compromising IFC’s project selection, analyses and supervision. To help reverse this trend the study recommended actions in the areas of incentives, organizational alignment, staffing and human resources, knowledge access, and macro assessments. Management is pleased to report that in conjunction with the corporate Diagnostics and reorganization effected on January 1, 2017, some of the recommended actions—such as assigning the same investment team throughout the xxi MANAGEMENT COMMENTS project cycle, strengthening the accountability structure, and clarifying roles in investment teams—are already underway. In the context of the IFC 3.0 strategy, the newly established Economics and Private Sector Development Vice Presidential Unit will strengthen macro, market, and country assessments for project selection and design. Management will carefully work with staff for high quality work in investment executions toward a rebound in development performance in the near future. Management plans to share the study with the Board after an action plan is internally approved. Telecom, Media, Technology, and Venture Capital and Funds Department. With respect to the development outcome success rate and the analysis, management wishes to make three points. First, because of the nature of the venture capital business, the approach to development performance success is inevitably different from that in more traditional businesses. Most projects are expected not to perform well, but there should be a few significant successful projects. IFC’s portfolio reviews have demonstrated this. Second, the observation that post-crisis support for “first-time fund managers” was one of the key reasons for the low success rate is not aligned with IFC’s experience. Third, regarding the funds business, management is concerned that a wide gap has been prominent between the development performance measured by XPSR cohorts and DOTS, especially in recent years. While management is aware of framework differences, the gap is much wider than that in other industries. In this regard, a closer examination of the sample would be helpful. Infrastructure and Natural Resources Department. The underlying data show that the share of investments in the Infrastructure and Natural Resources cohort was large at 85 percent, with high quality work—strong appraisal, screening and structuring (at 73 percent) as recognized in the report, and also by supervision and administration (at 88 percent)—as well as a strong role and contribution by IFC (at 85 percent). While IFC’s overall development performance was 65 percent (on a non- weighted basis), the share of positively rated investment outcomes increased two years consecutively to 73 percent, and 85 percent of the cohort was rated positively in environmental and social effects for the second year. In addition, IFC notes the report finding that project design, structuring and adverse external factors, such as geopolitical instability, regulatory changes and commodity price fluctuations led to less-than-satisfactory development outcomes. Management also notes the recommendation for teams to consider ex-ante downstream assessments and factor in conservative estimates on business growth and economic returns in project structuring. Advisory Services. Management notes the 61 percent development effectiveness success, noting that not all FY15 completed projects were rated. Most notable dips xxii MANAGEMENT COMMENTS are stated in the report to have occurred outside of the development effectiveness metric, namely in the areas of IFC’s role and contribution and of design quality— metrics that were introduced after January 2015. Small Samples. Management would appreciate that results calculated with small number of projects (for example, less than five) not be included in the report’s analysis. This is particularly important in the tables of the Regional Annexes and Advisory Service Business Areas. Change in Development Outcome Success Rate Calculation in Relation to No Opinion Possible. Management would appreciate IEG’s consideration in changing the methodology used in the report’s analysis to exclude No Opinion Possible projects from the denominator in the calculation of development outcomes. MIGA Management Comments MIGA Development Results. The report notes that the development outcome success rates of the guarantee projects of the Multilateral Investment Guarantee Agency (MIGA) remained steady at around 61 percent. MIGA notes the important lesson identified in IEG’s evaluation of the MIGA guarantee to a Tunisian state- owned shipping company: that MIGA’s non-honoring guarantees can have a valuable countercyclical role in helping projects and companies with sound fundamentals to access financing that would otherwise be too costly or out of reach. Feedback Loops. MIGA welcomes the report’s recognition of the progress of adaptive management specific to MIGA. As the report notes, in recent years MIGA has consistently used its self-evaluations and validations mechanism to identify lessons, and has used the knowledge gained in its new business reviews through a combination of information systems and knowledge events. In particular, MIGA notes from IEG’s 2016 Report on Self-Evaluation Systems that MIGA’s seminar series on Learning from Evaluations has been identified as good practice within the World Bank Group for learning from evaluation. Client Capacity Development. While the report states that MIGA has a limited role in client capacity development, it also notes that MIGA provides active guidance and monitoring for strengthening environmental and social performance results. MIGA notes the example cited in the report of an oil and gas sector project in Uzbekistan, where the MIGA team helped solve critical environmental and social issues by convening external industry experts. Political Risk Insurance Industry Development. MIGA welcomes the report’s recognition of MIGA’s important role in shaping the political risk insurance xxiii MANAGEMENT COMMENTS industry. In particular, the report notes MIGA’s support to the industry through its reinsurance business. In 2013, MIGA modified its frameworks for allowing the use of reinsurance at shorter tenors compared with the underlying guarantee contracts. The change allowed MIGA to expand the panel of facultative reinsurers for optimizing reinsurance capacity for larger, longer-tenor projects that are beneficial to MIGA clients. Streamlining Project Evaluations. In discussing the MAR, the report notes IEG’s recommendation to streamline the project evaluation approach and process to align more closely with MIGA’s business model and conditions on data gathering, which stemmed from IEG’s 2013 Biennial Report on Operations Evaluation: Assessing the Monitoring and Evaluation Systems of IFC and MIGA. MIGA notes that this recommendation has been implemented satisfactorily—with significant progress in streamlining project evaluations—and hence was retired in 2015, after discussion and agreement between MIGA and IEG. Methodological issues. MIGA notes that a comprehensive and appropriate assessment of MIGA’s development outcome performance is presented only in the main text of the report. There it shows the distribution of ratings as: (i) satisfactory or better; (ii) No Opinion Possible (NOP); and (iii) less than satisfactory. IEG does not apply this comprehensive approach consistently in the report. For example, in the Regional Results and Performance discussion in appendix I and in footnote 15 of chapter 2, the computation of “success” rates include NOP-rated projects in the total project count. The result of this “success” rate formulation is that it treats NOP-rated projects as equivalent to projects rated less than satisfactory. This approach is mathematically flawed and misleading, detracting from the IEG objective of providing a comprehensive assessment of MIGA performance, and hence not useful, in MIGA’s view. More generally, the mathematical error in the success rate computation would diminish the accountability and learning value of IEG’s flagship report, given the erroneous basis for the findings. MIGA notes that the change to MIGA’s success rate computation was made without any prior notification or consultation with MIGA, in violation of the agreed protocol between management of World Bank Group institutions and IEG. MIGA believes that the presentation in the main section of the report of MIGA projects is the appropriate one, including with respect to the calculation of success rates, and that this approach should be applied throughout the report. This approach also provides continuity and consistency with the approach used earlier for computing success rates (i.e., excluding NOP-rated projects from the Total Project Count) in previous RAPs. xxiv 1. Managing for Results Highlights  The World Bank Group has taken several important steps over more than a decade to address the principles of managing for development results; progress is notable on instituting a structure and process for measurement and less so on adaptive management and learning.  The World Bank Group’s management for results is affected by weaknesses in measurement of outcomes, adaptive management based on evidence, and by generally weak client systems that generate results data.  The World Bank Group is addressing client capacity to manage for results through various instruments and approaches; the effectiveness and results of these efforts have not yet been assessed.  The World Bank Group is at a good point to take stock of its progress and systematically plan for and address the now well-known constraints to strengthening both measurement and adaptive management and learning. Why Address Managing for Results Now? Important developments since 2013 have resulted in a renewed and qualitatively different emphasis on results within the World Bank Group. These developments include the following:  The World Bank Group established a new strategy in 2013 for contributing to the twin goals of ending extreme poverty and promoting shared prosperity.  The 18th Replenishment of IDA (IDA18), the largest replenishment to date, took place in 2016 with a greater emphasis on results.  Forward Look: A Vision for the World Bank Group in 2030 recently detailed the organization’s role in global economic development. More broadly, as the Millennium Development Goals ended in 2015, the development community renewed its commitment to a set of more ambitious Sustainable Development Goals. Analyses based on evaluations of World Bank Group projects demonstrate empirically that key elements of managing for results (M4R) are important for achieving outcomes. For example, a recent study showed that a good monitoring and evaluation (M&E) framework (that is, data and evidence) is positively and significantly associated with project outcome ratings, controlling for a number of 1 CHAPTER 1 MANAGING FOR RESULTS factors (Raimondo 2016). A World Bank Internal Audit Department report (IAD 2015) similarly found that 85 percent of problem projects with a high M&E rating received a satisfactory outcome rating from the Independent Evaluation Group (IEG), compared with 45 percent of problem projects with a low M&E rating. How the purposeful use of data and evidence help to achieve results has been well documented externally as well (Oportunidades program in Mexico, the Government Accountability Office, and the UK National Audit Office, for example).1 International agreements, including the 2005 Paris Declaration, the 2008 Accra Agenda for Action, and the 2011 Busan Declaration, also highlighted the importance of M4R, and described how to operationalize the M4R in international development. Drawing on cumulative international development experience and research, they emphasized that the results—that is, development results—are for and of the client countries that need strong client M4R capacity for their achievement (see, for example, the Busan Partnership for Effective Development Co-operation).2 This is an opportune time to assess the World Bank Group’s progress on managing for results. This chapter acknowledges the accomplishments to date and synthesizes and presents cumulative evidence from IEG studies to highlight areas that require in-depth review and concrete action. What Is Managing for Development Results? Managing for results (M4R) is generally defined as a management strategy or approach aligned with achieving organizational results. For this review’s purposes, M4R refers specifically to managing for development results. M4R’s concepts and approaches have a long history in both the private and public sectors, and much academic and applied literature exists on the topic (for example, GAO 2015 and Kang 2005). The literature integrates the concepts of measurement, assessment, management, learning, and decision making, and highlights two interdependent principles of M4R with an emphasis on outcomes, not simply on inputs, processes, and outputs. The two principles are:  Developing a robust system for measuring results (for example, a results framework). First, desired results are identified, defined, and communicated clearly; and, second, data related to the results and to preceding factors important for ongoing decision making (based on a change theory) are collected and monitored on an ongoing basis (figure 1.1).  Instituting adaptive management and ongoing learning. First, data from the measurement system are used for different types of decision making for 2 CHAPTER 1 MANAGING FOR RESULTS continual improvement to achieve results; and second, data and evidence are also used to learn about and understand the drivers of results. Figure 1.1. Principles of Managing for Results Source: IEG. Using the principles above, the 2016 Results and Performance of the World Bank Group (RAP) report addresses how, and how effectively, the World Bank Group approached M4R. Specifically:  How, and how effectively, did it measure development results?  How, and how effectively, did it engage in and support adaptive management and learning to achieve results? The report also provides a brief exploratory assessment of how the World Bank Group supported client capacity for M4R, given its acknowledged importance in achieving development results. Limitations of this Review The report starts with the premise that the primary goal of the World Bank Group’s M4R is to drive development results for its clients (through its projects, investments, guarantees, country strategies, and other client services). It summarizes the World Bank Group’s M4R efforts regarding the two questions as they relate directly to development results. Therefore, it does not review operational and financial processes and systems, such as personnel performance management, procurement, financial management, or revenue models, although these also intrinsically support (or hinder) managing for development results. The report relies on a pattern of evidence and findings from existing IEG evaluations. It does not cover the entire portfolio comprehensively, nor does it cover World Bank Group initiatives that are addressing M4R but have not yet been evaluated. The exploratory review of World Bank Group support for client M4R capacity is based primarily on a desk review of project-level evaluations and the limited number of existing country strategies since 3 CHAPTER 1 MANAGING FOR RESULTS FY14, as in-depth evaluation of this topic has not yet been undertaken. This report presents a potential baseline but does not draw comprehensive conclusions regarding this topic. M4R History—More than a Decade with Some Success The World Bank Group has addressed the two M4R principles—measurement and adaptive management and learning—for more than a decade and successively refined its approach, though at a somewhat uneven pace. IEG’s annual report on operations evaluation in 1998 was the first report that substantially addressed M4R, explaining that the World Bank had not yet formally adopted M4R, but that it was highly relevant for the Strategic Compact, the then- ongoing renewal process (World Bank 1998).3 It concluded that instituting a culture for M4R would take sustained effort for several years. The World Bank formally articulated its M4R approach a few years later in Better Measuring, Monitoring, and Managing for Development Results (World Bank 2002). Four years later, Accelerating the Results Agenda: Progress and Next Steps, a report to the Committee on Development Effectiveness, outlined further actions (World Bank 2006). The World Bank recognized that the principles of M4R in international development should apply to its work, so the plan also emphasized enhancing countries’ statistical systems, M&E capacity, and knowledge base to manage for results. The third main renewal of these efforts began with the 2013 strategy, with enhanced focus on results and on monitoring and measuring results as “One World Bank Group” (World Bank 2013). As part of these efforts, the World Bank introduced a results measurement system (RMS) for the International Development Association (IDA) in 2002 to track systematically key country outcomes and IDA’s contributions to those outcomes (IDA 2006). This system was enhanced during subsequent IDA replenishments, and an updated RMS for IDA18 was agreed in December 2016. The World Bank in 2009 gradually introduced core sector indicators across its IDA portfolio to improve the consistency and quality of sectoral and thematic results monitoring and reporting at the project level, and 2014 saw the introduction of the first Corporate Scorecard covering all three World Bank Group institutions. The Corporate Scorecard built on the IDA RMS and was expanded to include International Bank for Reconstruction and Development (IBRD) countries, to track and communicate results, and to be an important input into decision making by focusing management attention on key indicators. Scorecards for both the World Bank Group and the World Bank have a matching three-tier structure with key indicators: goals and development context, 4 CHAPTER 1 MANAGING FOR RESULTS results, and World Bank Group performance (World Bank 2014, 2016k,l). The Corporate Scorecard will be renewed in FY18 (World Bank 2016c). The World Bank also introduced results-based country assistance strategies (CASs) in 2005 and the Program-for-Results (PforR) in 2011. A renewed World Bank Group- wide evidence-based country engagement model consisting of systematic country diagnostics (SCDs) and country partnership frameworks (CPFs) was developed in 2013. It was designed to contribute concretely to the sustainable achievement of the World Bank Group’s twin goals within countries. The World Bank established the Development Impact Evaluation (DIME) Initiative in 2005 and has continued attending to strengthening client statistical capacity with trust funds and statistical operations.4 The Global Delivery Initiative, launched in 2015, provides practitioners with collective and cumulative evidence to build and share evidence-based knowledge of implementation science. The Results Measurement and Evidence Stream was established as a network of support for World Bank Group professionals engaged in M&E. Other initiatives have been undertaken at the vice presidency levels as well, including for example the “Initiative to Enhance Outcomes” in East Asia and the Pacific and the “African Gender Innovation Lab” to enhance focus on results and evidence throughout the project cycle. Recently, the World Bank Group agreed on an action plan to address data gaps, the first pillar of which supports country capacity to produce, disseminate, and use data.5 In addition, actions are underway with respect to strengthening incentives, behaviors, and processes related to achieving development outcomes, based on recent IEG evaluation recommendations. Externally, the World Bank Group participates in the Managing for Development Results Community of Practice for mutual learning and benchmarking among international development banks.6 The World Bank also introduced structural and functional reforms between 2010 and 2013, reflecting changing perspectives on the best way to manage for results more adaptively. The reorganization into Global Practices pivoted the institution toward becoming a “solutions bank.” At the same time, the Quality Assurance Group (established in 1996), which monitored portfolio quality at entry (including results frameworks) was disbanded in 2010, and the World Bank Group-wide Results Steering Group (established in 2006) discontinued functioning. Units in the Operations Policy and Country Services (OPCS) Vice Presidency took on both of those functions. M4R REFORMS SPECIFIC TO IFC Monitoring and evaluation of IFC operations started in 1994 with the creation of the Operations Evaluation Unit (later merged into IEG). IFC, through its Development Impact Unit, developed the Development Outcome Tracking System (DOTS) in 5 CHAPTER 1 MANAGING FOR RESULTS FY05 to monitor and measure the development outcomes of its investment and advisory services, and began testing the IFC Development Goals (IDGs) and the Corporate Scorecard in FY11 to help drive strategy implementation and business decision making alongside other reforms. IFC also viewed the IDGs (Climate Change, Micro/Individual Finance, SME Finance, Infrastructure and Agribusiness) as a tool to help drive alignment across investment and advisory services activities. IFC continued to refine and improve the IDGs between FY13 and FY15 while positioning itself to conduct meta-evaluations (for example, effects of a cluster of investment projects within a country or sector on job creation) and contribute to CPFs and joint implementation plans. IFC has been leading international financial institution initiatives since FY13 to harmonize indicators in their private sector operations and to incorporate them into the results reporting systems. The IFC Corporate Scorecard has two dimensions: results (Tier II) and performance (Tier III). The IDGs constitute the core of the results, which are presented in the broader categories of growth, sustainability, and inclusiveness. The scorecard is intended to serve as an effective tool for IFC strategy implementation and a comprehensive review. M4R REFORMS SPECIFIC TO MIGA The Multilateral Investment Guarantee Agency (MIGA) established its own development results monitoring system, the Development Effectiveness Indicator System, modeled on DOTS. MIGA measured five key performance indicators to assess its performance and productivity for more than a decade and expanded the key performance indicators in FY15 to develop a Corporate Scorecard that is presented in the World Bank Group Scorecard. MIGA’s scorecard reports on development impact, financial sustainability, and client satisfaction, and captures both financial and development results, along with operational efficiency indicators and achievement of strategic priorities, such as guarantees for projects in situations of fragility, conflict, and violence (FCV) and in IDA countries. The World Bank Group Scorecard features two MIGA performance indicators: development outcome rating and the time for operational delivery (concept-to-guarantee issuance time). IEG’S ROLE IN M4R IEG has a core role within the World Bank Group M4R system through its systematic validation of self-evaluation reports and its independent evaluations. The evaluations provide recommendations to which management of the World Bank Group institutions responds with an action plan for those with which it agrees (chapter 3 provides more details). The annual RAP reports systematically on World Bank Group results as shown through its project validations and evaluation 6 CHAPTER 1 MANAGING FOR RESULTS findings. In recent years IEG has also increased its focus on learning and is piloting a learning engagements business line that uses existing IEG evidence to support evidence-based learning within the World Bank Group. BOTTOM LINE: THE WORLD BANK GROUP HAS TAKEN KEY STEPS ON M4R PRINCIPLES Reforms over more than a decade have clearly led to positive changes within the World Bank Group, particularly with respect to the steady emphasis on results measurement. These changes signaled the growing importance of M4R and placed the World Bank Group at the forefront of these efforts. Based on assessments of its M4R processes and systems, the 2012 Multilateral Organization Performance Assessment Network report found that the World Bank Group showed leadership in managing for and reporting on development results and was considered strong in its evaluation of results and knowledge dissemination. The benefits of the ongoing and recent M4R-related reforms will likely continue to accrue in the future. As the sections below summarize, IEG finds that the steps completed are necessary, but not sufficient, and the World Bank Group needs to continue to address some core challenges. What Is the Evidence on the World Bank Group’s M4R? This section reviews how effectively the World Bank Group performed with respect to the two principles of M4R—measurement and adaptive management and learning—based on IEG evidence. MEASURING DEVELOPMENT RESULTS The M4R principle of results measurement requires collecting timely, good-quality data on outcomes of interest and on factors that affect these outcomes (per the SMART criteria and basic measurement principles).7 The World Bank Group contributes to the twin goals and to outcomes that are more proximal. It typically measures its results through M&E systems established as part of the World Bank Group’s client services.8 However, evidence in a vast majority of fully disclosed major IEG evaluations and learning products completed in FY15 and FY16 (27 out of 33) shows that key aspects of results measurement—data availability and focus on outcomes—will need attention for a sustained period (appendix A presents the list of IEG evaluations conducted in FY15–16 and their findings related to this topic). Measuring World Bank Group Goals The Poverty Focus of Country Programs: Lessons from World Bank Experience (IEG 2015j) highlighted large gaps in income and consumption data for several countries, 7 CHAPTER 1 MANAGING FOR RESULTS especially in Africa and FCV countries. The evaluation identified several challenges in producing high-quality data for measuring poverty. The World Bank Group’s Poverty and Shared Prosperity 2016: Taking on Inequality (World Bank 2016d) similarly noted progress, but also highlighted data deprivation as a key challenge. Measuring Operations’ Outcomes IEG finds good focus on the outputs of World Bank Group operations (important for implementation), but often notes insufficient emphasis on measuring outcomes to which the World Bank Group operations contribute. Outcomes are defined as the beneficiary-level (including individuals, households, firms) changes and benefits from project processes and outputs, such as improved nutrition intake, shortened commuting time, utilization of financial services, and improved air quality. Learning from IDA Experience: Lessons from IEG Evaluations (IEG 2016g) summarized existing evidence from several evaluations and found a paucity of data on the impact of World Bank Group work on the many types of outcomes it aimed to address, such as youth employment, nutritional and welfare outcomes of food crisis and mitigation programs, and climate change and adaptation-related outcomes. IEG evaluation on capital market development (IEG 2016t) made a similar point, concluding that data issues affect the capital markets program, and databases do not capture local currency bond market development. Operations also need to strengthen their focus on articulating and measuring outcomes. World Bank Group Support to Electricity Access, FY2000–2014: An Independent Evaluation (IEG 2015q) noted shortcomings in tracking economic and welfare outcomes. An IEG evaluation on financial inclusion (IEG 2015b) found that 45 percent of the projects tracked process and outputs, but only 2 percent reported on beneficiary effects, such as improvements in welfare or increases in income. Industry, Competitiveness, and Jobs (IEG 2016e) noted that the complex challenges of articulating and measuring jobs-related objectives need to be addressed, given that employment is a key aspect of the productivity and competitiveness agenda. An IEG evaluation on early childhood development (IEG 2015s) made similar observations. This deficiency also at times affects PforR. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument (IEG 2016k) found that the results frameworks are often reasonably coherent, but objectives that lead to clear results at the outcome level are rare. It concluded that the objectives in the project appraisal document (PAD) could go beyond the program period itself to consider longer-term outcome-level objectives. 8 CHAPTER 1 MANAGING FOR RESULTS Measuring outcomes regarding gender and FCV status also requires attention. RAP 2015 discussed the inadequacy of gender-related indicators.9 When present, the indicators tended to measure outputs instead of outcomes. In September 2016, the World Bank revised its protocol for recording integration of gender in projects. 10 Learning from IDA Experience: Lessons from IEG Evaluations IEG (2016g) illustrated that projects in FCV situations have conflict-related objectives, but the results frameworks do not identify their outcomes. IEG’s review of development policy operations’ results frameworks found that objectives were often vague, and the results frameworks measured processes instead of outcomes. For example, Lessons from Environmental Policy Lending (IEG 2016i) found a risk that the World Bank is focusing on the short term and could consider long-term, policy-related performance. However, the current World Bank evaluation system does not support the longer-term data collection required. Other Aspects of Designing Robust Results Frameworks Baselines are an important component of results frameworks for measuring improvements in the outcomes to which the World Bank Group aims to contribute. Learning and Results in the World Bank Operations: Toward a New Learning Strategy (IEG 2015g) found that projects often do not collect data when the projects begin, and implementation status and results reports often misleadingly indicate baselines as zero. Results Frameworks in Country Strategies: Lessons from Evaluations (IEG 2015n) noted that despite instituting results-based CASs, the results frameworks in country strategies had weak links between designed interventions and outcomes and weak indicators to track outcomes. For example, IEG’s World Bank Group Engagement in Resource-Rich Developing Countries (IEG 2015p) found that although diversification was the overarching strategy objective in these countries, no mechanisms were in place to monitor this objective. The forthcoming IEG evaluation on SCD and CPF process and implementation finds that the results chains are better defined, and the identification of monitorable and measurable indicators for the results has improved over the past, though they do not always capture the intended objectives. On the other hand, the role of the World Bank Group’s program in contributing to the achievement of outcomes is often not well thought through and explained. Measuring IFC Development Results A review by an external auditor suggested that although IFC has pioneered several best practices in aspects of M&E reporting (such as Performance Standards), IFC should continue to strengthen beneficiary-level indicators and limit the use of estimations and extrapolations of results to ensure accuracy (IFC 2016). IEG concurs 9 CHAPTER 1 MANAGING FOR RESULTS with this view based on its independent findings from project validations and thematic evaluations (IEG 2014a, 2015b,d). Gaps exist in IFC M4R at the beneficiary level, in results related to private sector development,11 and in joint practices. IEG finds that the project M&E indicators (investment services or advisory services) are not always tracked consistently through project maturity or through the DOTS systems (appendix B), and the absolute numbers presented in the reach indicators are not reflected in relative terms to provide a good understanding of the country or regional context. Furthermore, the two indicators tracked in the Department Scorecard and the IFC Corporate Scorecard—economic performance percentage rating and private sector development percentage rating—are from the DOTS framework and not from IEG- validated results, which show lower levels of achievement. The private sector development ratings do not necessarily capture auditable, verified outcome results, but instead rely on estimates from IFC and, in many cases, results from unverified client surveys, suggesting data quality needs improving. Even when data related to private sector development are strong at the beneficiary level, the overall country-level performance is not captured effectively because the World Bank and IFC’s roles and responsibilities in the country-level frameworks are not yet well defined. This is especially true when the project is a large, first-time IFC intervention in the sector (the most notable cases are in the Sub-Saharan Africa and Latin America and the Caribbean Regions). The forthcoming IEG report, An Early Assessment of SCD/CPF Process and Implementation, also emphasizes the need for a more explicit inclusion in the results frameworks of the current and future areas where IFC and MIGA intend to focus their support. Furthermore, IEG’s review of self-evaluations and validations for FY13–15 shows that IFC engagements with repeat clients (appendix B) were more conventional than an exploration of ways to incrementally improve results at the borrower and beneficiary levels. Although additionality (financial and nonfinancial) does not distinguish between new clients and repeat clients, the latter could potentially provide verifiable, auditable data at the beneficiary level because repeat clients are more familiar with IFC standards and procedures. Measuring MIGA Development Results IEG validations of traditional political risk insurance (PRI) guarantee projects suggest that MIGA has continued to strengthen in results frameworks and align its Performance Standards monitoring and reporting to that of IFC, yielding positive results. Recent IEG validations of MIGA non-honoring projects (three projects) 10 CHAPTER 1 MANAGING FOR RESULTS suggest that the current monitoring, reporting, and evaluation framework, though suitable for traditional PRI guarantees, could be improved for the newer instrument to capture public sector risks.12 The ability to collect data and outcomes at the level of public sector obligors and beneficiaries is challenging for MIGA based on existing frameworks, thus warranting additional risk assessments during underwriting and contract management. The paucity of project performance data has led to rating of No Opinion Possible for the development effectiveness indicator (seven projects in the FY10–15 review period). In sequenced or joint Bank Group projects, IEG project evaluations suggest that MIGA should enhance its project-level results monitoring and measurement to capture client-level and country-level results and for guarantee holders who are also IFC clients. Recent IEG evaluations during FY13–15 (specifically China’s water sector, Sub-Saharan Africa extractives sector, and Turkey’s transport sector) suggest that internal World Bank Group protocols hinder MIGA’s ability to access project performance indicators, even though such information is readily available to IFC as the project lender or project company shareholder. In infrastructure and financial markets sectors, MIGA has several key repeat- business clients (two or more projects with the same guarantee holder between FY09 and FY15) who are familiar with MIGA and the World Bank Group’s development mandate and could be more receptive toward monitoring and measuring incremental beneficiary-level results through the life of the guarantee. Management of the World Bank Group institutions could facilitate such an incremental approach, especially for joint World Bank Group projects and for MIGA clients who are also IFC clients. Although MIGA has other risk assessment indicators in place, such as project business performance risk, sovereign risk, and risk to MIGA profitability, gaps exist in its articulation, identification, and discussion of risks to development outcomes at both underwriting and during the life of the guarantee, specifically in non-honoring guarantee projects exposed to public sector default risk. In such cases, MIGA can learn and adapt from the World Bank’s results frameworks. ADAPTIVE MANAGEMENT AND LEARNING TO ACHIEVE RESULTS Adaptive management and learning can be instituted at the project (or program) and corporate levels to drive development results, but it takes intentionality and discipline to do so. Evidence and data are fundamental inputs to informed judgments that underpin adaptive management and learning, and they need to be tailored to specific purposes at different levels of the organization. 11 CHAPTER 1 MANAGING FOR RESULTS Two main mechanisms exist to drive project development outcomes: using evidence generated by M&E systems to learn and make adjustments, and applying evidence- based lessons to new project design and implementation. At the corporate level, adaptive management involves using the scorecard (and associated instruments, such as memoranda of understanding) to focus on strategic issues and make deliberate choices in organizational directions, taking into account leading indicators based on empirical evidence and actual results achieved. Two recent IEG evaluations documented the challenges in these areas. World Bank projects are generally weak in their attempts to learn empirically about the validity of the change theory underpinning the project or program, and about other factors that might affect outcomes. The outcomes need not be expected to be achieved entirely and uniquely by World Bank Group investments, but managing for results requires some expected level of verifiable contribution. Learning and Results in World Bank Operations: Toward a New Learning Strategy (IEG 2015g) found that only about 28 percent of the projects reviewed for the study discuss the diverse factors that might have affected the project outcomes. Fifty-eight percent of the projects with some outcome evidence used a pre-post evaluation design to assess efficacy in implementation completion and results reports (ICRs) with no control or comparison groups or attempts to provide potential alternative explanations based on a change theory. The Poverty Focus of Country Programs (IEG 2015j) and Learning Note: Additional Financing for Transport and Information and Communication Technology (IEG 2015h) highlighted that attributing the outcomes measured to World Bank–supported interventions is often not possible, thus leaving open questions about the efficacy of the World Bank Group’s investments. Furthermore, 27 percent of projects reviewed for Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group (IEG 2016a) conducted their mid-term review in the third quarter, relatively late for timely adjustments to project design and implementation. The projects rarely assessed unintended consequences. The World Bank Group introduced impact evaluations more than 10 years ago for learning, improving project design and interventions, and identifying impact. IEG’s World Bank Group Impact Evaluations: Relevance and Effectiveness (IEG 2012b) recommended identifying topics for impact evaluation strategically and improving the integration of evaluations into project design. However, IEG (2016a) reported that strategic selection was still not evident, and the Management Action Record shows that the recommendation-related actions are not complete (chapter 3). 12 CHAPTER 1 MANAGING FOR RESULTS On the positive side, the World Bank developed safeguard policies to guard against—and mitigate—potential negative outcomes associated with its investment operations. IEG’s safeguard evaluation (IEG 2010) found that the World Bank Group’s safeguards and sustainability policies have helped to avoid or mitigate large-scale environmental and social (E&S) risks in its investment projects, but it also found inconsistent categorization of risk as well as deficiencies in monitoring of E&S safeguards results. An assessment of safeguards compliance is beyond the scope of this review. However, an assessment of safeguard-related reporting in 119 ICRs reviewed by IEG between August 2015 and June 2016 was conducted to check whether ICRs of projects that triggered safeguards contained the safeguard-related information required by the ICR guidelines.13 Of the 119 projects, 103 triggered safeguards, so that a summary of safeguards compliance would be required in those 103 ICRs. The assessment found that 53 of those 103 ICRs (51 percent) contained at least one sentence stating whether or not all triggered safeguards policies had been complied with, but the remaining 49 percent did not. Of the 53 ICRs that included an explicit statement regarding compliance for all E&S safeguards, 32 (60 percent) included a summary of “any problems that arose and their resolution.” Of the 15 Category A (highest risk) projects in this group, 11 ICRs lacked a summary of whether or not all triggered safeguards policies had been complied with. The 2013 World Bank Group Strategy (World Bank 2013) cited the importance of engaging with citizens to understand the results they value and experience, and using that information to improve program effectiveness. IEG (2016a) showed that less than half of the projects with clearly identifiable beneficiaries and that exited the portfolio in FY14 used beneficiary surveys. Although beneficiary feedback at the design stage has improved, shifting to collecting feedback during implementation continues to be challenging. Interviews conducted by IEG (2016a) showed that 78 percent of staff specifically mentioned that there are either negative or no incentives for candid self-evaluation that would lead to learning. Corporate results reporting can distort incentives and lead staff to favor reporting positive outcomes or become excessively risk averse. At the same time, the review highlighted the World Bank’s approval and disbursement culture, thus indicating that inputs have priority over outcomes (IEG 2015g). In addition, there continues to be an organizational need for better matching staff capacity to the requirements for different aspects of M4R and for strengthening staff skills, competencies, and career paths in areas where there are gaps (IEG 2016a). Project M&E quality reflects these concerns. IEG rated only a small portion of FY15 project exits (27 percent) as significant or high, thus limiting M&E’s utility for evidence-based learning and adaptive management. 13 CHAPTER 1 MANAGING FOR RESULTS No clear information is available on how the World Bank Group uses evidence and data for strategic adaptive management at the corporate level. The Corporate Scorecard provides three tiers of data, yet it is unclear how Tier II indicators that communicate the World Bank Group’s specific contributions to client countries’ development incentivize adaptive management. The results are based on absolute numbers achieved, rather than percent of target, and operations in a few large countries are likely to skew the results. Furthermore, they do not convey a sense of how the World Bank Group is doing with respect to its targets. The Board also requested actionable indicators that can help inform strategy in pursuit of results. Additional indicators are tracked at the vice presidential unit and project levels, which could provide learning (Blanc et al. 2016, for example). Whether the volume of indicators and their reporting makes sense, and whether they support or hinder learning and management at different levels of the organization are questions that need review. ADAPTIVE MANAGEMENT SPECIFIC TO IFC IFC continues to focus on adaptive management, but challenges remain. The FY16 Board-endorsed budget paper addresses IFC’s efforts to enhance institutional and scorecard results frameworks by triangulating project results with country frameworks, budget and personnel resources, and IDGs and targets. Lessons in IFC (IEG 2016h) highlighted the importance of the LessonFinder database (lessons from self-evaluations and validations of IFC and MIGA projects) for operational team learning, and lessons not already emphasized or mainstreamed in new business operations. The report highlighted IFC’s strong oral communication culture when it comes to learning, sharing, and applying lessons. Therefore, some internal learning efforts are deliberately focused on face-to-face engagements. The IFC Special Operations department is one place from which investment teams actively seek lessons and integrate them into operations.14 The unit’s objective is to reverse potentially bad outcomes of underperforming projects in part by using project data (financial, risk, and economic) to improve outcomes. A Special Operations database captures successful turnaround stories, which the unit shares with IFC investment teams in various forums, training programs, and events. Integrating lessons from evaluated investment projects into new business poses challenges, despite the emphasis. Capturing early warning signs in the project through portfolio management, using dedicated client relationship managers, and implementing active postcompletion supervision were reasons for successful adaptive management at the project level. A 2014 IEG survey of IFC staff found that IFC does not have an effective lesson learning system, even though staff recognize the value of lesson learning and corporate culture does not repress it. Seventy 14 CHAPTER 1 MANAGING FOR RESULTS percent of staff reported either negative, weak, no link to rewards and recognition for lesson learning activities (IEG 2016h). Furthermore, advisory services institutionalized lesson learning more than investment services, mainly because project supervision reports and project completion reports are mandatory for all advisory projects, and the reports include a section for lessons learned. Investment Services does not systematically require lessons identification, and neither approval (Board) nor supervision credit risk rating (CRR) documents include a mandatory section for lessons learned. Only the Expanded Project Supervision Report (self- evaluation document) includes such a section. The learning culture and good practice from advisory services on lesson learning could be incorporated early into the concept review and investment review stages of investment services activities. Despite these efforts, a FY16 IFC corporate staff survey revealed that staff considered volume targets and project profitability as the most important metrics for management, compared with development impact and client satisfaction levels. Inconsistency in other areas of adaptive management in IFC advisory projects continues to hinder outcome achievement, especially when circumstances in the field change during project implementation. IFC advisory projects that have strong intentions for M4R principles at approval (specifically in the microfinance and insurance business areas) do not adjust the project log frame to reflect changes in project scope, timing, and deliverables. An emerging finding from IEG’s qualitative review (projects evaluated in the recent three-year period) is the need for uniform approach in M&E log frame revisions during both project implementation and systematic lessons capture from M&E, with support through appropriate incentives from the IFC Advisory Services leadership team. In the past year, advisory services teams have started instituting stronger controls around log frame changes outside of the A2F/FIG/FAM business lines. Mandate-to-disbursement is one of the newer indicators introduced in the IFC Scorecard in FY14. Intuitively, this is a good indicator to track lending efficiency at the department or industry level. However, statistical analysis for the recent cohort of projects (FY13–15) reveals that mandate-to-disbursement is not a driver of development outcome success ratings (appendix F). Newer indicators of this sort may have other purposes for corporate-level M4R, but it could be worth checking periodically for any links to beneficiary (private sector development results) or borrower (satisfaction rates from client survey) results for adaptive management. ADAPTIVE MANAGEMENT SPECIFIC TO MIGA Recent evaluations of MIGA projects show progress in adaptive management. MIGA has consistently used its self-evaluations and validations mechanism in recent 15 CHAPTER 1 MANAGING FOR RESULTS years to identify lessons; and it has used the knowledge gained in its new business reviews through a combination of information systems and knowledge events. In select cases, MIGA conducted broad analysis (China’s water sector, for example) to study the macroeconomic, sectoral, and country conditions before intervening through issuances of new guarantees. BOTTOM LINE: ADDRESSING M4R NEEDS EMPHASIS ON BOTH M4R PRINCIPLES The World Bank Group has acknowledged the importance of M4R. However, the challenges of data availability, focus on outcomes, and the overall quality of results frameworks impede results measurement—regardless of whether results are achieved. This situation likely undermines the potential of the scorecard and related management instruments as strong tools for adaptive management. It is not clear how current metrics on results inform (or can inform) decision making and learning at different organizational levels. The Strategic Actions Program for Addressing Development Data Gaps (2016) and the indicator reforms acknowledge the climb ahead and are a good start for addressing the World Bank Group's measurement challenges. In addition, incentives, strengthening staff capacity in areas of gaps in M4R, and instilling culture of evidence-based learning and decision making need to be strengthened. Action and reforms underway in these areas will need to be considered within a holistic approach. The World Bank Group Support to Client M4R Capacity Clients’ institutional contexts and the quality of governance play a significant role in outcomes (see for example World Development Report 2017), and principal-agent problems mediate results achievement (Gaarder and Bartsch 2015, for example). The international community has also repeatedly identified weak client M4R as a constraint to measuring and achieving results, and the World Bank Group staff have corroborated this observation (IEG 2016a). Staff believes that learning with clients is the best guarantor of results, but learning from self-evaluations—typically in collaboration with the client—remains relatively weak, particularly due to low client capacity, leading to a vicious cycle (IEG 2016a). IEG evaluations (IEG 2015j) also identified insufficient capacity and government budget as obstacles to collecting data and managing for results. Recognizing that M4R capacity is an insufficient but necessary condition for achieving results, this section explores the World Bank Group’s role in strengthening clients’ capacity. Throughout the years, the World Bank Group has invested in client M4R capacity through projects, advisory services and analytics, and dedicated statistical capacity- strengthening programs for data systems. Country strategies have also addressed 16 CHAPTER 1 MANAGING FOR RESULTS the topic. Given the importance of the results agenda, this section presents baseline information to consider for moving forward. The M4R principles of measurement and management can be operationalized through several elements, including strengthening data systems, monitoring, evaluation, evidence-based programming and policy making, results- or performance-based budgeting, and results- or performance-based financing. M4R includes strengthening corporate governance for private sector clients (figure 1.2).15 The next sections focus on whether and to what extent these M4R elements were included in the World Bank Group’s operations and country strategies. Figure 1.2. Operationalizing Principles of Managing for WORLD BANK DEVELOPMENT OF Results in World Bank Group Operations CLIENT CAPACITY IEG reviewed 265 implementation completion report reviews (ICRRs) for World Bank projects that closed in FY14 and FY15, and 82 recipient-executed trust funds that closed in FY13– 15. The review assessed whether World Bank operations addressed the M4R elements to strengthen client capacity. Source: IEG. The review found that Note: M4R = managing for results. about half of operations addressed one or more elements of clients’ M4R (49 percent). Among these, the majority supported monitoring capacity (83 percent), and 70 percent supported evaluations. Twenty-six percent specifically included impact evaluations, but a little more than half of these operations clearly stated that they used information from the impact evaluation to influence a policy, program, or project improvement. Most of these elements were activities embedded in larger components (79 percent), although a few projects included them as a full component (16 percent). Some operations specifically addressed M4R as an objective (13 percent). Monitoring and evaluation are steps in an M4R system, but M4R’s main goal is for relevant actors to use the evidence base and data produced by that system for adaptive management and learning. As 17 CHAPTER 1 MANAGING FOR RESULTS shown in figure 1.3, the least common aspect of client capacity addressed was evidence-based decision making (including use for financing and budgeting). The 49 percent of projects that specifically focused on client M4R were in 71 countries, including 29 FCV countries.16 Latin America and the Caribbean led all regions in projects with one or more M4R elements (figure 1.4), and 51 percent of IBRD operations addressed M4R compared with 42 percent of IDA projects. IEG also found that 11 percent of advisory services and analytics (ASA) during FY13–15 supported countries’ M4R capacity, based on a review of the theme codes related to M4R. This figure may be an underestimate, as advisory services with other theme codes may have elements of M4R, although these are not likely to be significant. Figure 1.3. Monitoring Was the Most Figure 1.4. A Majority of Latin America Common M4R Element Addressed for Clients Projects Addressed M4R Capacity Monitoring 83 Latin America and the Caribbean 66 Evaluation 70 South Asia 54 Data systems 50 Africa 50 Evidence-based policy/program 38 Europe and Central Asia 40 Results-based financing 14 East Asia and Pacific 40 Results-based Budgeting 2 Middle East and North Africa 30 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 Percentage Percentage Source: IEG calculations based on implementation completion report review. Note: M4R = managing for results. Neither the effectiveness of the M4R activities nor the resources devoted to them were feasible to assess for this review because this information is not obvious from the ICRRs. In the few projects that had M4R as an objective, IEG outcome ratings were moderately satisfactory and higher. IEG also found that among the 51 percent of the projects that did not address client M4R, more than half (57 percent) identified it as an issue (lack of government capacity, for example) in the PADs, project papers, or ICRRs. Furthermore, both the PADs and the ICRRs in 17 percent of cases mentioned M4R as an issue. Additional analysis of the project performance assessment reports posted in 2015 and 2016 found that lack of or weak M4R in the country were mentioned in 24 percent of cases. The reasons why the World Bank did not address M4R in these cases is not evident from the review. Potential reasons include lack of client demand, the capacity was being addressed in other ways, or simply lack of attention to this issue. 18 CHAPTER 1 MANAGING FOR RESULTS IFC DEVELOPMENT OF CLIENT CAPACITY IFC enhances client capacity for M4R primarily through advisory services, its environmental and social (E&S) Performance Standards in Investment Services, dedicated instruments such as corporate governance, and nonfinancial additionality support for enhancements to management information systems, data systems, risk control systems, M&E systems, and financial reporting systems. IEG analysis of IFC M4R activities at the project level (appendix B) found that in the recent cohort of IEG evaluated and validated IFC projects, 69 of the 81 investment projects (85 percent) and 61 of the 71 advisory projects (86 percent) had M4R characteristics. Appendix B describes the methodology for this analysis. The review found that IFC’s management of results implementing E&S Performance Standards was superior in its approach and execution compared with other M4R tools at IFC, but IFC could further strengthen the approach for clients who repeatedly default17 on lender terms (specifically on E&S Action Plan compliance). In engaging with clients in IDA and middle-income countries, the IFC approach to building capacity through E&S-related data systems was superior compared with other M4R tools advocated at the project level. Although IFC’s performance on E&S-related outcomes is positive and shows improvement over the long term, IEG’s analysis (FY01–15) of clients’ E&S capacity development (FY01–15) indicate that E&S performance has leveled off. It was noted that since 2006, the clients were required to meet additional E&S requirements, such as Labor and Working Conditions, under IFC’s Performance Standards that were not included and monitored by IFC prior to 2006. IEG validations recommended that to improve client capacity in this area, further emphasis is needed on development and implementation of environmental and social management systems, E&S monitoring and reporting and full implementation of all gaps identified in Environmental and Social Action Plans (ESAPs) by the client. For projects implemented through financial intermediaries, additional efforts are needed on compliance verification of the subprojects’ Performance Standards, if they are triggered. IFC’s corporate governance advisory support can potentially be a cornerstone of external M4R (capacity development for enhancing client results), but results so far are mixed. IEG validations during this review period (particularly in Mongolia, Mozambique, and Lebanon) highlight the client’s right intentions at approval to enhance corporate governance systems and processes to manage for development results, but the outcomes and impact achievements are mixed due to client capacity, exogenous factors, and IFC work quality. 19 CHAPTER 1 MANAGING FOR RESULTS Umbrella programs, such as corporate governance in Latin America and Asia, were designed to strengthen the risk management and corporate governance policies and procedures of IFC client financial institutions to make them more resilient to crises and enhance their capacity to grow sustainably. The programs broadly achieved their objectives. However, evidence from the projects also shows inconsistent client implementation due to changing internal priorities and the lack of influence and a hands-on approach from IFC in improving corporate governance or linking it to potential investment opportunities. Client data systems and beneficiary results are intrinsically linked. To meet expectations on the results framework, IFC needs to scope such components in the project design when feasible. Clients’ weak management information system presents challenges to the further development of institutional capacity. MIGA DEVELOPMENT OF CLIENT CAPACITY MIGA has a limited role in client capacity development but an important role in the development of the political risk insurance industry. MIGA’s guarantees are usually only one element of a project’s financing and risk management package and are often requested when the project design is complete or nearly complete and when the project is close to financial closure. On the E&S performance aspects (on which MIGA influences clients like IFC), all MIGA-supported projects are reviewed at underwriting for capacity improvement potential through the Performance Standards framework. Post-issuance, the project company conducts an annual monitoring review and MIGA specialists validate it. Appropriate course corrections and recommendations for strengthening results at the client level are provided by the specialists based on the review. For example, in the oil and gas industry project in Uzbekistan, the MIGA team helped solve critical E&S issues by convening external industry experts. MIGA also supports the industry through its reinsurance business, which in 2013 adjusted MIGA frameworks to allow use of reinsurance at shorter tenors compared with those in its guarantee contracts. This change allowed MIGA to expand the panel of facultative reinsurers it uses to optimize reinsurance capacity for larger, longer-tenor projects that, in turn, facilitate scale-ups for MIGA clients.18 ADDRESSING M4R CAPACITY AT THE COUNTRY LEVEL At the country level, IEG examined 40 country strategies that contained ratings by objective to determine whether, and to what extent, the strategies incorporated M4R.19 The review revealed that some objectives clearly indicated an intent to manage for results—for example, “improve national statistical capacity” (Botswana) 20 CHAPTER 1 MANAGING FOR RESULTS or “update and increase the quality of social demographic information and agriculture data to better support decision making and the evaluation of public policies” (Bolivia). The objectives were most often part of an effort to improve public sector management— for example, “improved financial management, procurement, and results monitoring and evaluation capacity” (Honduras). About half of the country strategies included at least one M4R objective, and they collectively contained 32 M4R-related objectives. The majority of country strategies in the Latin America and the Caribbean Region (nine of 13) and the Middle East and North Africa Region (three of four) included M4R objectives. However, fewer included M4R objects in the East Asia and Pacific Region (one of two), Africa (three of nine), South Asia (one of three), and Europe and Central Asia (two of nine). Country strategies focused more on evidence-based policy making (13 objectives) than on data systems (12 objectives) and performance-based budgeting (7 objectives). However, higher- and upper-middle–income countries were more likely to pursue an objective related to evidence-based policy making or the use of data or evidence in decision making. Not surprisingly, the nature of the objective differed depending on the country circumstances and priorities. For example, “improving citizen security” was a key focus of the Honduras country strategy, which included support for developing municipal security plans that incorporated geo-referenced crime statistics under the objective “strengthen capacities of communities and local governments in social prevention of crime and violence.” Strengthening data systems was more of a priority in lower-income countries. Various World Bank instruments support M4R objectives. In Uruguay, the First and Second Programmatic Public Sector, Competitiveness, and Social Inclusion Development Policy Loans, and the Institution Building Technical Assistance Loan supported the objective to “introduce performance oriented budgeting.” All three loans included strengthening performance-based budgeting as part of a wider program to improve government effectiveness. However, the World Bank Group was not always successful in meeting its M4R objectives. IEG assessed only 12 of the 32 M4R-oriented objectives as achieved or mostly achieved. This is lower than the 51 percent average for all objectives. These results are preliminary and with small numbers, but availability of a larger pool would warrant a full assessment. 21 CHAPTER 1 MANAGING FOR RESULTS BOTTOM LINE: THE WORLD BANK GROUP PUT RELATIVELY MORE EMPHASIS ON MEASUREMENT THAN ON ADAPTIVE MANAGEMENT, AND IT WOULD BENEFIT FROM A REVIEW OF ITS APPROACH TO CLIENT CAPACITY This exploratory review presents an approximate baseline of the World Bank Group’s efforts at strengthening client’s M4R capacity. The emphasis on data systems, monitoring, and evaluation should be lauded for building the foundation for evidence-based decision making. The review also underscores the potential for doing more in this area, as indicated by cases that identified the need for strengthening client M4R capacity. Individual project-based approaches may not yield the dividends expected by themselves, and long-standing weaknesses recognized through years of staff experience may require more broad-based solutions. This is especially important given that 53 percent of the World Bank’s commitments are rated high or substantial for institutional risks, which includes the government’s capacity to achieve results (World Bank 2016u). An assessment of the effectiveness of actions taken to date would help the World Bank Group refine its approach to client M4R. Conclusions The World Bank Group has collectively addressed M4R with some success in the past 15 years, as have the individual institutions. Current evidence shows that efforts gave priority to building a system of measurement. However, results measurement itself needs further strengthening, particularly with respect to beneficiary-level results. Strengthening adaptive management and learning using robust evidence also requires attention. Several initiatives and reforms, and actions responding to recent IEG evaluations touching on this topic, are already under way. The World Bank Group supported client M4R capacity, particularly with respect to measurement. However, more attention is needed to enable learning and use of evidence for adaptive management (for example, for mid-course corrections in programs). The World Bank Group is a demand-driven organization. However, there may be space to address M4R strategically and practically from the supply side as well. IFC and MIGA have strong potential to improve M4R for repeat clients and in new products. World Bank Group strategy and Forward Look (World Bank 2013, 2016a) both strongly emphasize results and evidence. The World Bank Group thus may consider a self-review of its M4R efforts (including all ongoing reforms and actions) to develop a holistic and concrete action plan at the corporate, country, project, and client levels with a view to its purposeful implementation. A review of several IEG evaluations referenced that provide specific recommendations could help guide this 22 CHAPTER 1 MANAGING FOR RESULTS agenda (also see Chapter 3). This would allow the Board and stakeholders to benchmark progress against World Bank Group’s own goals in this area. 1See for example: http://www.gao.gov/products/GAO-17-2SP; http://web.worldbank.org/archive/website00819C/WEB/PDF/CASE_-62.PDF; https://www.nao.org.uk/wp-content/uploads/2010/07/1011284es.pdf. 2Download the text of the Busan Partnership for Effective Development Co-operation at http://www.oecd.org/dac/effectiveness/49650173.pdf 3The Strategic Compact, launched in April 1997, was an agreement between the World Bank and its shareholders to invest in additional resources for a three-year period to deliver and respond more effectively—through a fundamentally transformed institution—in reducing poverty. 4A forthcoming IEG evaluation on World Bank Group effectiveness in supporting data for development will cover this topic in depth. 5World Bank Group Strategic Actions Program for Addressing Development Data Gaps: 2016-2030. 6For more information, see Common Performance Assessment System reports at http://www.mfdr.org/COMPAS/index.html. 7The SMART criteria for setting objectives are specific, measureable, achieveable, realistic, and time-based. Learning about the theory of change requires tracing key inputs, outputs, and intermediate outcomes. 8Some World Bank Group services, such as advisory services and analytics, were not systematically evaluated and are not reported in the scorecard. 9RAP 2015 found that “Indicators used in country strategies and projects were generally inadequate to capture gender results. When present, indicators were narrow in scope and tended to measure outputs instead of outcomes.” Although the analysis looked for PDO indicators that captured gender gaps and their evolution, gender inequality measures, or gender biases, few such indicators were found in use. More often, projects used core sector indicators, numbers of female beneficiaries, or sex-disaggregated person-level project indicators. 10The revised gender tag procotol, whose effect on operations remains to be seen, included clarification of related guidelines. The new gender tag is required at Project Concept Note Stage for all IBRD/IDA projects approved since July 2016. The new gender tag aims to identify projects that are addressing (or can address) gender gaps identified in the Country Partnership Framework, in a specific sector, or one of the four outcomes of the Gender Strategy. Guidelines specify that for the project to be appropriately tagged, the Project Appraisal Document must include a clear results chain linking gender gaps identified in analysis, with actions to close them, and relevant indicators in the results framework. 11Private sector development rating is assigned to a subindicator under the overall development outcome indicator in IFC investment projects. 23 12IEG evaluated three Non-Honoring of Financial Obligations projects in FY14 and FY15; one of them was not rated because the guarantee was cancelled. 13The “Safeguard and Fiduciary Compliance” section of the World Bank’s 2014 harmonized Implementation Completion and Results Report Guidelines contain the following guidance in the section on Safeguard and Fiduciary Compliance: “Summarize key safeguard and fiduciary issues in the operation, compliance with the World Bank policy and procedural requirements, and any problems that arose and their resolution, as applicable,” and “Record any significant deviations or waivers from the World Bank safeguards/fiduciary policies and procedures.” A list of the related OP/BP is also provided in the guidance. 14IFC Industry departments have dedicated knowledge activities as well through MAS Talks, FIG Knowledge Bytes, and Infra PostVivems. 15 The framework developed for this report is based on OECD-DAC (2008), page 6. 16For more information, see the Operations Policy and Country Services (OPCS) Harmonized list of Fragile Situations FY15 at http://siteresources.worldbank.org/EXTLICUS/Resources/511777- 1269623894864/FY15FragileSituationList.pdf. 17Thirty-two out of the 229 investment projects in the FY13-15 cohort were rated below satisfactory on environmental and social (E&S) effects; 10 out of the 32 were rated specifically for E&S Action Plan compliance shortcomings. 18 Facultativeinsurance is reinsurance for a single risk or a defined package of risks. The ceding company (the primary insurer) is not compelled to submit these risks to the reinsurer, but the reinsurer is not compelled to provide reinsurance protection, either. 19The World Bank Group outlines the various objectives it plans to achieve through its country strategies and prepares a self-assessment at the end of the strategy period, the Country Assistance Strategy Completion Report (CASCR), or completion and learning review (CLR). IEG validates the CASCR or CLR and provides a separate assessment (the CASCR or CLR review). IEG has been rating the level of achievement for each objective since FY14, providing insights into which objectives were met and which were not. Ratings are available for 618 World Bank Group country objectives in 40 countries. This review examined the CASCR and CLR reviews for these 40 countries to determine to what extent M4R was a factor in the strategies and how the World Bank Group promoted M4R through its country programs using the same framework as the projects’ analysis. 24 2. Recent Performance of World Bank Group Operations Highlights  At 72 percent moderately satisfactory or above, World Bank project outcome ratings remained stable overall. Only a few projects with very large volume drove an increase in volume-weighted outcome ratings from 81 percent in FY10–12 to 87 percent in FY13–15.  The South Asia Region and the Social Protection Global Practice had higher outcome ratings, compared with other regions and global practices.  IFC development outcome success ratings continued the broader downward trend in outcomes since the financial crisis in 2008. Financial Institutions Group performance had a slight uptick from prior years, but other industries saw drops in success rates.  IFC work quality and IFC net commitments to the projects are the primary drivers of project outcome ratings in the evaluated period, but limited data capture and insights on beneficiary- level results highlight a strand of weakness. IFC work quality success rate also declined.  At 66 percent moderately satisfactory or above, country development outcome ratings in FY13– 16 remained below the FY17 target of 70 percent. At 65 percent, country-level World Bank performance ratings also remained below the FY17 target of 75 percent.  In regions other than Africa, most countries that had cycled through at least two country strategies during FY07–16 maintained a satisfactory development outcome rating, or improved from unsatisfactory to satisfactory. However, ratings remained unsatisfactory, or declined from satisfactory to unsatisfactory across country strategy cycles for a large group in Africa. World Bank Group Commitments World Bank Group commitments increased moderately from FY15 to FY16 (figure 2.1). IBRD lending increased from $23.5 billion in FY15 to $29.7 billion in FY16, but IDA commitments decreased from $19 billion to $16.2 billion. The overall number of projects approved per year has declined since FY14, a trend attributable to a growing presence of large-volume projects from FY13 to FY16. During this time, projects with commitment of $500 million or more rose from 20 percent of the total volume of new commitments to 44 percent while constituting only 3 percent to 10 percent of the total number of new projects (appendix C, figure C.1). Development policy financing (DPF) and Program-for-Results (P4R) led the overall increase of lending commitment volume during FY15–16, but the volume of investment project financing (IPF) decreased. DPF commitments rose from $9.2 billion to $14.7 billion, and P4R continued to expand from $2.2 billion to $7.0 billion, 25 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS and IPF commitments decreased from $30.5 billion to $23.8 billion. The number of new DPF and IPF projects decreased, and the number of new P4R projects increased. The average commitment per project for DPF increased from $164 million to $319 million, but dropped from $123 million to $109 million for IPF (appendix C, figure C.1). Figure 2.1. World Bank Group Commitments Increased between FY13 and FY16 a. IBRD and IDA commitments combined increased for the fourth year in a row, although the number of projects decreased 80 500 IDA IBRD N Projects 400 60 300 $, billions Number of projects 40 200 20 100 0 0 FY02 FY04 FY06 FY08 FY10 FY12 FY14 FY16 Fiscal year b. IFC commitments continued to increase in 2016 14 12 10 $, billions 8 6 4 2 0 2002 2004 2006 2008 2010 2012 2014 2016 Calendar year c. MIGA guarantees increased in 2016 6 Traditional products New 4 $, billions 2 0 2002 2004 2006 2008 2010 2012 2014 2016 Calendar year Source: World Bank Business Intelligence; IFC and MIGA databases. Note: Commitments for IFC exclude mobilization. IFC began reporting average outstanding short-term commitments (not total commitments) in FY15 and no longer aggregates short-term commitments with long-term commitments. 26 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS IFC commitments in FY16 increased by 4 percent compared with the previous year to reach $17.8 billion (long-term and short-term commitments combined). FY16 was the second-best year for IFC in volume after FY13 ($18.4 billion). IFC’s long-term commitments stood at $11.7 billion. Furthermore, IFC mobilized nearly $7 billion from private capital sources. MIGA issued $4.3 billion of guarantees in FY16 to support investors and help them mitigate risk. Projects spanned all regions and sectors, and 59 percent of projects were in at least one of MIGA’s four priority areas. At the end of the year, MIGA’s outstanding gross exposure was $14.2 billion, of which 45 percent was in IDA- eligible countries and 10 percent in fragility, conflict, and violence (FCV) countries. Of this $14.2 billion, MIGA ceded $7.5 billion to its reinsurance partners. The next sections discuss performance trends observed in three member institutions of the World Bank Group. Findings should be interpreted with the understanding that the World Bank, IFC, and MIGA differ in their operating models and in the time frames generally applied to their projects. IFC investment projects are sampled, evaluated, and reported for calendar year (CY). IFC Advisory Services and MIGA projects are evaluated and reported on a fiscal year (FY) basis. Performance trends below are measured by project ratings based on tightly structured review of completion documents. In this analysis, IEG mentions increases or decreases in ratings and other differences across groups only if it found the difference to be statistically significant. However, exceptions are in the MIGA discussion (MIGA has too few project ratings in a single year), especially for a new product line (non-honoring guarantees), and in appendixes I and J (the regional and global practice cluster updates). World Bank Project Performance The overall performance of World Bank projects improved moderately among projects closed during the FY13–15 period. The percentage of projects with outcome ratings of moderately satisfactory or above (MS+) remained essentially the same at 72 percent for FY13–15 and 70 percent for the FY10–12 period, below the corporate target of 75 percent by FY17.1 Weighted by volume, the percent MS+ rose from 81 percent for FY10–12 to 87 percent for FY13–15, above the FY17 corporate target of 80 percent (see figure 2.2). IEG reported outcome ratings here for comparison with past RAP reports; however, outcomes are not entirely within the World Bank’s control. The World Bank performance rating, on the other hand, assesses the extent to which services the World Bank provided ensured the operation’s quality and 27 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS supported effective implementation, elements that are within the World Bank’s control. Figure 2.2 also includes this rating’s trend. The volume-weighted percentage of projects with World Bank performance rated MS+ rose from 78 percent for FY10– 12 to 87 percent for FY13–15. In the future, IEG expects to determine the best way to analyze and report outcomes and World Bank performance, based on ongoing discussions regarding reforming the World Bank’s self-evaluation system and incentivizing evidence-based learning. Figure 2.2. Volume-Weighted Percentage of Projects Rated MS+ on Outcome and World Bank Performance Improved from FY10–12 to FY13–15 a. World Bank lending projects with outcome rated moderately satisfactory or above a. World Bank lending projects with outcome rated moderately satisfactory or above 100 Outcome, volume-weighted, by net commitment ($) Outcome, unweighted, by number of projects (N) Percentage 87 80 72 60 Fiscal year, three-year rolling b. World Bank lending projects with World Bank Performance rated moderately satisfactory or above b. World Bank lending projects with World Bank Performance rated moderately satisfactory or above 100 World Bank performance, volume-weighted, by net commitment ($) World Bank performance, unweighted, by number of projects (N) Percentage 87 80 72 60 Fiscal year, three-year rolling Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: IEG rated 791 projects that closed in FY13–15 with a total net commitment (volume) of $77.3 billion, among which 697 were IPF projects with net commitment totaling $58 billion, and 80 were DPF projects with net commitment totaling $18.6 billion. DPF = development policy financing; IPF= investment project financing; MS+ = moderately satisfactory or above. 28 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS The performance of a few projects with very large volume drove the increase in the volume-weighted percent MS+. Among the FY13–15 group, 15 projects with net commitment of $800 million or more accounted for 1.9 percent of the projects and 24 percent of the volume.2 IEG rated all 15 of these projects MS+ on outcome. Among the FY10–12 group, 12 projects had a net commitment of $800 million or more, accounting for 1.2 percent of projects and 17 percent of volume. IEG rated 11 of the 12 projects MS+ and the remaining project (P095205, with a net commitment of $1.3 billion) unsatisfactory on outcome. Appendix C, table C.1 lists projects with volumes of $800 million or more that closed during FY10–12 and FY13–15. IEG sorted countries by volume across projects closed in both the FY10–12 and the FY13–15 groups, and 10 countries were in the top 50 percent of volume in either group: India, Mexico, Turkey, Brazil, Indonesia, China, Poland, Colombia, Vietnam, and Argentina. Almost all are IBRD countries, and any volume-weighted percentage reported is influenced by project performance in these countries. Appendix C presents the project ratings over time for these 10 countries (all very large volume) (figure C.3) as well as for the more than 100 countries that make up the bottom 50 percent of volume (figure C.4). Project outcome ratings (percent MS+) for DPFs were higher than for IPFs in the FY13–15 period when measured by number of projects, and in FY10–12 when measured by commitment volume. Note, however, that the outcome rating methodology differs. For IPFs, efficiency is one component of the outcome rating (the others are relevance and efficacy), and for DPFs, only relevance and efficacy inform the outcome rating. Volume-weighted outcome ratings for IPFs improved from FY10–12 to FY13–15 (figure 2.3). IBRD-funded operations had higher ratings than IDA projects as measured by volume in both periods (FY10–12 and FY13–15). Volume-weighted outcome ratings for IDA improved during the two periods. This pattern likely relates to the observation that the few projects with very large volume received high ratings (and especially so in FY13–15), and the average project volume for IDA increased from $58.1 million in FY10–12 to $65.4 million in FY13–15. Projects in FCV countries had lower outcome ratings (when weighted by volume) than did projects in non-FCV countries, in both periods. For projects in both FCV and non-FCV countries, volume-weighted outcome ratings increased from FY10–12 to FY13–15. Furthermore, the average size of projects in FCV situations increased from net commitment of $38.4 million in FY10–12 to $45.9 million in FY13–15, and 29 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS correlation between project size and outcome ratings was stronger in FY13–15 than in FY10–12. Figure 2.3. Notable Gains and Gaps Are Found in Outcome Ratings a. Among IPFs, volume-weighted outcome ratings increased from FY10–12 to FY13–15. DPFs had higher unweighted outcome ratings than IPFs in FY13–15, and volume-weighted ratings were about equal. BY NUMBER OF PROJECTS BY VOLUME 88% 83% DPF 83% 86% DPF 75% IPF 76% 71% IPF 68% FY10-12 FY13-15 FY10-12 FY13-15 b. IBRD projects had higher volume-weighted outcome ratings than IDA projects across the two periods, but the gap decreased. Volume-weighted outcome ratings for IDA projects improved from FY10–12 to FY13–15. BY NUMBER OF PROJECTS BY VOLUME 89% IBRD 84% IBRD 74% 83% 74% 73% IDA IDA 69% 68% FY10-12 FY13-15 FY10-12 FY13-15 c. Non-FCV countries had higher volume-weighted project outcome ratings than FCV countries in both periods. Volume-weighted ratings improved in both FCV and non-FCV countries. BY NUMBER OF PROJECTS BY VOLUME 87% non-FCV 80% non-FCV 71% 73% 77% FCV 67% FCV 64% 65% FY10-12 FY13-15 FY10-12 FY13-15 Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: Fiscal year is the project’s closing fiscal year. DPF = development policy financing; FCV = fragility, conflict, and violence; IPF= investment project financing. The dashed boxes indicate a statistically significant difference. 30 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS OVERVIEW OF REGIONAL PERFORMANCE Project performance in South Asia was higher than that in other regions in FY13–15 (82 percent rated MS+), similar to the performance reported in RAP 2015. Projects in the Africa and the Middle East and North Africa Regions in the same period had lower ratings than other regions. It is worth noting that this cohort of projects was active during a time of massive political transition and violent conflict in many countries in the Middle East and North Africa. The Africa Region also had lower outcome ratings than other regions in FY10–12.3 Figure 2.4. World Bank Project Ratings Varied across Regions a. Outcome FY10-12 FY13-15 Bank-wide South Asia Europe and Central Asia East Asia and Pacific Latin America and the Caribbean Africa Middle East and North Africa 10 20 30 40 50 60 70 80 90 Percentage rated MS+ B. M&E QUALITY FY10-12 FY13-15 Bank-wide South Asia Europe and Central Asia East Asia and Pacific Africa Latin America and the Caribbean Middle East and North Africa 10 20 30 40 50 60 70 80 90 Percentage rated MS+ Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: For regions that appear to only have a dot for FY13–15: In the outcome graph in the East Asia and Pacific Region, the percent of projects with outcome rated MS was about equal for the FY10–12 group and the FY13–15 group. indicates the difference in ratings was significant across the two time periods for this region. Both the South Asia and Africa Regions improved their M&E quality ratings from FY10–12 to FY13–15 (figure 2.4).4 In South Asia, projects in the Social Protection and Labor Global Practice and in the Health, Nutrition, and Population practice helped drive this improvement. In Africa, projects in Water and in Health, Nutrition, and Population partly drove the improvement. 31 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS OVERVIEW OF GLOBAL PRACTICE PERFORMANCE Social Protection and Labor again ranked first among Global Practices during FY13– 15, with 90 percent of projects rated MS+ on outcome (figure 2.5), was the top Global Practice in M&E quality, and ranked second in quality at entry and quality of supervision (after Macroeconomics and Fiscal Management). RAP 2015 noted good design, supervision, and attention to the evidence base as factors related to performance of Social Protection and Labor projects.5 Figure 2.5. Outcome Ratings Varied across Global Practices, with Increased Ratings Noted for Agriculture and Education FY10-12 FY13-15 Bank-wide Social Protection and Labor Macroeconomics and Fiscal Management Transport and ICT Agriculture Education Finance and Markets Health, Nutrition and Population Social, Urban, Rural, and Resilience Water Energy and Extractives Trade and Competitiveness Environment and Natural Resources Governance 40 50 60 70 80 90 100 Percentage rated MS+ Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: The number of projects with IEG ratings in each period is in appendix J. indicates the difference in ratings was significant across the two time periods for the Global Practice. The Poverty Global Practice is excluded from this figure because there were fewer than ten projects with ratings across both periods. The one Poverty project that closed during FY10–12 had a rating of MU. Among the 7 Poverty projects closed in FY13–15 with IEG ratings, 5 were rated MS+. Other Global Practices with outcome ratings exceeding the corporate target of 75 percent by FY17 include Agriculture (79 percent), Macroeconomics and Fiscal Management (82 percent), and Transport and ICT (79 percent). Agriculture and Education saw improvements in percent MS+ from FY10–12 to FY13–15, mainly driven by projects in the Africa Region. Governance, with 56 percent MS+ outcome ratings (for projects closed in FY13–15) had lower ratings than other Global Practices. 32 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Previous IEG work supports the common-sense hypothesis that careful design, attentive supervision, country capacity, and clearly articulating project logic are important for achievement of outcomes.6 IEG’s RAP 2015 found that for IPFs, project performance correlated with quality at entry, quality of supervision, M&E quality, and project size (to a much lesser extent). However, it was the change in project size instead of the initial project size drove performance (as measured by project outcome rating). The underlying dynamic this suggests is that projects that perform well are more likely to receive additional financing instead of project size driving performance. Furthermore, projects in countries with larger population and higher Country Policy and Institutional Assessment (CPIA) ratings were associated with higher ratings. Projects that were ever labeled a problem project and projects with more task team leaders during the project were associated with lower ratings. IEG’s RAP 2014 found that application of past lessons, effective risk mitigation, and well-articulated project objectives and results frameworks were associated with higher ratings. SHORTCOMINGS IN UNSATISFACTORY PROJECTS A review of 41 projects rated moderately unsatisfactory or below (MU−) out of the 199 projects completed in FY15 with IEG ratings available as of September 7, 2016, found that all projects showed evidence of significant, major, or severe shortcomings in relevance, efficacy, or efficiency.7 Furthermore, most (61 percent) mentioned significant, major, or severe shortcomings in all three elements.8 In 23 of the 41 ICRRs (more than half), the low rating also corresponded to inadequate evidence to substantiate the outcome rating. For 18 out of these 23, the evidence presented in the ICR was not sufficient to substantiate achievement of at least some objectives (that is, efficacy), and five of those had insufficient evidence for all objectives (appendix D provides more details). GENDER INTEGRATION IN PROJECTS A pilot analysis found that gender was an explicit part of the project development objective (PDO) or at least one of its components in 38 percent of projects reviewed for this analysis (60 of 156 projects).9 This is similar to the 43 percent finding in RAP 2015.10 Of the 60 projects, 48 of the projects (80 percent) reported some sex- disaggregated or male- or female-specific indicators at some level in the ICR, or one of the indicators was the share or percentage of female beneficiaries. Gender was an explicit part of the PDO in 11 projects. Among these, 10 projects reported on gender indicators at the PDO level in the ICR (appendix E provides more details). 33 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Trends in Outcome Ratings for IFC Investment and Advisory Projects Outcome ratings for IFC investment projects, at 54 percent, are down by 3 percentage points from the previous year and continue to trend downward (figure 2.6) Among the four subindicators of development outcome, drop in success rates was lowest for the indicator on environmental and social (E&S) effects. IFC work quality and net commitment to projects were the primary drivers of development outcomes (appendix F, table F.1). Figure 2.6. Development Outcome Ratings for IFC Investment Projects Decreased from 2009 to 2015 80 Financial crisis Percentage 60 40 20 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 Calendar year Source: IEG data for IFC project ratings. Note: The graph represents the share of projects with development outcome rated mostly successful or higher on a three- year rolling average by calendar year. One project rated No Opinion Possible has been excluded from the trends. For FY15, evaluation completion rate for investment projects is 99 percent. Sub-Saharan Africa performed best among the regions in the review period while the performance of the other five regions trended down (figure 2.7.) Projects in Sub- Saharan Africa were backed by financially strong sponsors who could withstand the market downturns. This cohort of projects was concentrated more on financial institutions. IFC projects in the Middle East and North Africa Region had the biggest drop in success rates for the review period compared with the prior three-year average, plunging nearly 14 percentage points, driven by business planning failures and unexpected regulatory interventions. IFC performance (work quality) in the Middle East and North Africa Region also experienced significant drop, compared with other regions. Four of the six regions experienced strong growth in IFC profitability (that is, Investment Outcome), except for Latin America and the Caribbean and East Asia and Pacific Regions. 34 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.7. Development Outcome Ratings for IFC Investment Projects Varied among Regions; Sub-Saharan Africa Development Results Trended Up Sub-Saharan Africa East Asia and Pacific Europe and Central Asia 100 80 Percentage 60 40 20 0 Latin America and the Caribbean Middle East and North Africa South Asia 100 80 Percentage 60 40 20 0 Source: IEG data for IFC project ratings. Note: The graphs represent the share of projects with development outcome rated mostly successful or higher. Development outcome ratings for real sector projects dropped six percentage points, and financial sector projects moved up by one percentage point.11 Greenfield projects were not as successful as expansion projects, driven mainly by project business success and IFC work quality (figure 2.8). The success rates of IDA (and IDA plus non-IDA blended, 63 in total) projects showed a positive increase of 3 percent, edging closer to the long-term IFC average. Projects with more than one IFC instrument (loan and equity) performed better than loan-only and equity-only projects on both development outcome and IFC performance. Performance of equity-only projects (64 evaluated in the review period) reached a 10-year low. Figure 2.8. Development Outcome Ratings for IFC Investment Projects Varied across Industries; FIG Performance Was Up but CTT Dropped Below 40 Percent Financial Institutions Group Infrastructure and Natural Resources Manufacturing, Agribusiness Telecom, Media, Technology, and and Services Venture Investing 100 80 Percentage 60 40 20 0 Source: IEG data for IFC project ratings. Note: The graphs represent the percentage of projects rated in each calendar year (three-year rolling average) with development outcome rated mostly successful or higher. 35 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Infrastructure and Natural Resources (INR). Project design, structuring, and adverse external factors such as geopolitical instability, regulatory changes, and commodity price fluctuations led to less-than-satisfactory development outcomes, despite strong appraisal and structuring from IFC. The main lesson from this portfolio subset is that teams should consider ex ante downstream assessments12 when structuring projects and factor for conservative estimates on business growth and economic returns. IFC profitability was higher than the prior review period and was the highest among the four industries. Financial Institutions Group (FIG). There was an uptick in the performance of FIG projects. In underperforming projects, teams identified overall risks to the outcomes, but proposed mitigation mechanisms did not work as expected primarily due to macroeconomic risks. Project appraisals validated by IEG indicate insufficient scenario analyses and sensitivity tests (more than 50 percent in the cohort), so when unexpected events happened, there was no contingency plan. Manufacturing, Agribusiness, and Services (MAS). Supporting south-south investments through engagement with repeat clients was a positive aspect for projects in this cohort but performance was below the prior review period. Project design assumptions for growth, regulatory support, and expected demand did not match reality. This was particularly true for greenfield initiatives with project sponsors in countries where the sponsor had limited experience, as was the case in 23 out of the 45 greenfield projects in the cohort. Market diversification strategies of IFC clients made sense on paper, but project execution faltered because of poor resource allocation or lack of adaptability in the target market. Telecom, Media, Technology and Venture Investing (CTT). Development outcome success rate of CTT projects continued to lag other industry groups at 40 percent. This is in part due to IFC support in the post-crisis environment of 2008-2010 of innovative business models and first-time fund managers. A significant portion of these projects did not meet the market test, lacked the financial viability to fund scale-up plans or develop sufficient project pipeline and exits. More than two-thirds of the projects were rated less than satisfactory for project screening, appraisal, and structuring aspects. IFC Advisory Services. The development effectiveness success rating for IFC Advisory Services slipped from 65 percent to 61 percent (three-year rolling average), and is below the IFC Corporate Scorecard target of 65 percent.13 Marginal dips in performance (3–5 percent) occurred across all performance indicators, most notably on indicators of IFC role and contribution and the advisory interventions’ design 36 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS quality.14 Based on weights by funding levels for advisory projects, the three-year rolling average success rate dropped from 70 percent to 67 percent. IEG validations show evidence of postcompletion monitoring and assessment in a few cases conducted by IFC in a purposefully sampled way. Nevertheless, little information is available on the impacts of IFC delivery of advisory services on long- term, sustainable private sector development. As IEG presented in the previous chapter, IFC may be missing an opportunity to prove its value and capture the impacts by not measuring the effects on client capacity and client results through M&E and data capture. An emerging lesson on this aspect is to assess clients’ capacity regarding management information systems and information technology as part of the M&E framework analysis. Trends in Outcome Ratings for MIGA Projects MIGA development outcomes. To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating.15 A rating of NOP implies that it is not possible to make conclusions on the development effectiveness of the project evaluated. Seven projects in the review period (FY10–15) were rated as NOP for development outcomes due to the paucity of project performance data. Out of the seven projects that were rated NOP, four had been cancelled and three remained active at the time the evaluation commenced.16 On a six-year rolling average basis, excluding projects rated NOP from the total evaluated, development outcomes of MIGA projects remained nearly steady at 61 percent (figure 2.9a). This indicator, including NOP in the total of evaluated projects, declined by 4 percent from the prior review period (FY09–14). MIGA self-evaluation and IEG validation of three non-honoring guarantee projects showed mixed development outcome results in the review period. One consistent finding is that the underwriting methods, monitoring, and supervision of the project credit risk, obligor risk profile, and overall performance reporting in self-evaluations need strengthening. Furthermore, IEG suggests that MIGA assess the risk to development outcome at underwriting stage and in self-evaluations explicitly. MIGA Regional performance. In the recent three-year review period, the Middle East and North Africa Region was the best performer (Figure 2.9b). East Asia and Pacific and Latin America and the Caribbean Regions had no successful projects while the South Asia Region had no evaluated projects in the review period. Europe and Central Asia had relatively higher instances of projects rated as NOP mainly due to the paucity of project performance data. 37 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.9. MIGA Development Outcome Ratings for MIGA Projects Remained Steady over a Six-Year Period a. MIGA development outcome b. Regional performance (FY13-15) Satisfactory+ Unsatisfactory- No opinion possible South Asia N/A N=0 FY09-14 N/A N=0 63% 38% N=56 59% 36% 5% N=59 Latin America and 100% N=1 Caribbean 100% N=1 FY10-15 61% 39% N=64 55% 35% 10% N=71 East Asia Pacific 100% N=3 75% 25% N=4 Sub-Saharan Africa 64% 36% N=11 58% 33% 8% N=12 FY12-14 65% 35% N=26 59% 31% 10% N=29 Europe and Central Asia 80% 20% N=10 53% 13% 33% N=15 FY13-15 63% 37% N=27 Middle East and North 100% N=2 50% 29% 21% N=34 Africa 100% N=2 Source: IEG data for MIGA project ratings. Note: Percentages may not add up to 100 because of rounding. Percentages are the share of MIGA projects during the fiscal years that had a rating of mostly successful or above. A rating of No Opinion Possible is given when a development indicator lacks appropriate and sufficient performance data at the obligor, project, or beneficiary level. In figure 2.9b, NA denotes Not Applicable, i.e., no projects were evaluated or validated in the review period. See endnotes 15 and 16 for methodological details. Country-Level Performance This section considers the performance of the World Bank Group’s strategy and approach within each country as defined in the country partnership framework (CPF) or the country assistance strategy (CAS), if the strategy started before implementation of the CPF. As stated in the CPF section of the World Bank’s Open Knowledge Repository, the CPF, prepared in collaboration with a member country and starting from the country’s own vision of its development goals, “identifies the key objectives and development results through which the World Bank Group intends to support a member country in its efforts to end extreme poverty and boost shared prosperity in a sustainable manner.”17 It includes a proposed package of investments and knowledge services (covering the World Bank, IFC, and MIGA, as appropriate) needed to implement the World Bank Group’s strategy. A CPF also draws on a systematic country diagnostic and places greater emphasis on learning. Table 2.1 shows the number of reviews during FY07–16. 38 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Table 2.1. Number of Countries with CPFs and CASs Reviewed by IEG, FY07–16 Countries (number) Region Total Total With one With two With three countries reviewed review reviews reviews Sub-Saharan Africa 48 33 13 20 0 East Asia and Pacific 23 10 7 2 1 Europe and Central Asia 23 21 1 14 6 Latin America and the Caribbean 30 25 1 21 3 Middle East and North Africa 12 8 5 3 0. South Asia 8 6 1 4 1 Total 144 103 28 64 11 Source: IEG. Note: Total number of countries is based on the list of IBRD, IDA, and blend countries from Operations Policy and Country Services Operational Policies 3.10, Appendix F (effective July 1, 2016). Six OECS countries were reviewed together. The number of countries reviewed includes those that IEG reviewed, but without ratings. CAS = country assistance strategy; CPF = country partnership framework; OECS = Organisation of Eastern Caribbean States. World Bank Group country outcomes remain below the FY17 corporate target of 70 percent. Across the World Bank Group, IEG rated 66 percent of country programs MS+ for development outcomes during FY13–16. IBRD countries continue to perform above the corporate target, and IDA countries (which include IBRD-IDA blend countries18) continue to perform below the target (figure 2.10). Ratings of FCV countries fluctuate more, mostly because of the small number of countries rated in each period, which ranges from three to eight. IEG used an average for the four-year period because these ratings fluctuate annually, reflecting the variance in the number of Country Assistance Strategy Completion Reports (CASCRs) and completion and learning reviews (CLRs) reviewed across fiscal years.19 Figure 2.10. Country Development Outcome Ratings Exceed Target in IBRD Countries, but Not in IDA and FCV Countries Share of CLRs with country development outcome rated MS+ 90 IBRD Percentage 70 Target Bank-wide FCV 50 IDA 30 FY07-10 FY08-11 FY09-12 FY10-13 FY11-14 FY12-15 FY13-16 (N=69) (N=66) (N=68) (N=70) (N=74) (N=74) (N=74) Source: IEG completion and learning review ratings data. Note: The black line represents the World Bank Group target of 70 percent of CLRs rated MS+ on country development outcome. CLR = completion and learning review; MS+ = moderately satisfactory or above; FCV = fragility, conflict, and violence; N = number of CLRs. Six OECS countries were reviewed together with a single rating. 39 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.11. Country Strategy Development Outcome Ratings in Three Regions Exceeded the Target during FY07–16 Share of CLRs with country development outcome rated MS+ FY07-16 FY13-16 40% Target: 70% 40% Target: 70% AFR 46% N= 52 44% N=25 MNA 45% N=11 50% N=4 EAP 77% N=13 67% N=6 LCR 71% N=42 76% N=17 SAR 67% N=12 80% N=5 ECA 74% N=46 88% N=17 Bank-wide 63% N=176 66% N=74 Source: IEG completion and learning review ratings data. Note: Dotted lines represent the World Bank Group FY17 target of 70 percent. The numbers outside of the bars indicate the total number of CLRs completed for countries in each region (the denominators for the percentages). AFR = Sub-Saharan Africa; CLR = completion and learning review; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; N = number of CLRs; SAR = South Asia. Six OECS countries were reviewed together with a single rating. East Asia and Pacific, Europe and Central Asia, and Latin America and the Caribbean Regions show country development outcomes that exceed the corporate target. IEG rated more than 70 percent of the country programs in these regions MS+ during FY07–16, which is at or above the corporate target (figure 2.11). Both the Africa and Middle East and North Africa Regions performed below the corporate target. In the more recent period, FY13–16, country development outcomes of the Latin America and the Caribbean, South Asia, and Europe and Central Asia Regions is above the corporate target. It is important to note that the number of strategies evaluated varies over time and between Regions and performance in a few countries can affect overall performance. IEG also found that during a period of two or more country strategy reviews, CASCRs and CLRs for 17 countries in five World Bank regions (except for East Asia and Pacific) improved from unsatisfactory to satisfactory. However, CASs and CLRs for 12 countries in the Africa and Europe and Central Asia Regions declined from satisfactory to unsatisfactory, and eight in Africa and three in other regions stayed at the unsatisfactory level, primarily because country programs were too optimistic (figure 2.12). 40 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.12. Countries with More Than One Country Strategy during FY07–16 Show Varied Pattern of Change in Country Development Outcome Rating Number of countrya strategy changes in which development outcome rating: Remained MS+ Improved MU- to MS+ Declined MS+ to MU- Remained MU- AFR 5 1 6 8 EAP 4 ECA 12 8 5 LCR 13 5 1 3 MNA 1 1 1 SAR 3 2 1 Source: IEG calculations. Note: AFR = Sub-Saharan Africa; CLR = completion and learning review; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; MS+ = moderately satisfactory or above; MU− = moderately unsatisfactory or below; SAR = South Asia. Six OECS countries were reviewed together with one rating. Countries with consecutive MU− strategies are Bolivia, Djibouti, The Gambia, Honduras, Lesotho, Malawi, Mauritania, Niger, Pakistan, Paraguay, South Africa, Uganda, and Zambia. At 66 percent, CLR review ratings of World Bank Group performance was below the target of 75 percent good or better in FY13–16 (figure 2.13). While it was 78 percent for FY07–10 and 76 percent for both FY08–11 and FY11–14, the performance was also lower than the target in FY12–15 at 69 percent.20 Disaggregation by IBRD and IDA countries shows that performance was below the World Bank Group’s target for FY13–16 in both groups.21 Although performance in IBRD countries was above the FY17 corporate target in some periods, performance in IDA countries remained below the target, and the two converged in FY12–15 and FY13–16. The self-assessed performance ratings are higher than IEG’s ratings—the World Bank Group’s self- assessment was 90 percent for FY12–15 and 85 percent for FY13–16.22 The fluctuation of IEG ratings over time reflects the variation in countries rated in each time period. An examination of the most recent three periods (FY11–14, FY12–15, and FY13–16) found that higher performance ratings are associated with stronger country capacities, as measured by CPIA and other related indicators.23 41 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.13. Percentage of Country Strategies with World Bank Group Performance Rated MS+/Good+ Varied 100 90 Bank-wide 80 IBRD Percentage 70 IDA 60 FCV 50 Target 40 30 FY07-10 FY08-11 FY09-12 FY10-13 FY11-14 FY12-15 FY13-16 (N=69) (N=66) (N=68) (N=70) (N=74) (N=74) (N=74) Source: IEG completion and learning review ratings data. Note: The black line represents the World Bank Group target of 70 percent of CLRs rated MS+ on World Bank Group performance. CLR = completion and learning review; MS+ = moderately satisfactory or above; FCV = fragility, conflict, and violence; N = number of CLRs. Six OECS countries were reviewed together with a single rating. Figure 2.14. In Most Regions, the Percentage of CLRs with World Bank Performance Rated Good+ or MS+ Was Below Target in FY13-16 FY07-16 FY13-16 Target: 75% Target: 75% 40% 40% LCR 69% N=42 59% N=17 AFR 62% N=52 60% N=25 SAR 75% N=12 60% N=5 EAP 85% N=13 67% N=6 MNA 73% N=11 75% N=4 ECA 80% N=46 82% N=17 Bank-wide 72% N=176 66% N=74 Source: IEG CLR ratings data. Note: Dotted lines represent the World Bank Group FY17 target of 75 percent of CLRs rated moderately satisfactory or above on country development outcome. AFR = Sub-Saharan Africa; CLR = completion and learning review; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; MS+ = moderately satisfactory or above; N = number of CLRs; SAR = South Asia. Disaggregation by regions shows that East Asia and Pacific, Europe and Central Asia, Middle East and North Africa, and South Asia achieved ratings above or just slightly below the corporate target of 75 percent for the review period FY07–16 (figure 2.14). However, Latin America and the Caribbean and Africa were below target at 69 percent and 62 percent, respectively. The difference between Africa and other regions is statistically significant. In the more recent period, only Europe and Central Asia and Middle East and North Africa are at or above the target. IEG performed two sets of analyses to understand the factors affecting World Bank Group country outcomes. First, IEG conducted a regression analysis to examine 42 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS which factors influence country development outcome ratings. Second, IEG reviewed the objectives articulated in the strategies to determine whether the set of objectives differs between countries with higher versus lower CPIA, given the relationship between country capacity and country development outcomes. Appendix G describes these analyses in more detail. The regression analysis found that government capacity24 (as measured by the CPIA rating), economic performance, per capita gross domestic product, World Bank Group program size, and the country portfolio quality correlate significantly with country development outcomes (statistically significant at the 5 percent). Among World Bank Group factors, lending program size (as measured by the log of the total number of ongoing projects at the start of the strategy period and projects approved during the strategy period), and the IEG project outcomes rating (the percentage of projects rated MS+ after project closing) were positively associated with country development outcome ratings. The percentage of projects at risk correlated negatively with the country development outcome ratings. Statistical significance was weak for the correlations with project risk ratings and with IEG ratings of project outcome—lending program size was statistically significant at the 5 percent confidence level, and projects at risk and IEG project ratings were significant at the 10 percent confidence level (appendix G provides details). These findings show that evidence on performance matters, but portfolio size may well be a proxy for the World Bank’s capacity to manage the program. Analysis in RAP 2015 found that country capacity (as measured through the CPIA rating) was associated with project performance. The current analysis shows that it is also directly associated with country strategy development outcome. As noted at the beginning of this section, country development outcome ratings assess the achievement of objectives in the country strategy, and project outcome ratings assess the project development outcome only. However, a small correlation exists between the two, as shown in table 2.2. Appendix H also contains a disaggregation of CLRs by country development outcome rating and percent of projects rated MS+ (figure H.1). 43 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Table 2.2. Country Development Outcome Ratings, Project Outcome Ratings, and Project Risk Ratings, FY07–16 Country development Average percentage of projects Average percentage of outcome rating with outcome rating MS+ projects at risk Satisfactory 93 10 Moderately satisfactory 78 16 Moderately unsatisfactory 73 21 Unsatisfactory 67 25 Source: IEG calculations. Note: Data are based on World Bank projects referenced in completion and learning reviews completed in FY07–16. The World Bank Group engages in a wide range of activities in its country programs that contribute to the objectives articulated in its country strategy. The World Bank Group and IEG began rating the achievement of country strategy objectives and using these ratings as the basis for determining the overall country development outcome rating in FY14. IEG rated 618 objectives in 41 CASCR and CLR reviews between FY14 and FY16.25 As part of the second analysis, IEG categorized these objectives by the newly defined FY17 themes to analyze their achievement in different CPIA contexts, given that this factor relates to country outcomes.26 Findings show that the overall composition of thematic objectives is similar across lower and higher CPIA levels. The most frequently observed objectives relate to: urban and rural development, public sector management, environment and natural resource management, and human development and gender. However, the share of objectives related to the urban and rural development theme is larger in countries with lower CPIA scores (figure 2.15). Overall, objectives in countries with higher CPIA scores are significantly more likely to be achieved than those in countries with lower CPIA scores. This correlation is generally not significant at the theme level due to smaller numbers. However, the pattern of higher thematic achievement in higher CPIA countries remains, although the achievement gap between higher and lower CPIA countries varies (figure 2.15). The pattern suggests that objectives related to economic policy, finance, and environment and natural resource management were relatively harder to achieve in countries with lower CPIA ratings. Objectives related to public sector management were especially difficult to achieve, regardless of CPIA (figure 2.15). Education-related objectives contributed to lower ratings for the human development and gender theme in the higher CPIA countries. This is a relatively small dataset that warrants additional analysis as more objectives-level data become available. 44 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS Figure 2.15. Thematic Objectives and Their Achievement Ratings, by CPIA a. Objectives related to urban and rural development constitute the largest share of objectives in country programs, especially in countries with lower CPIA scores. Average share of Average share of Theme objectives in lower CPIA objectives in higher countries CPIA countries Urban and rural development 25% 19% Public sector management 16% 18% Environment and natural resource management 16% 16% Human development and gender 16% 17% Private sector development 10% 13% Social development and inclusion 9% 7% Finance 4% 6% Economic policy 2% 6% b. The pattern in achievement of objectives (the percent of objectives in each group that IEG rated as achieved or mostly achieved) differed across themes and across CPIA groups. Share of achieved / Share of achieved / mostly achieved mostly achieved objectives in lower objectives in higher CPIA countries CPIA countries Theme Difference Urban and rural development 51% 56% 5% Public sector management 30% 38% 8% Environment and natural resource management 40% 53% 13% Human development and gender 60% 49% -11% Private sector development 48% 57% 9% Social development and inclusion 55% 59% 4% Finance 47% 82% 35% Economic policy 38% 72% 35% Source: IEG CLR ratings data. Note: Analysis based on categorization of objectives reported in CLRs completed in FY14–16 into eight themes. A few objectives corresponded to more than one theme. IEG categorized the CPIA score as lower or higher based on the median CPIA scores of the end year of the CASCR periods for all countries in the sample. Higher CPIA scores are above the median and lower CPIA scores are equal to or below the median score. Some percentages do not add up to 100 percent because of rounding. CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; CPIA = Country Policy And Institutional Assessment. Lack of realism (or being too ambitious) and deficiencies in results frameworks were two main reasons for lower achievement of country objectives. A review of the 120 objectives rated as not achieved found that delays in the operation supporting the objective or dropping it, operations taking longer than originally anticipated, and 45 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS weaknesses in the implementing agency accounted for 34 percent of the reasons for objectives not being achieved. FY16 country strategy reviews found that many programs were broad and lacked selectivity (too many objectives). About one- quarter were ambitious relative to the government’s capacity, the political environment for undertaking reforms, and agencies’ project implementation capacity. Together, these reflect country teams’ lack of realism about what programs could accomplish during the strategy period, given the operational inputs and support (or lack thereof) for country capacity. Insufficient evidence was cited as a reason for 23 percent of the objectives receiving ratings of not achieved (lack of information or weaknesses in the results framework, such as lack of baselines and targets), which made assessment difficult. Past IEG reviews identified similar issues in several reports (for example, IEG 2012a, 2013b, 2014c, 2015m).27 The results framework and realism of country strategies (based on well-informed risk-taking and understanding of country capacity) are areas in which IEG still sees potential for improvement. Results frameworks and program design are linked, and ambitious objectives need to be commensurately underpinned with operations that can support that ambition. Most results frameworks did not accurately reflect IFC or MIGA’s contributions. Capacity development provided often proved to be inadequate. These shortcomings made it difficult to manage country programs and assess the extent to which the programs met or were meeting World Bank Group country objectives. This, in turn, affected IEG’s country development outcome rating. This situation can continue across two consecutive strategies. Going forward, ex ante assessment of risks using the relatively new Systematic Operational Risk Rating tool, may provide improved information about the realism or relative ambitiousness of country strategies. Another step the World Bank Group could take to strengthen country program management is to use the new CPF process and refocus on outcomes, results chains, and strategy design at the performance and learning review stage. IEG’s review found that the World Bank Group was not taking full advantage of the review stage to reexamine these areas. For example, Colombia’s original strategy proved to be unrealistic and the World Bank Group then curtailed the strategy accordingly during the progress review period; in Bangladesh, a key project was canceled and then the country program was adjusted accordingly. The results frameworks were restructured to reflect these changes. However, most changes involve dropping or adding objectives and indicators without necessarily strengthening the links between the operational program and the outcomes expected. Going forward, the World Bank Group should take full advantage of the performance and learning review and strengthen the results framework as a country program adaptive management tool. It can do this by reexamining the validity of the result chains to 46 CHAPTER 2 RECENT PERFORMANCE OF WORLD BANK GROUP OPERATIONS determine if the ongoing or newly proposed interventions (including IFC and MIGA’s) fully contribute to the planned objective and the country goal. It can also reconsider the indicators and targets to ensure that indicators capture the expected outcomes and are achievable within the country strategy’s remaining time frame. The forthcoming IEG evaluation of SCD and CPF will analyze these issues in more depth. 1 Data are based on 89 percent of ICR reviews completed for projects closed in FY15. See figure C.2 in appendix C. Data includes ICRR ratings for all fiscal years and project performance assessment report ratings through FY14. 2IEG chose the cutoff of $800 million because these projects represented 20 percent of the total volume (net commitment) among projects closed during the six-year period (FY10–15) that had IEG outcome ratings available as of August 2, 2016. 3Although outcome ratings for some Regions were higher (or lower) in FY13–15 than in FY10–12, none of the increases or decreases were found to be statistically significant. As noted earlier, the text in the main report mentions increases, decreases or other differences only if they were found to be statistically significant. 4Although M&E quality ratings for some Regions were higher (or lower) in FY13–15 than in FY10–12, not all the increases or decreases were found to be statistically significant. The two where an increase was found are marked by a gray box around the name of the Region. As noted earlier, the text in the main report mentions increases, decreases or other differences only if they were found to be statistically significant. 5RAP 2015 noted the following factors related to relatively high ratings in Social Protection and Labor: (1) many Social Protection and Labor projects are strongly evidence based and have relatively high ratings for quality at entry; (2) relatively high ratings for quality of supervision; (3) evaluation is often built into project design; and (4) relatively frequent use of impact evaluations. RAP 2015 also noted that the Social Protection and Labor Global Practice had effectively implemented recommendations from IEG’s evaluation of social safety nets (IEG 2011b). 6Findings in previous IEG work are consistent with the study on factors affecting World Bank project outcomes (Denizer, Kaufmann, and Kraay 2011). The authors also found country capacity, project size, supervision cost, and problem project flag are correlated with project outcomes. 7This analysis is based on a review of 199 ICR reviews for projects completed in FY15 for which ratings were available as of September 7, 2016 (41 projects had ratings of moderately unsatisfactory or below on outcome). 8The three adjectives (significant, major, and severe) are important because they are part of the IEG and Operations Policy and Country Services harmonized guidelines for determining an outcome rating. Part of the definition of each rating corresponds to the presence or degree of shortcomings, as follows: 47 Moderately unsatisfactory: “There were significant shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance” Unsatisfactory: “There were major shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance” Highly unsatisfactory: “There were severe shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance.” 9IEG piloted an addition to the ICR review process in FY16 to include systematic documentation of gender dimensions in individual projects. As part of the pilot, ICR reviewers examined whether gender is a relevant aspect of the project and whether the ICR reports sex-disaggregated or female- or male-specific indicators. The pilot extended the analysis conducted for the 2015 Results and Performance of the World Bank Group (RAP) report, and IEG conducted the pilot for 156 projects closed in FY15 with completed ICRRs. 10The analysis performed for RAP 2015, which IEG based on a review of project appraisal documents instead of ICRs and used a different review instrument, examined (among other elements) whether at least one of the actions or components planned as part of a project integrated gender. The analysis found that 43 percent of the projects reviewed integrated gender into one or more actions or components in the project. 11Financial sector refers to all projects evaluated by the Financial Institutions Group and fund projects from the Venture Investing department with a focus in financial institutions. All other projects are classified under the real sector. 12In FY17, IFC started piloting ex-ante assessments of downstream effects in power and extractives sectors. 13 For FY15, Advisory Services evaluation completion rate was 97 percent. 14Indicator for Advisory Services work quality/design quality is not on three-year rolling average since it was introduced after January 1, 2015. 15Out of 71 projects in the FY10–15 period, 7 projects were rated as No Opinion Possible (NOP). The success % including NOP uses 71 as the N whereas the success % excluding NOP uses 64 as the N for calculations. Out of 59 projects in the FY09-14 period, three projects were rated as No Opinion Possible. The calculation for success % including NOP uses 59 as the N whereas the success % excluding NOP uses 56 as the N. Out of 34 projects in the FY13-15 period, seven projects were rated as No Opinion Possible. The success % including NOP (FY13-15) uses 34 as the N whereas the success % excluding NOP uses 27 as the N for calculations. Out of 29 projects in the FY12-14 period, three projects were rated as No Opinion Possible. The success % including NOP uses 29 as the N whereas the success % excluding NOP uses 26 as the N. In the regional performance analysis, South Asia region had no evaluated projects in the review period. For Latin America and Caribbean region, N=1 for calculating both success % excluding NOP and success % including NOP. For Middle East and North Africa region N=2 for calculating both success % excluding NOP and success % including NOP. For Sub-Saharan Africa region, N=12 for calculating success % including NOP and N=11 for calculating success % excluding NOP. For Europe and Central Asia, N=15 for calculating success % including NOP and N=10 for calculating 48 success % excluding NOP. For East Asia Pacific region, N=4 for calculating success % including NOP and N=3 for calculating success % excluding NOP. 16 NOPs by evaluation year and cancellation status Status at the time of Status at the time Evaluation Project evaluation program of evaluation final year number identification issuance 2014 1 Cancelled Cancelled 2014 2 Cancelled Cancelled 2014 3 Cancelled Cancelled 2015 4 Active Active 2015 5 Active Active 2015 6 Cancelled Cancelled 2015 7 Active Cancelled Project IDs not provided due to confidentiality. See the Country Partnership Framework Collection within the World Bank’s Open 17 Knowledge Repository, at https://openknowledge.worldbank.org/handle/10986/23100. For this analysis, the following countries were categorized as IDA-IBRD blend during 18 FY13-16: Bolivia, Cabo Verde, Republic of Congo, Moldova, Nigeria, Pakistan, Papua New Guinea, Sri Lanka, Timor-Leste, and Uzbekistan. 19For example, across the World Bank Group, 53 percent of 19 country programs were rated moderately satisfactory or better in FY13, improving to 77 percent of 22 programs in FY14 and to 85 percent of 13 programs in FY15, but dropping to 55 percent of 20 programs in FY16. 20The World Bank Group and IEG used a six-point ratings scale (highly satisfactory, satisfactory, moderately satisfactory, unsatisfactory, and highly unsatisfactory) for World Bank Group performance until 2013, when a four-point scale was implemented (superior, good, fair, and poor). For simplicity, “good” and “superior” ratings are considered equivalent to “moderately satisfactory or better” (highly satisfactory, satisfactory, and moderately satisfactory) for this report. 21Further analysis of factors of achievement of CAS objectives and country development outcome ratings is presented in paragraphs 2.41 through 2.49. 22All Country Assistance Strategy Completion Reports (CASCRs) and completion and learning reviews (CLRs) began rating World Bank Group performance in FY12. Although some rated performance before FY12, the numbers are limited and are not comparable. 49 23Other country-level factors include annual GDP growth rate, GDP per capita, Human Development Index score, and the Worldwide Governance Indicators on government effectiveness and regulatory quality. 24Government capacity affects country development outcomes directly and also indirectly through project outcomes. The impact of country capacity on project outcomes is widely recognized, as seen in two studies (Denizer, Kaufmann, and Kraay 2011; Moll, Geli, and Saavedra 2015). 25FY14 was a transition year in which eight of the 20 CASCR reviews rated objectives. Haiti (FY16) had separate assessments for the CAS and interim strategy note and counts as two country programs, bringing the total number of CASCRs and CLRs reviews to 41. 26 The World Bank Group adjusted its themes in July 2016, and IEG used the new themes for this analysis. The objectives reflect the themes to which they are mapped. For example, “increase access to sanitation services” (Uzbekistan) is included under Urban and Rural Development; “improve the financial stability of the health sector” (Montenegro) is under Human Development and Gender, and “increased access to electricity” (multiple countries) is under Environment and Natural Resource Management (which includes Energy). 27IEG (2014c) observed a decline in country development outcomes during FY06–13 and attributed it to portfolio deterioration, deficient results frameworks, and the global food, fuel, and financial crisis and the international recession afterward. 50 3. Management Action Record The purpose of IEG evaluations is to help the World Bank Group achieve development results by supporting evidence-based adaptive management and learning. To this end, IEG and management of the World Bank Group institutions established the Management Action Record (MAR) to track the implementation of management actions in response to IEG recommendations. The MAR also fulfills accountability by informing the Board’s Committee on Development Effectiveness and the public about progress on management actions. In addition, the MAR provides an opportunity to reflect how the World Bank Group has learned from evaluations and incorporated this learning in its work. This chapter describes the annual MAR follow-up process and its implementation in FY16. Tracking the Implementation of IEG Recommendations IEG has tracked implementation of its recommendations since the late 1990s for the World Bank and since early 2000s for IFC and MIGA. Since 2013, the management of the World Bank Group institutions has provided detailed action plans with baselines, targets, and indicators in response to IEG recommendations, against which both IEG and management assess implementation progress. A pilot is also under way to assess progress against the objectives of IEG’s recommendations rather than specific action plans and actions. The MAR tracks only those recommendations that are agreed or partially agreed upon by management of the World Bank Group institutions. For recommendations that are accepted, management creates an action plan for implementation. The implementation for each agreed recommendation is independently assessed by management and IEG for four years, after which the recommendation is “retired” from the MAR review process. The analysis in this chapter is based on ratings and reviews of FY16 implementation progress, previous RAPs, and focus groups conducted to understand the impact from the four evaluations retiring from the MAR update cycle. Status of Bank Group Implementation of IEG Recommendations In 2016, IEG tracked actions for 150 recommendations across all three Bank Group institutions drawn from the 26 evaluations produced between FY12 and FY15.1 51 CHAPTER 3 MANAGEMENT ACTION RECORD Consistent with previous years, management self-rating of implementation progress is higher than that of IEG. The disconnect in ratings is usually larger in the first two years of implementation as management assesses progress at the early implementation stage more positively than IEG does (figure 3.1). IEG often points to the need for more work to have been completed or greater results to have been achieved. This trend leads management to retire recommendations from the implementation cycle earlier than IEG does and often before the end of the fourth year, which explains the differences in the recommendation totals shown in the figure. Figure 3.1. Management Self-Ratings of Satisfactory Implementation Were Consistently Higher Than IEG Ratings between FY12 and FY16 92% 100% 81% Mgmt 61% 70% 72% IEG 56% 32% Mgmt N=147 N=108 N=50 N=13 IEG N=150 N=118 N=71 N=18 1st Year 2nd Year 3rd Year 4th Year Source: IEG’s Management Action Record database. For evaluations in the fourth year of review, IEG rates implementation progress as substantial or higher for 72 percent of recommendations. The ratings are slightly higher for recommendations reviewed both by IEG and management, where IEG’s rating is at 77 percent substantial or higher. This is lower than IEG’s ratings of 81 percent for 2015, 83 percent for 2014, and 90 percent for 2013. Figure 3.2. The Most Common IEG M4R-Related Recommendation Addressed Monitoring and Evaluation Monitoring and evaluation 28 Increasing internal capacity 21 Adopting new strategy/direction 18 Increasing client capacity 11 0 5 10 15 20 25 30 Percentage Source: IEG’s Management Action Record database. 52 CHAPTER 3 MANAGEMENT ACTION RECORD Multiple evaluations make similar recommendations, which suggests common challenges and opportunities for the World Bank Group. About a third of the IEG recommendations active in this year’s update focus on capacity building, slightly less than a third on M&E, and approximately one-fifth on the World Bank Group’s strategic directions and approaches (figure 3.2). Recommendations related to capacity building focused more on building internal Bank Group capacity (21 percent) than on building client capacity (about 11 percent). These recommendations generally highlighted the need to enhance staff skills, ensure adequate internal knowledge transfer, and better understand country context and demands. Recommendations focused on M&E show the largest disconnect in the implementation ratings between IEG and management in year four (figure 3.3). This is a recurring trend in MAR updates and previous RAPs. Figure 3.3 Recommendations on M&E Had the Largest Gap between IEG and Management in Year Four Source: Data from IEG’s Management Action Record database. In the past few years, recommendations related to M&E evolved to focus more on learning from results and on client-related issues such as country ownership, capacity, and context. For instance, one recommendation in World Bank Group Support to Electricity Access proposed the following (IEG 2015q): “Across country clients, promote uniformity and comparability in indicators, and help improve country capacity for designing, implementing, and utilizing the M&E frameworks.” Another example is from IEG’s evaluation Knowledge Based Country Program: “The Bank should ensure and monitor high-quality results frameworks that respond to client concerns, and link more tightly knowledge services tasks with country partnership strategies milestones and outcome indicators.” (IEG 2016f, xxxii) Further analysis shows that IEG’s M&E-related recommendations also shifted to suggesting more complex approaches. An example is the IEG recommendations 53 CHAPTER 3 MANAGEMENT ACTION RECORD from the Biennial Report on Operations Evaluation: Assessing the Monitoring and Evaluation Systems of IFC and MIGA (IEG 2008, 2013a). In 2008, IEG recommended that IFC “consider developing more robust and consistent metrics for monitoring and assessing its additionality and development impact at the sector and country levels as IFC transitions to a more programmatic approach to interventions” (IEG 2008, xiii). In 2013, it recommended “self-evaluation system to ensure projects are sufficiently mature to assess more meaningfully their development results. In doing so, IFC might either consider conducting self-evaluation two to three years post completion, possibly on a sample of projects as done for XPSRs, or launching a post completion system based on clear selection criteria for projects to be included” (IEG 2008, p. xl). In addition, IEG recommended to “streamline the project-evaluation approach and process to align more closely with MIGA’s business model and conditions on data gathering.” In addition, among 150 recommendations reviewed in this year’s cycle, management only partially agreed with about 13 percent. For such recommendations, and in some cases even for recommendations that were fully agreed on, the management action plan did not fully encompass the spirit of IEG’s recommendation and thus not all actions taken were in line with IEG’s original intent. Over one-third (36 percent) of the partially agreed recommendations focus on internal capacity building and M&E. Although it is a small number, analysis of management responses found that the partially agreed recommendations most often cited feasibility of implementation and the World Bank’s internal capacity, including time and budget, as main constraints for full implementation. For instance, in response to IEG’s recommendation in Knowledge-Based Country Programs seeking better staff incentives and a balanced financial and budget framework between lending and knowledge activities, management highlighted its commitment to achieve excellence in knowledge services, but could not commit to specific budget allocation principles, at the time when the entire budget process is being reviewed to align it to the new World Bank Group strategy under development. Another example is the recommendation made in Poverty Focus of Country Programs: Lessons from World Bank Experience, which called for attention at project inception to evaluability. Management agreed with the thrust of the recommendations to strengthen project M&E frameworks, but found the specific remedies suggested hard or impractical to translate into monitorable actions given project complexities. IEG Recommendations on M4R Of the 277 recommendations adopted since 2008, including 150 tracked this year, 34 percent had an M4R dimension that related to either capacity development or 54 CHAPTER 3 MANAGEMENT ACTION RECORD M&E.2 The most common management actions in response to M4R recommendations were improving country statistical capacity, introducing core sector indicators, strengthening Bank Group M&E, and incorporating evidence- based knowledge work into project design. Of the retired and active fourth year recommendations adopted since 2008 that have one or more M4R dimensions, IEG rated 68 percent substantial, high, or completed. This is lower than the 77 percent implementation rate for all recommendations after four years. A large proportion (62 percent) of M4R recommendations focused on M&E, but as mentioned above, management and IEG most often disagree regarding the implementation success of these recommendations. An example of such disagreement is implementation ratings of recommendations from IEG’s review of impact evaluations, which were rated substantial and above by management 100 percent of the time over the past four years, while IEG rated progress substantial and above only 60–75 percent of the time in the first three years of implementation. However, neither IEG nor management rated them complete by the fourth year (box 3.1). Box 3.1. Using Impact Evaluations for Better Development Results: The Road Ahead Impact evaluations are now widely used to better understand the results of development interventions and establish counterfactuals. Implementation of the recommendations from the evaluation on the Relevance and Effectiveness of World Bank Group Impact Evaluations acknowledges achievements by the World Bank Group in impact evaluation, including the critical role of development impact evaluation (DIME) in coordinating work across World Bank Group units and the work of hubs, such as i2i, SIEF, and the Africa Gender Innovation Lab, to integrate impact evaluation into project designs and reviews. Nonetheless, the systematic use of impact evaluations for project design and policy advice remains to be improved. Much of the funding for impact evaluation still comes through external funds that are not consolidated. Most project-supported impact evaluations depend on the resourcefulness of project leads to find external funds and no actions have been taken to streamline impact evaluation budgets within the general budget of the World Bank Group. This funding approach, and lack of coordination among impact evaluation hubs, leads to a less than strategic approach to the identification of key learning gaps where such evaluations could be strategically deployed. Source: MAR database and focus group discussions with impact evaluation experts across Bank Group institutions and relevant IEG managers and task team leaders. Influence of IEG Recommendations Evidence is a key input for adaptive management and learning, and IEG plays a role in the World Bank Group’s M4R by providing knowledge based on evaluations. To 55 CHAPTER 3 MANAGEMENT ACTION RECORD better understand how IEG’s recommendations may have influenced the World Bank Group’s operational work and learning in the past four years, IEG conducted eight focus groups targeting IEG and management representatives working on the studies retiring from the MAR update cycle in 2016.3 The focus group discussions revealed that IEG recommendations are most influential when they are first discussed with Bank Group counterparts and the Board of Directors. Participants acknowledged that IEG studies helped catalyze discussions on specific issues and provided space for building knowledge on a given topic. Usually in response to IEG evaluations, management committed more budget and time to learn and gather evidence on a theme. For instance, IFC representatives acknowledged that IEG’s evaluation Assessing IFC’s Poverty Focus and Results (IEG 2011a) led to a much- needed review of the literature on poverty issues and how IFC should incorporate poverty in its strategy and operational discussions. Focus group participants also mentioned that these changes were also due to the new requirements to understand the twin goals and how they would translate into IFC’s operational work. Using the MAR for Adaptive Management and Learning Since the MAR update process focuses strictly on the achievements related to targets and indicators outlined in management action plans, some actions may not be relevant or necessary several years after implementation starts. Therefore, in FY16 IEG and its Bank Group counterparts agreed to implement several pilots to make the MAR process itself more adaptable to help achieve the results intended by the recommendations rather than the implementation of specific actions. One of the pilot initiatives requires IEG and management to treat action plans for Report on Self-Evaluation Systems (ROSES) and Learning and Results in World Bank Operations: Toward a New Learning Strategy (Phase II) in an adaptive manner to allow modifications to implementation throughout four years to account for changing development needs and the World Bank Group context. These pilots aim to shift the focus of implementation—and of the update process—to the achievement of the outcomes intended by the recommendation, rather than focusing on the achievement of targets set up by the action plan several years earlier. Progress on this initiative will be analyzed and reported to the Board by the end of FY17. Conclusion IEG’s progress rating for FY16, although lower than in previous years, acknowledges the steps taken by the World Bank Group toward achieving 56 CHAPTER 3 MANAGEMENT ACTION RECORD development goals. More work needs to be done to address IEG recommendations focused on M4R, including M&E, particularly as the World Bank Group aims to better integrate results data in its decision-making processes 1Implementation of each recommendation is tracked and recorded over four years, with Management of the World Bank Group and IEG making their own assessments using a five- point scale of ratings: negligible, moderate, substantial, high, and complete. Ratings for FY12 recommendations are available for four years, while those of the FY15 recommendations are only available for one year, accounting for the difference in number of ratings for each year in figure 3.1. 2Analysis focused on 277 recommendations from 44 evaluations developed since 2008 to determine whether they incorporated an M4R dimension, including: (i) strengthening monitoring and evaluation (M&E) in countries; (ii) improving M&E within the World Bank Group; and (iii) using evidence for decision making. Among the recommendations, 34 percent included at least one M4R dimension. Improving M&E within the World Bank Group accounted for just under two-thirds (62 percent) of the M4R dimensions while country M&E and evidence-based decision making accounted for 18 percent and 19 percent respectively. Some recommendations were the same for World Bank, IFC, and MIGA, thus the total number of unique recommendations with an M4R aspect was 81 (29 percent). 3The studies retiring in the 2016 MAR update are the following: World Bank Country-Level Engagement on Governance and Anticorruption; World Bank Progress in Harmonization and Alignment in Low-Income Countries; The World Bank Group Impact Evaluations: Relevance and Effectiveness; and Assessing IFC's Poverty Focus and Results. The focus groups were conducted for three of the four studies because the relevant parties were not available for the discussions on Progress in Harmonization and Alignment in Low-Income Countries. 57 Appendix A. IEG Evaluations and Learning Products FY15–17 and Findings Related to M4R FY17 Evaluations Evaluation or learning product purpose Findings or lessons related to M4R 1 Program-for-Results: An Early-Stage Assess the overall experience with the The results frameworks are often reasonably coherent, but Assessment of the Process and design, preparation, and early objectives that lead to clear results at the outcome level are Effects of a New Lending Instrument implementation of Program-for-Results rare. PAD objectives and supporting discussions could go (PforR) operations. beyond the program period itself to consider longer-term objectives at the outcome level and sustainability of program improvements. 2 Industry Competitiveness and Jobs: Assess the implication of the World Bank Employment is a central aspect of the productivity and An Evaluation of World Bank Group Group’s industry-specific support on job competitiveness agenda. Yet, only a small proportion of the Industry-Specific Support to Promote quantity and quality. World Bank Group portfolio specifically refers to jobs in Industry Competitiveness and Its objectives, interventions, or indicators, and even less so it Implications for Jobs measures implications of productivity on jobs. The evaluation illustrates the conceptual and practical challenges in measuring the net impact of interventions on jobs. Task team leaders may have found it challenging to identify jobs objectives given the quantitative and qualitative attributes and both conceptual and measurement challenges related to jobs effects of sectoral competitiveness interventions. This is an important agenda that requires progress on issues ranging from research to results framework, to strengthen the employment focus of industry competitiveness work. 3 The World Bank Group’s Support to Review the World Bank Group’s contribution Most output was of good quality and some was adopted. Capital Market Development to the preparation and modification of a Outcomes are harder to assess, and allowances must be made sound legal and regulatory framework for for long lags in final results in the legal and regulatory area. securities. Data issues affect the capital markets program, and databases Review the World Bank Group’s support to do not capture local currency bond market development. long-term finance of infrastructure projects, environment finance, and real sectors through capital markets instruments. 59 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15–16 AND FINDINGS RELATED TO M4R 4 World Bank Group Engagement in Review issues in the Operational Framework Data availability is a severe constraint in small states. Small States: The Cases of the OECS, for Engaging with Small States. Emphasis on data collection and knowledge work in small PICs, Cabo Verde, Djibouti, Mauritius, states is key. Data is a priority because it is vital to and the Seychelles monitoring and measuring progress. 5 World Bank Group Engagement in Assess the drivers of success and failure in Weak monitoring and evaluation (M&E) frameworks Situations of Fragility, Conflict, and fragility, conflict, and violence (FCV) under undermined the assessment of results in many of the cases Violence: An Independent Evaluation the World Bank Group’s control, and how observed and diminished the projects’ learning potential, have they affected the results and even when they were designed as pilots. Project design and performance of the World Bank Group’s implementation generally reflected FCV factors, but results assistance to FCV. rarely captured them. Staff incentives and performance measurement systems linked to project performance and volume targets do not align with increasing IFC engagement in FCV. Similarly, results measurement frameworks may not be fully adapted to FCV contexts. 6 Behind the Mirror: A Report on the Assess the adequacy of the World Bank Several findings show that the M&E systems and incentives are Self-Evaluation Systems of the World Group’s self-evaluation systems to: not conducive to the intended uses of the self-evaluation Bank Group Inform decision making as it relates to system as outlined in the review questions. operational performance management. Furthermore, staff believes that the World Bank Group does not Verify achievement of results and promote contribute sufficiently to building clients’ M&E capacity. accountability for results. Support learning from experience. FY17 Learning products Evaluation or learning product purpose Findings related to M4R 7 Learning product (Category 1): A survey of practices and results observed in PBG teams could find it useful to work with clients on Findings from Evaluations of Policy- the implementation of policy-based decomposing and benchmarking the interest rates of PGE- Based Guarantees guarantees (PBGs). supported debt instruments to better assess the extent of improvements in credit terms. World Bank self-evaluation of PBGs through implementation completion and results reports can be an appropriate platform for such analysis. 8 Learning product: Financial Viability of Review the effectiveness of World Bank Of the full cohort of 49 development policy operations (DPOs), the Electricity Sector in Developing interventions during FY00–15 in supporting 35 projects had evaluable final outcome data on financial Countries: Recent Trends and client countries for improving the financial indicators for financial performance improvements. In IEG’s Effectiveness of World Bank performance and long-term viability of their framework, financial viability is one of the main intermediate Interventions electricity sectors. sector outcomes affecting access. 60 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15-16 AND FINDINGS RELATED TO M4R Some DPOs have weak results frameworks, lacking clear statements of objectives and outcomes and offering low-impact prior actions. This makes it difficult to monitor and measure their development impact. This shortcoming is present in some newly approved DPOs that follow the modified Operations Policy and Country Services template. 9 Learning Product -Learning in IFC Review the LessonFinder database. The LessonFinder database is an important tool for operational teams to leverage for learning from experiences, projects, and programs. 10 Learning product: Learning from IDA Synthesis of findings from recent IEG Data scarcity limits the ability to assess the impact of World Experience: Lessons from IEG evaluations and analysis focusing on the Bank Group support to youth employment. Few countries Evaluations special themes under the 16th assessed the impact of food crisis on the poor, and most Replenishment of IDA (IDA16) and IDA17 to projects did not specify actual and expected program coverage support the IDA18 replenishment to assess the project’s likely contribution to the population most discussions. needing assistance due to scarce data in low-income countries. Weak country institutions and inadequate data are constraints for World Bank support for social safety nets in IDA countries. Climate change and inclusive growth themes are now followed up with measurable indicators. 11 Learning product: Lessons from A review of the World Bank Group’s Monitoring and evaluation systems for most environmental Environmental Policy Lending experience with development policy financing DPOs have been weak. For the 31 operations with IEG M&E (DPF) in the environment sector to provide quality ratings, 21 had ratings of negligible or modest, only 10 lessons and inform stakeholders on how to had a rating of substantial, and none had a rating of high. design and implement this instrument. Use of monitoring and evaluation is rare; systems are used largely for reporting and for tracking progress on triggers for future operations in a programmatic series instead of for assessing the need for course correction. Monitoring and evaluation systems are not being established to assess the long-term impacts of World Bank engagement. Even in aggregate, World Bank documents provide little evidence on which to assess overall improvement in environment-related physical and human outcomes. 61 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15–16 AND FINDINGS RELATED TO M4R The World Bank could work to find opportunities for joint assessments with the country of long-term, policy-related performance, but for a meaningful assessment, this would require long-term data collection. 12 Learning product: The Role of Political A review of the World Bank’s experience in The relative lack of reform-specific political economy analysis in Economy Analysis in Development the use of political economy analysis in DPOs the context of variation in the quality of reform actions may be a Policy Operations and contribute to the World Bank’s ongoing symptom of a weakness in the results orientation of DPOs in reflections on the design and use of DPOs specific policy and institutional reforms. Stronger integration of DPFs with systematic country diagnostics and country partnership frameworks for identifying most critical reforms and monitoring results achievement can be a way to strengthen the instrument further. FY16 Evaluations Evaluation or learning product purpose Findings related to M4R 13 Results and Performance of the World Assess the extent to which M&E systems RAP 2015 discussed the inadequacy of the indicators used in Bank Group 2015: An Independent measure and report on gender results in country strategies and projects to capture gender results. Evaluation (RAP) operations and country strategies 14 Financial Inclusion: A Foothold on the Evaluate the effectiveness of the World Bank Of the reviewed projects, 45 percent tracked process and Ladder toward Prosperity? An IEG Group in creating better functioning markets outputs, but only 2 percent reported on beneficiary effects, such Evaluation of World Bank Group that provide improved access to and quality as welfare improvements or income increases. Support for Financial Inclusion for of financial services to the poor and Low-Income Households and microenterprises on a sustainable basis Microenterprises 15 World Bank Group Engagement in Assess how the World Bank Group Although diversification was the overarching strategy objective Resource-Rich Developing Countries: addresses economic diversification and non- in these countries, there were no mechanisms in place to The Cases of the Plurinational State of extractive growth, including promoting monitor this objective. Bolivia, Kazakhstan, Mongolia, and growth, job creation, and private sector entry Zambia. Clustered Country Program into non-extractive sectors Evaluation Synthesis Report 16 World Bank Group Support to Assess the extent to which the World Bank M&E weakness is more marked in low- and medium-access Electricity Access, FY2000–FY2014: Group has been effective in the past and, countries, largely due to lack of indicators, weak baseline data, An Independent Evaluation, going forward, how well it is equipped to put and inadequate monitoring capacity. The shortcomings are Volume II: Together for Energy: How its country clients on track to achieve highest regarding the tracking of economic and welfare Partnership Programs Support Energy universal access to electricity that is outcomes, including gender considerations, but the World Bank Access 62 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15-16 AND FINDINGS RELATED TO M4R adequate, affordable, and of the required is more aware of this matter and recently made improvements quality and reliability in M&E frameworks in this regard. FY16 Learning products Evaluation or learning product purpose Findings related to M4R 17 Learning product: World Bank Group Generate knowledge and provide lessons IEG could not measure effectiveness on affordable housing Support for Housing Finance learned from the World Bank Group’s support interventions in several World Bank Group projects because to housing finance they lacked adequate targeting, baseline indicators, and data on actual achievements. 18 Learning product: Lessons from Land Synthesis of cross-cutting findings and Project design must include the social aspects of land Administration Projects: A Review of lessons from IEG of World Bank–supported administration projects and the needs of the vulnerable. Social Project Performance Assessments projects dealing with land administration impacts need monitoring, but projects should be flexible to issues and the factors that contributed to adapt in response to implementation experience. Projects need broader development outcomes disaggregated data to assess impact on different population segments. In many cases, the link between improving land administration and other expected outcomes could not be assessed because of monitoring issues. Some projects did not attempt to measure them and made assertions based on assumptions. In other cases, attribution challenges complicated the evidence. 19 Learning product: Supporting Review IEG evidence to understand the Projects that claim to be pilots rarely include mid-term testing or Transformational Change for Poverty mechanisms and conditions for enumerate objective criteria that will trigger scaling up (or they Reduction and Shared Prosperity: transformational engagements and the do not have a scaling-up plan within the life of the project), and Lessons from World Bank Group implications for the World Bank Group if it they often do not collect baseline data. Pilot projects need Experience seeks to rely on such engagements for its rigorous evaluations to inform future projects, but this rarely goals happens. 20 Learning product: IFC’s Additionality RESTRICTED DISTRIBUTION 21 Learning product: Managing Review the World Bank’s actions, policies, The World Bank’s monitoring documents seldom contain Environmental and Social Risks in procedures, and guidance, and the information on actual E&S effects or the efficacy of mitigation Development Policy Financing management of potentially adverse measures, except for positive E&S outcomes that were part of environmental and social (E&S) effects of the program’s objectives. In particular, implementation DPOs completion reports (ICRs) rarely discuss negative E&S issues for DPFs in the sample IEG reviewed. The task team did not formally document any monitoring of E&S risks that it undertook. Implementation status reports focus on tracking achievement of intended goals instead of assessing unintended side effects. 63 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15–16 AND FINDINGS RELATED TO M4R 22 Learning product: Jobs in IFC’s Restricted Distribution Manufacturing Projects: Lessons from Project Evaluation 23 Learning product: Additional Financing Review how additional financing was used Of 32 projects sampled, 27 had a modest rating for their M&E for Transport and Information and and how it affected project outcomes through frameworks, mainly because the project outcome indicators Communication Technology a subset of the additional financing portfolio could not measure the achievement of project development for which relevant data were readily available. objectives sufficiently or accurately, or the achievement of the indicator targets could not be attributed to the project. This suggested that projects do not fully use the additional financing stage as an opportunity to improve the projects’ results framework. 24 Learning product: The Quality of Help fill gaps in the World Bank’s knowledge Key challenges of poverty assessment measurability include a Results Frameworks in Development about the results orientation of DPOs and the lack of results indicators in M&E frameworks that are linked to Policy Operations implications for achieving institutional change the prior actions, failure to report data on result indicators, and that improves a borrower’s sustained growth dropping results indicators altogether. In some cases, the and poverty reduction outlook. expected results go beyond the DPO’s timeline and, therefore, the ICR does not report them. However, recent changes in M&E frameworks have contributed positively to the overall clarity of results frameworks. These changes include streamlining the M&E framework and better reporting of actual values at completion. The greatest remaining challenge is the use of output-oriented results indicators. FY15 Evaluations Evaluation or learning product purpose Findings related to M4R 25 Evaluation II: Evaluate how the World Bank Group learns in Often, projects do not collect data when the projects actually Learning and Results in World Bank its lending operations. Assess the scope for begin, and the ICRs often misleadingly indicate baselines as Operations: Toward a New Learning improving how it generates, accesses, and zero. Strategy uses learning and knowledge from Eighty-five percent of the projects reviewed present at least operations. some outcome evidence in their ICRs, but two-thirds of these projects had little discussion of the diverse factors that might have affected the outcome. Fifty-eight percent of the projects with some outcome evidence used a pre-post evaluation design for assessing efficacy in ICRs with no control or comparison groups or attempts to provide potential alternative explanations based on a change theory. 64 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15-16 AND FINDINGS RELATED TO M4R 27 Evaluation I: Evaluate how the World Bank Group learns in World Bank staff perceives the lack of institutional incentives as Learning and Results in World Bank its lending operations. Assess the scope for one of the biggest obstacles to learning and knowledge sharing Operations: How the World Bank improving how it generates, accesses, and in the World Bank. Learns uses learning and knowledge in these operations. 28 World Bank Support to Early Examine the World Bank’s design and Within the results frameworks analyzed, the projects typically Childhood Development: An implementation of projects supporting early tracked outputs instead of outcomes. When projects noted Independent Evaluation childhood development interventions to outcomes, they were health measures such as infant, child, or inform future operations. maternal mortality rates, which are not attributed solely to the interventions. 29 The Poverty Focus of Country Assess whether the World Bank had The evaluation identified several challenges in producing high- Programs: Lessons from World Bank appropriate data to understand the nature of quality data for measuring poverty, from design and Experience poverty and for robust analytical work on implementation problems in a single survey to the lack of poverty. survey comparability over time, to problems in constructing panel data. Insufficient capacity and government budget are key obstacles to collecting data, and the report calls for strengthening statistical agencies’ capacity. Poverty-related projects often measure project milestones, but not outcomes. 30 Results and Performance of the World Review the World Bank Group’s experience The World Bank Group strengthened its results orientation, but Bank Group 2014 with the Millennium Development Goals and did not articulate a clear results chain for the Millennium lessons for its engagement with the post- Development Goals. 2015 development agenda. 31 The Big Business of Small Inform the discussion of the extent and World Bank Group projects targeting small and medium Enterprises: Evaluation of the World nature of the World Bank Group’s future enterprises should define and justify the beneficiary group Bank Group Experience with Targeted engagement in targeted small and medium specification, provide specific targeting mechanisms, and Support to Small and Medium-Size enterprise support activities. include impact indicators in its results and M&E frameworks. Enterprises, 2006–12 65 APPENDIX A ADDITIONAL IEG EVALUATIONS AND LEARNING PRODUCTS FY15–16 AND FINDINGS RELATED TO M4R FY15 Learning products Evaluation or learning product purpose Findings related to M4R 32 Learning product: How Does Assess the extent of integration of knowledge None noted. Knowledge on Public Expenditures on public expenditures with the design of Integrate with the Design of DPOs. Development Policy Operations? 33 Learning product: Africa Microfinance Restricted Distribution Program Review 34 Learning product (Category 1): Restricted Distribution Investments in Renewable Energy Generation 35 Opportunities and Challenges from Summarize IEG’s findings and Global and regional engagements are not tracked in any Working in Partnership: Findings from recommendations from its Global Partnership portfolio database and are not expected to report on results, IEG’s Work on Partnership Programs Program reviews. thus making it possible for major risks—and major and Trust Funds opportunities—in the partnership portfolio to go unrecognized. In a sample of 17 independent evaluations of partnership programs, weak M&E frameworks compromised all of the programs: program objectives and strategies were ill defined, and programs’ M&E systems failed to pay attention to outcomes or did not collect data on them. 36 Learning product: Assess the quality of the macro-fiscal None noted. Quality of Macro-Fiscal Frameworks in frameworks in DPOs. Development Policy Operations 37 Results Frameworks in Country Review IEG evidence regarding the major IEG’s country program evaluations and reviews of Country Strategies: Lessons from Evaluations stages of developing a results framework at Assistance Strategy Completion Reports (CASCRs) show (Disclosed in FY14.) the country program level. some common deficiencies, including a focus on outputs instead of outcomes, a weak link between designed interventions and outcomes, and not monitoring indicators to track outcomes. Several recent IEG evaluations and reports, including the 2012 The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness and RAP 2013, showed that a weak country assistance strategy results framework is a key determinant of unsatisfactory outcome performance at the country program level. 66 Appendix B. Methodology for Client Support Portfolio Review 1. As part of the 2016 Results and Performance of the World Bank Group (RAP) report, the Independent Evaluation Group (IEG) conducted an exploratory study to assess the degree to which the World Bank Group supported M4R through its projects. The analysis recognizes that World Bank operations are demand-based, and demand for M4R may depend largely on countries’ incentives and on institutional arrangements that might support or inhibit M4R. Selection of ICRRs for Client M4R Review for World Bank Projects 2. IEG selected the ICRRs of 347 projects for review, including:  215 projects closed in FY14 and posted an ICRR as of July 20, 2016, and 50 closed in FY15 and posted an ICRR as of May 15, 2016 for a total of 265 projects closed in FY14 and FY15.  82 recipient-executed trust funds (RETFs) operations that closed during FY13–15 and posted an ICRR as of July 20, 2016.1 A longer timeframe was selected for RETFs due to their smaller numbers in the portfolio. Table B.1. World Bank Projects Selected for Review, by Fiscal Year and Product Line Product line Closing FY IBRD and IDA RETF Total 2013 0 46 46 2014 215 15 230 2015 50 21 71 Total 265 82 347 Sources: World Bank Business Intelligence and IEG World Bank project ratings data. Note: RETF = recipient-executed trust fund. 3. Overall, the selected projects were distributed across World Bank regions and global practices similarly to the larger IEG ICRR 2013-2015 portfolio (tables B.2.a and B.2.b). 1 IEG validates ICRs for recipient-executed trust funds of $5 million and above. 67 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW Table B.2.a. ICRR Portfolio Composition by Region IEG projects rated 2013–2015 M4R portfolio selected Region Number Percent Number Percent Total 771 100 347 100 Sub-Saharan Africa 239 31 100 29 Latin America and the Caribbean 153 20 68 20 East Asia and Pacific 136 18 70 20 Europe and Central Asia 112 15 40 12 South Asia 77 10 39 11 Middle East and North Africa 54 7 30 9 Source: IEG ICRR database. Note: M4R = managing for results. Table B.2.b. ICRR Portfolio Composition by Global Practice M4R portfolio selected IEG projects rated 2013–2015 Global practice Number Percent Number Percent Total 674 100 320 100 Agriculture 66 10 32 10 Education 75 11 53 17 Energy & Extractives 50 7 19 6 Environment and Natural Resources 39 6 7 2 Finance & Markets 28 4 8 3 Governance 49 7 31 10 Health, Nutrition, and Population 59 9 36 11 Macro Economics and Fiscal 47 7 15 5 Management Other 1 0 1 0 Poverty and Equity 6 1 4 1 Social Protection and Labor 36 5 17 5 Social, Urban, Rural and Resilience 86 13 38 12 Global Practice Trade and Competitiveness 16 2 3 1 Transport and ICT 63 9 28 9 Water 53 8 28 9 Source: IEG calculations and Business Intelligence. Note: Some GP codes were missing, hence the difference in Ns between B.2 and B.3. Identifying M4R and Coding for Review 4. The ICRR dataset includes a rich set of information regarding project design, implementation, and results. The review comprised reading through the ICRRs and classifying project objectives, components, and activities into the following key 68 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW elements through which the M4R principles are operationalized in World Bank operations:  Data systems: Key words include data, statistics, statistical capacity, information systems, information and technology systems, poverty data.  Monitoring: Examples of keywords used include monitoring systems, monitoring, performance measurement, performance management, performance information, results management. (The Organisation for Economic Co-operation and Development defines monitoring as “A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.” [OECD-DAC 2002, 28]).  Evaluation: Key words included evaluation, impact evaluation, evidence, improve performance, mid-term review, among others. (The OECD defines evaluation as “The systematic and objective assessment of an ongoing or completed project, program, or policy, its design, implementation, and results.” [OECD-DAC 2002, 21-22]).  Evidence-based program/policy: Examples of key words included use of evidence, use of results, feeding back information into decision-making, policy.  Results- and performance-based budgeting. Examples of keywords used to search for this element include performance-based budgeting, results-based budgeting, performance-based allocation, and results-informed budgeting. (Robinson and Brumby 2005 provides an exposition of these related concepts.)  Results- and performance-based financing. Examples of keywords used to search for this element include strengthening efficiency, institutional performance of the government or sector, results-based financing agreements, results-based grants, among others. Fritsche and others (2014) define performance-based financing as pay-for-performance programs. 5. IEG first reviewed the 347 ICRRs to classify operations based on whether they addressed M4R, using the key words to find the appropriate sections and paragraphs to review. An operation addressed M4R if it met the following two conditions:  Projects included one or more M4R elements as part of their design (data systems, monitoring, evaluation, evidence-based policy or program, results- 69 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW or performance-based budgeting, results- or performance-based financing,) as a project component, subcomponent, or activity.  IEG sought to identify through information in the ICRR that the M4R component, subcomponent, or activity was directed at strengthening the country’s capacity at the sectoral or cross-sectoral levels. Therefore, IEG did not include in the “M4R” category projects that had an M4R component, subcomponent, or activity with the primary purpose of the project itself, including monitoring and evaluating project outputs or outcomes. 6. To reduce errors of omission, the review included projects in the M4R category, if the difference between project M4R (most often its M&E) and its support to strengthening the country’s M4R capacity at the sectoral and cross-sectoral level was not clear from reading the ICRR. 7. Furthermore, projects with the following theme codes were automatically included: managing for development results (90), economic statistics, modeling, and forecasting (22), and poverty strategy, analysis, and monitoring (53). IEG excluded four projects because of possible errors in theme code classifications. 8. Of the 347 operations reviewed, IEG coded 169 operations as having addressed client M4R capacity at the sectoral and cross-sectoral levels (table B.3). Table B.3. Selected Portfolio for further M4R Categorization M4R projects Number Percentage Yes 169 49 No 178 51 Total 347 100 Source: IEG calculations based on data from Business Intelligence. Note: M4R = managing for results. 9. Once IEG classified an operation as an M4R operation, based on reading through the ICRR and identifying M4R elements in the project, a rich set of variables was coded:  the extent of the M4R coverage: entire project (all components, one or some components of the project, or an activity or subactivity within a component);  which M4R element was included in the project (results-based budgeting, results-based financing, data systems, evidence-based program or policy, data systems, monitoring, and evaluation);  what type of capacity the project strengthened; institutional and legal frameworks, organizational and systems development, or human resources development; and 70 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW  whether the risk to development outcome or lessons learned sections of the ICRR mentioned M4R as an issue (problems such as lack of government capacity, no information or data systems in place, among others). Limitations 10. IEG based the review on the ICRRs and did not review other World Bank documents. Therefore, it is possible that some projects that addressed client M4R but did not include the information in the ICRRs were not in the group reviewed. However, as clarified previously, IEG reduced errors of omission by including projects in its sample in which the difference between project M&E and strengthening the country’s M4R capacity at the sectoral and cross-sectoral levels was not clear from reading the ICRR. Analysis of Operations Not Addressing Client M4R 11. From the 178 operations classified as non-M4R operations, IEG conducted an additional analysis of 139 IBRD and IDA operations. The analysis did not include recipient-executed trust funds. IEG dropped 15 of the 139 operations because project documents were not available. Therefore, the final non-M4R analysis included 124 IBRD/IDA operations. 12. The analysis reviewed whether the project appraisal documents (PADs), project papers, and ICRRs mentioned M4R issues, defined as low government capacity in any M4R component, lack of any of the M4R systems, and lack of political support to advance M4R reform, among others. 13. The sections of PADs and project papers reviewed included the country context, sectoral and institutional context, and lessons learned and reflected in the project design. In the ICRRs, IEG reviewed the rationale for risk to development outcome rating; M&E design, implementation, and utilization; and lessons sections. 14. The second part of the analysis included project performance assessment reports (PPARs). IEG reviewed 54 PPARs, 26 of which posted in 2015 and 28 posted in 2016 (as of July 18, 2016). IEG used the same methodology as the one used for classifying projects as “M4R” and categorizing their M4R elements. Selection of World Bank Advisory Services and Analytics 15. IEG considered advisory services and analytics activities addressing client M4R capacity if they contained the following World Bank theme codes: managing for development results (90), economic statistics, modeling and forecasting (22) and poverty strategy, analysis and monitoring (53). Among the 3,640 advisory services 71 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW and analytics activities completed between FY13 and FY15 as identified through a Business Intelligence report as of August 18, 2016, IEG determined that 407 activities addressed M4R (table B.4). Table B.4. World Bank M4R Advisory Services and Analytics Portfolio, by Fiscal Year Closing FY M4R advisory services and analytics FY13 60 FY14 134 FY15 213 Total 407 Source: IEG calculations based on data from Business Intelligence. Note: M4R = managing for results. IFC M4R Analysis IFC M4R Methodology and Results 16. Corporate-level M4R. Managing data systems and related process and tools to enhance results framework and reporting of performance is corporate-level M4R. The following methodology was adopted for the analysis:  Assess the cascading of targets and indicators from corporate to department to project level.  Review the adaptive management for IFC Development Goals and their flow into the Corporate Scorecards.  Identify the data collection methods and use of core sector indicators (degree and intensity). ◦ Observations on degree and intensity ◦ Observations on absolute values, percentages, and relative values ◦ Observations on verifiability, auditability, and timeliness of monitoring results. 17. Client and beneficiary-level M4R. Capacity development through IFC additionality in enhancing systems, tools, and processes for capturing development results is defined as the client and beneficiary-level M4R (table B.5). The methodology adopted is as follows:  Assess ratings on Expanded Project Supervision Reports (XPSRs) and project completion reports with an emphasis on IFC nonfinancial additionality.  Identify seven components that adhered M4R principles. 72 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW Table B.5. Reporting Systems M&E systems Financial reporting systems Corporate governance systems Client survey and data systems E&S performance systems Capacity development Risk management systems Source: IEG. Note: E&S = environmental and social. 18. Identify keywords for rigorous contextual and keyword searches in projects as part of the methodology (table B.6).  Frequency testing on 30 keywords based on the components  Narrow the list depending on false positives and false negatives  Assess M4R’s frequency, co-occurrence, and intensity in the portfolio Table B.6. M4R Keywords M&E ESMS Monitoring Environment system Monitoring system Environmental and social Governance Beneficiary survey Corporate governance Client survey Financial system Risk system Financial reporting Risk measurement Education system Risk monitoring Data Corporate risk Data system Performance measurement MIS Performance management Statistical capacity Evidence Institutional capacity Evaluation system Regulatory capacity Results based Information system Results oriented Note: MIS = management Information System; M&E = monitoring and evaluation; ESMS = environmental and social management system IFC M4R Degree and Intensity 19. Degree refers to the number of evaluated projects that discussed the M4R keywords. Intensity refers to the number of keyword occurrences within a project, region, or industry or business line (tables B.7-B.10). 73 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW Table B.7. IFC Investments by Region Region Degree Percent Sub-Saharan Africa 186 33 Latin America and the Caribbean 117 21 East Asia and Pacific 72 13 Europe and Central Asia 66 11 South Asia 61 12 Middle East and North Africa 40 7 World 16 3 Total 558 100 Source: IFC. Note: Degree refers to the number of evaluated projects that discussed the M4R keywords. Table B.8. IFC Investments by Industry Cluster Data Industry cluster Sum of degree Sum of intensity Technology, Telecom, Media and Venture Investing 57 11 Financial Institutions Group 272 12 Infrastructure and Natural Resources 96 3 Manufacturing, Agribusiness and Services 133 0 Total 521 20 Sources: IFC, IEG. Note: Degree refers to the number of evaluated projects that discussed the M4R keywords, and intensity refers to the number of keyword occurrences within a project, region, or industry or business line. Table B.9. IFC Advisory Services M4R by Region Region Degree Percent Latin America and the Caribbean 130 27 East Asia and Pacific 107 22 South Asia 87 18 Sub-Saharan Africa 69 14 Middle East and North Africa 59 12 Europe and Central Asia 19 4 World 8 2 Total 479 100 Sources: IFC, IEG. Note: Degree refers to the number of evaluated projects that discussed the M4R keywords. 74 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW Table B.10. IFC Advisory Services M4R by Business Line Data Business line Sum of degree Sum of intensity CAS 30 0 CAS-PPP 55 0 ESG 46 2 FAM 54 5 FIG 165 8 INR 52 2 MAS 13 0 TAC 58 3 TTV 6 0 Total 479 20 Note: Degree refers to the number of evaluated projects that discussed the M4R keywords, and intensity refers to the number of keyword occurrences within a project, region, or industry or business line. 20. IEG conducted repeat client M4R analysis (FY13–15) to test the number of repeat clients within a cohort of evaluated projects and where opportunity exists to enhance the M&E frameworks and embed M4R principles (tables B.11 and B.12). Table B.11. Repeat Client and Repeat Sponsor Projects: IFC Investments IEG Reviewed during FY13–15 FY Repeat client projects Repeat sponsor projects All projects 2013 13 27 64 2014 13 36 80 2015 7 38 85 Total 33 101 229 Note: Repeat client refers to project entities that have received support from IFC in more than one project (with a unique project identification number in the IFC management information systems). Repeat Sponsor refers to private entities that have received support from IFC in more than one project and are sponsors of the project entities. This analysis doesn’t differentiate projects within one country or across multiple countries or across business lines. Table B.12. Repeat Client IFC Advisory Projects IEG Evaluated during FY13–15 Share of repeat FY Repeat client projects projects All projects 2013 35 50 70 2014 30 56 54 2015 41 67 55 Total 106 57 179 Note: FY=fiscal year; CY = calendar year. 75 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW IFC Ratings Variance across Three Ratings Systems 21. IEG analyzed the ratings variance across the Development Outcome Tracking System (DOTS), XPSRs, and IEG validations. DOTS ratings are based on the entire portfolio of active clients as opposed to a sample (which is the case for XPSRs); DOTS ratings are client-based, as opposed to project-based in XPSRs; DOTS ratings are updated annually, capturing changes in performance over time while an XPSR takes a snap shot of the project once in its life. IEG validations are undertaken following the XPSR and are conducted once in the life of the project. 22. IEG assessment indicates that the variance between DOTS and IEG ratings, as indicated by the disconnect rate was 19 percent in FY13, 6 percent in FY14, and 14 percent in FY15. The disconnect was contributed by lowered IEG ratings for all subindicators except Economic Sustainability. Though moderate, the review also suggested that DOTS (as an M4R system) was not completely updated for the portfolio to reflect project-level outcomes. Among all the subindicators, the DOTS ratings for project business success showed the largest divergence from IEG ratings. The validation of E&S effects had the least variance across the three systems, showing the highest degree of alignment. The breadth of the gap between XPSR and IEG ratings was consistent during the three years (table B.14). Table B.13. DOTS and IEG Ratings Disconnect in Development Outcome Indicators, FY13–15 (percent) Success rates Development Project business Economic Private sector basis outcome success sustainability E&S effects development 2015 DOTS ratings 67 59 61 77 70 IEG ratings 53 47 50 63 64 Disconnect 14 12 11 7 6 2014 DOTS ratings 64 30 55 68 82 IEG ratings 58 47 56 63 60 Disconnect 6 -17 -1 5 18 2013 DOTS ratings 74 60 62 83 81 IEG ratings 55 47 64 67 66 Disconnect 19 13 -2 16 15 Sources: IFC, IEG. Note: DOTS = Development Outcome Tracking System; E&S = environmental and social. 23. The ratings disconnect between XPSR and IEG ratings was 16 percent in FY13, 10 percent in FY14, and 16 percent in FY15 (table B.17). In the three years, 76 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW XPSR and IEG ratings had a distinctly smaller variance for the E&S effects subindicator than the other subindicators, showing a higher degree of alignment. Table B.14. XPSR and IEG Ratings Disconnect in Development Outcome Indicators, FY13–15 (percent) Success rates Development Project business Economic E&S effects Private sector basis outcome success sustainability development 2015 XPSR ratings 69 57 67 69 77 IEG ratings 53 47 50 63 64 Disconnect 16 10 17 6 13 2014 XPSR ratings 68 58 67 71 71 IEG ratings 58 47 56 63 60 Disconnect 10 11 11 8 11 2013 XPSR ratings 71 59 74 75 83 IEG ratings 55 47 64 67 66 Disconnect 16 12 10 8 17 Note: E&S = environmental and social; XPSR = Expanded Project Supervision Reports. 24. The XPSR overall development outcome ratings had higher success rates in FY14 and FY15 by 4 percent and 2 percent, respectively, but 3 percent lower than DOTS in FY13 (table B.15). The variance band was moderate. The project business success indicator had a large gap of 28 percent between DOTS and XPSR in FY14, again raising the question of whether DOTS was updated. The second biggest gap was in the economic sustainability indicator (12 percent in two years). The E&S effects indicator had the least variance between DOTS and XPSR, from 3 percent to 8 percent. Table B.15. DOTS and XPSR Ratings Disconnect in Development Outcome indicators, FY13–15 (percent) Success rates Development Project business Economic E&S effects Private sector basis outcome success sustainability development 2015 DOTS ratings 67 59 61 71 70 XPSR ratings 69 57 67 69 77 Disconnect -2 -2 -6 2 -7 2014 DOTS ratings 64 30 55 68 82 77 APPENDIX B METHODOLOGY FOR CLIENT SUPPORT PORTFOLIO REVIEW Success rates Development Project business Economic E&S effects Private sector basis outcome success sustainability development XPSR ratings 68 58 67 71 71 Disconnect -4 -28 -12 -3 11 2013 DOTS ratings 74 60 62 83 81 XPSR ratings 71 59 74 75 83 Disconnect 3 1 -12 8 -2 Source: IEG. Note: DOTS = Development Outcome Tracking System; E&S = environmental and social; XPSR = Expanded Project Supervision Reports. DOTS ratings are based on the entire portfolio of active clients as opposed to a sample (which is the case for XPSRs); DOTS ratings are updated annually, capturing changes in performance over time while an XPSR takes a snap shot of the project once in its life. 78 Appendix C. Additional Information on World Bank Portfolio Figure C.1. Additional Information on World Bank Lending Commitments a. Net commitments for DPL and P4R increased, though their numbers decreased. Net commitments in FY16 for the relatively new P4R totaled $5 billion, or 12 percent of volume P4R IPF DPL N DPL N IPF N P4R 60 300 50 250 Number of projects 40 200 $, billions 30 150 20 100 10 50 0 0 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY of project approval b. Average sizes of DPF and P4R increased, and the average size of IPF remained at about $100 million 350 DPL 300 IPF 250 P4R 200 $, millions 150 100 50 0 FY02 FY04 FY06 FY08 FY10 FY12 FY14 FY16 FY of project approval Source: World Bank Analysis for Office (database) data. Note: DPL = development policy lending; FY = fiscal year; IPF = investment project financing; N = number of projects; P4R = Program-for-Results. 79 APPENDIX C ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO Figure C.2. Number of ICRs Completed for Projects Closed in Each FY, with Coverage of ICR Reviews by IEG 400 99% 350 100% 98% 100% 100% 100% 300 100% 100% 98% 100% 100% 89% 100% 250 100% Number 200 356 150 306 317 322 277 287 291 253 277 239 225 246 265 243 100 50 0 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY of project closing Source: World Bank Business Intelligence; IEG ICRR process data. Note: This report is based on 95 percent coverage (792 ICRRs completed among 839 projects closed in FY13–15 with ICRs completed and received in IEG). Ratings data is as of October 7, 2016. By comparison, the 2015 Results and Performance of the World Bank Group report was based on 96 percent coverage (861 ICRRs completed as of November 17, 2015, among 901 projects closed in FY12–14 with ICRs completed and received in IEG). FY = fiscal year; ICR = implementation completion report; ICRR = implementation completion and results review, 80 APPENDIX C ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO Table C.1. Very Large Projects Closed in FY13–15 that Drive Volume-Weighted Ratings (Net Commitment $800 Million or More) Very large projects closed in FY13–15 with IEG ratings Project Lending Global Project revised Rating Outcome Quality at M&E quality Net commitment ID Project name instrument practice Country closing date type rating entry rating rating amount 1 P115067 Support to Oportunidades Project IPF SPL Mexico 31 Dec 13 ICRR S MS High 2,753,759,398 2 P101324 Minas Gerais Partnership II SWAP IPF Governance Brazil 31 Oct 14 ICRR MS S Modest 1,434,442,627 3 P102547 India: Elementary Education (Sarva Shiksha Abhiyan II) IPF Education India 30 Sep 12 ICRR MS MS Substantial 1,350,000,000 4 P130051 Development Policy Operation—DDO DPF MFM Romania 27 Oct 14 ICRR MS MS Substantial 1,333,300,000 5 P130459 Development Policy Loan 2 DPF MFM Poland 30 Jun 14 ICRR S S Modest 1,307,800,000 6 P116226 Support to the Social Protection System in Health IPF HNP Mexico 31 Dec 13 ICRR MS MS Modest 1,250,000,000 7 P107661 BOS Knowledge Improvement for Transparency and Accountability IPF Education Indonesia 31 Dec 12 ICRR MS MS Modest 1,100,000,000 8 P103022 Ethiopia Protection of Basic Services Program Phase II Project IPF SPL Ethiopia 30 Jun 13 ICRR MS MS Substantial 1,088,331,219 9 P111665 Rio de Janeiro Municipality Fiscal Consolidation for Efficiency DPF MFM Brazil 30 Jun 13 ICRR S S Substantial 1,045,000,000 10 P112258 Private Housing Finance Markets Strengthening Project IPF Finance and Markets Mexico 31 Dec 12 ICRR MS MS Modest 1,005,471,532 11 P114154 Public Sector Development Policy DPF MFM Thailand 30 Jun 13 ICRR MS MS Modest 1,000,000,000 12 P127433 First Development Policy Loan DPF MFM Poland 30 Jun 13 ICRR S S Modest 991,400,000 13 P101653 Power System Development Project IV IPF Energy and Extractives India 31 Jul 14 ICRR S S Substantial 976,864,011 14 P096858 Fourth Export Finance Intermediation Loan (EFIL IV) IPF Finance and Markets Turkey 31 Dec 14 ICRR S MS Substantial 899,998,490 15 P127787 Competitiveness and Savings Development Policy Loan (CSDPL) DPF MFM Turkey 30 Jun 14 ICRR MS MS Modest 800,000,000 Source: IEG World Bank project ratings data; World Bank Business Intelligence data. Note: DPF = development policy financing; HNP = Health, Nutrition, and Population Global Practice; ICRR = implementation completion and results review; IPF = investment policy financing; M&E = monitoring and evaluation; MFM = Macroeconomics and Fiscal Management Global Practice; SPL = Social Protection, and Labor Global Practice; S = satisfactory rating; MS = moderately satisfactory rating. 81 APPENDIX C ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO Table C.2. Very Large Projects Closed in FY10–12 that Drive Volume-Weighted Ratings (Net Commitment $800 Million or More) Very large projects closed in FY10–12 with IEG ratings M&E Net Project Lending Global Project revised Rating Outcome Quality at quality commitment ID Project name instrument practice Country closing date type rating entry rating rating amount 1 P116020 Banking Sector Support Loan DPF Finance and Markets India 30 Jun 10 ICRR MS MU Modest 2,000,000,000 2 P115608 Mexico Framework for Green Growth DPF Energy and Extractives Mexico 31 Dec 10 ICRR MS MU Substantial 1,503,750,000 Development Policy Loan 3 P118070 MX Economic Policies in Response to the DPF MFM Mexico 31 Dec 10 ICRR S S Substantial 1,503,750,000 Crisis DPL 4 P117666 Third Development Policy Loan DPF MFM Poland 31 Dec 10 ICRR S S High 1,331,300,000 5 P116125 Poland Employment, Entrepreneurship and DPF SPL Poland 31 Dec 09 ICRR S S High 1,300,240,000 Human Capital Dev. Policy 6 P095205 First Programmatic Development Policy Loan DPF Environment and Natural Brazil 31 Dec 10 ICRR U U Negligible 1,300,000,000 for Sustainable E Resources 7 P112495 Restoring Equitable Growth And Employment DPF MFM Turkey 1 Nov 10 ICRR MS MS Negligible 1,300,000,000 Programmatic Devel 8 P115426 Energy Efficiency Development Policy Loan DPF Energy and Extractives Poland 31 Mar 12 ICRR MS MU Substantial 1,114,500,000 9 P106767 Rio Grande do Sul Fiscal Sustainability DPF Governance Brazil 31 Dec 10 ICRR MS S Substantial 1,100,000,000 10 P119856 Kazakhstan Development Policy Loan DPF MFM Kazakhstan 7 Jan 11 ICRR MS S Modest 1,000,000,000 11 P074015 Protection of Basic Services IPF SPL Ethiopia 31 Dec 10 PPAR MS MS Modest 816,348,251 12 P110643 Programmatic Electricity Sector Development DPF Energy and Extractives Turkey 31 Dec 09 ICRR S MS Modest 800,000,000 Policy Loan Source: IEG World Bank project ratings data; World Bank Business Intelligence data. Note: DPF = development policy financing; HNP = Health, Nutrition, and Population Global Practice; ICRR = implementation completion and results review; IPF = investment policy financing; M&E = monitoring and evaluation; MFM = Macroeconomics and Fiscal Management Global Practice; SPL = Social Protection, and Labor Global Practice; S = satisfactory rating; MS = moderately satisfactory rating; MU = moderately unsatisfactory rating; U = unsatisfactory rating. 82 APPENDIX C ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO Figure C.3. Trend in Percentage of Projects Rated Moderately Satisfactory or Above by IEG on Outcome and on World Bank Performance, for 10 Countries Accounting for the Top 50 Percent of Volume a. World Bank lending projects with Outcome rated moderately satisfactory or above, three-year rolling (percent) Outcome, volume-weighted, by net commitment ($) 100 Outcome, unweighted, by number of projects (N) 89 Percentage 80 75 60 Fiscal year, three-year rolling b. World Bank lending projects with World Bank Performance rated moderately satisfactory or above, three- year rolling (percent) 100 Bank performance, volume-weighted, by net commitment ($) Bank performance, unweighted, by number of projects (N) 89 Percentage 80 74 60 Fiscal year, three-year rolling Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: Includes only projects in the countries accounting for the top 50 percent of project volume in the FY10-12 and FY13- 15 groups: India, Mexico, Turkey, Brazil, Indonesia, China, Poland, Colombia, Vietnam, and Argentina. 83 APPENDIX C ADDITIONAL INFORMATION ON WORLD BANK PORTFOLIO Figure C.4. Trend in Percentage of Projects Rated Moderately Satisfactory or Above by IEG on Outcome and on World Bank Performance, for the 100+ Countries Accounting for the Bottom 50 Percent of Volume a. World Bank lending projects with outcome rated moderately satisfactory or above, three-year rolling (percent) Outcome, volume-weighted, by net commitment ($) 100 Outcome, unweighted, by number of projects (N) Percentage 84 80 72 60 Fiscal year, three-year rolling b. World Bank lending projects with World Bank performance rated moderately satisfactory or above, three- year rolling (percent) Bank performance, volume-weighted, by net commitment ($) 100 Bank performance, unweighted, by number of projects (N) Percentage 84 80 71 60 Fiscal year, three-year rolling Source: World Bank Business Intelligence and IEG World Bank project ratings data. Note: Includes projects in the countries accounting for the bottom 50 percent of project volume in the FY10 –12 and FY13– 15 groups—that is, projects in countries other than India, Mexico, Turkey, Brazil, Indonesia, China, Poland, Colombia, Vietnam, and Argentina. 84 Appendix D. Methodology for Review of World Bank Projects with Below-the-Line Ratings 1. IEG reviewed 199 ICRRs for World Bank projects completed in FY15 for which IEG ratings were available as of September 7, 2016. Among these, 158 had project outcome ratings of moderately satisfactory or above (MS+, or above the line), and 41 had project outcome ratings of moderately unsatisfactory or below (MU −, or below the line). IEG conducted a further review of the 41 projects with below-the- line ratings, focusing on the following sections of the ICRR:  Relevance  Efficacy  Efficiency  Outcome  Reason for difference in ratings between the ICR and IEG’s ICRR. 2. IEG reviewed relevant text from these 41 ICRRs and coded it using the following categories (which were not mutually exclusive):  The low rating (MU−) corresponded to evidence of significant, major, or severe shortcomings related to the following: ◦ Relevance: both overall and related to relevance of design (35 ICRRs) ◦ Efficacy: including the number of project development objectives (PDOs) achieved versus not achieved (32 ICRRs) ◦ Efficiency (36 ICRRs)  The low rating (MU−) corresponded to a lack of evidence related to the following: ◦ Relevance (0 ICRRs) ◦ Efficacy: including the number of PDOs with insufficient or inadequate evidence (18 ICRRs) ◦ Efficiency (3 ICRRs) ◦ Overall: inadequate evidence was mentioned as a reason for a downgrade in general and not specific to efficacy or efficiency (3 ICRRs)  The 13 ICRRs whose MU− outcome rating from IEG was lower than the outcome rating the ICR assigned were coded to indicate the following reason for the difference, as follows: 85 APPENDIX D ADDITIONAL INFORMATION ON IFC PORTFOLIO ◦ Lower rating assigned because the ICR did not present sufficient evidence (13 ICRRs) ◦ Lower rating assigned because the ICR did not follow the harmonized guidelines correctly (0 ICRRs) ◦ Lower rating assigned for another reason (0 ICRRs) 3. IEG used these categories to summarize the results reported (figure D.1). 4. All 41 of the MU− projects reviewed showed evidence of significant, major, or severe shortcomings in relevance, efficacy, or efficiency, and most (61 percent) mentioned significant, major, or severe shortcomings in all three elements. The three adjectives (significant, major, and severe) are important because they are part of IEG and Operations Policy and Country Services harmonized guidelines for determining an outcome rating. Part of the definition of each rating corresponds to the presence or degree of shortcomings, as follows:  Moderately unsatisfactory: “There were significant shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance.”  Unsatisfactory: “There were major shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance.”  Highly unsatisfactory: “There were severe shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance.” 5. In 23 of the 41 ICRRs, the low rating also corresponded to inadequate evidence to substantiate the outcome rating. For 18 of these 23 (78 percent), the evidence presented in the ICR was not sufficient to substantiate achievement of at least some PDOs (that is, efficacy), and five of those had insufficient evidence for all PDOs. 6. The IEG-assigned outcome rating agreed with the outcome rating the ICR assigned in 28 of 41 below-the-line ICRRs (figure D1.) IEG reviewed the reasons for the difference in ratings in the other 13 downgraded projects and found that in all cases, the reason for the downgrade was that the evidence presented in the ICR was insufficient to justify an MS rating. (In all 13 of these downgraded projects, the ICRR also found efficacy or efficiency shortcomings and evidence of those shortcomings.) Furthermore, 10 of these 13 also mentioned shortcomings in relevance of design, an ICRR rating category that assesses the project results chain’s coherence (that is, delineating how the project inputs and activities were envisioned to lead to outputs and desired development outcomes). For these 10 projects, IEG rated the relevance of design as modest, which suggests that ambiguity in the project’s results chain can inhibit collection of adequate evidence of project outcomes. 86 APPENDIX D METHODOLOGY FOR REVIEW OF WORLD BANK PROJECTS WITH BELOW-THE-LINE RATINGS Figure D.1. All ICRRs of projects with below-the-line ratings mentioned significant, major, or severe shortcomings (with evidence) in at least one of relevance, efficacy, and efficiency, and most (61 percent) mentioned shortcomings in all three.  = Shortcomings  = Shortcomings and inadequate evidence  = Downgraded, with shortcomings and inadequate evidence Source: IEG analysis. Note: Among the 41 ICRRs that rated below the line, 25 (or 61 percent) had shortcomings in all three components of outcome rating (relevance, efficacy, and efficiency). All downgrades had both shortcomings (with evidence) for some components and inadequate evidence for other components. 87 Appendix E. Analysis of Extent to Which Gender Is Incorporated in Project Development Objectives, Components, and Indicators 1. IEG piloted an addition to the implementation completion report (ICR) review process in FY16 to include systematic documentation of the presence of gender dimensions in individual World Bank projects. As part of the pilot, ICR reviewers examined the following:  Whether gender is an explicit part of the project development objective (PDO) or one of the project components  Whether the ICR reports sex-disaggregated or female- or male-specific indicators. 2. IEG conducted the pilot to extend the analysis for IEG (2016n). IEG completed the pilot for 156 projects closed in FY15 with ICR reviews completed by IEG by the end of FY16. 3. Along with the results reported in chapter 2, the gender pilot analysis also noted variation across global practice clusters in the proportion of projects in which gender appeared to be relevant, and the proportion that incorporated gender into a PDO or component. As shown in figure E.1, the Human Development cluster had the highest proportion of projects that integrated gender into a PDO or component (68 percent), followed by the Sustainable Development cluster with 33 percent and the Equitable Growth, Finance, and Institutions cluster with 12 percent. These differences are statistically significant. Less variation occurred across regions, and the differences observed are not statistically significant. 88 APPENDIX E ANALYSIS OF EXTENT TO WHICH GENDER IS INCORPORATED IN PROJECT DEVELOPMENT OBJECTIVES, COMPONENTS, AND INDICATORS Figure E.1. Percentage of Projects that Incorporated Gender in PDOs or Components, by Global Practice Cluster and Region Percentage Bank-wide (N=156) 38 Human Development (N=40) 68 Sustainable Development (N=91) 33 Equitable Growth, Finance, and Institutions 12 (N=25) East Asia and Pacific (N=28) 46 Middle East and North Africa (N=13) 46 Africa (N=37) 46 South Asia (N=18) 39 Latin America and the Caribbean (N=41) 29 Europe and Central Asia (N=19) 26 Gender was an explicit part of the PDO Gender was an explicit part of at least one component Source: IEG analysis. Note: N = number of projects; PDO = project development objective. 4. Within the Human Development and Sustainable Development Global Practice clusters, the particular global practices also varied in the degree to which they integrated gender, though the number of projects in the analysis is too small to determine statistical significance (figure E.2). 89 APPENDIX E ANALYSIS OF EXTENT TO WHICH GENDER IS INCORPORATED IN PROJECT DEVELOPMENT OBJECTIVES, COMPONENTS, AND INDICATORS Figure E.2. Percentage of Projects that Incorporated Gender in PDOs or Components, within the Human Development and Sustainable Development Global Practices Percentage Bank-wide (N=156) 38 Human Development (N=40) 68 Sustainable Development (N=91) 33 Health, Nutrition & Population (N=18) 78 Education (N=13) 62 Social Protection and Labor (N=9) 56 Social, Urban, Rural, and Resilience (N=25) 48 Agriculture (N=24) 46 Water (N=14) 36 Transport and ICT (N=12) 17 Energy and Extractives (N=9) 0 Environment and Natural Resources (N=7) 0 Gender was an explicit part of the PDO Gender was an explicit part of at least one component Source: IEG analysis. Note: ICT = information and communication technologies; PDO = project development objective. 5. Bank-wide, among the ICRs for 60 projects that explicitly included gender in the PDO or at least one component, four-fifths reported sex-disaggregated indicators, male- or female-specific indicators, or at least included the share or percentage of female beneficiaries. Figure E.3 provides details by global practice cluster and region. 90 APPENDIX E ANALYSIS OF EXTENT TO WHICH GENDER IS INCORPORATED IN PROJECT DEVELOPMENT OBJECTIVES, COMPONENTS, AND INDICATORS Figure E.3. Presence of Gender-Relevant Indicators among Projects that Incorporated Gender as Part of the PDO or at Least One Component Among projects that incorporated gender as part of the PDO or at least one component, the percentage that reported at least some gender-specific indicators 100% 50% 0% 50% 100% Bank-wide (gender subset) (N=60) Human Development (N=27) Sustainable Development (N=30) Equitable Growth, Finance, and Institutions (N=3) Health, Nutrition, and Population (N=14) Social Protection and Labor (N=5) Education (N=8) Transport and ICT (N=2) Social, Urban, Rural, and Resilience (N=12) Agriculture (N=11) Water (N=5) Middle East and North Africa (N=6) Europe and Central Asia (N=5) Africa (N=17) Latin America and the Caribbean (N=12) South Asia (N=7) East Asia and Pacific (N=13) No gender-specific reporting ICR contains sex-disaggregated or male- or female-specific indicators ICR reports only the share or percent of female beneficiaries Source: IEG analysis Note: The figure above excludes global practices with one or fewer projects that mentioned gender in the PDO or at least one component. N = number of projects; PDO = project development objective. 91 Appendix F. Additional Information on the IFC Portfolio 1. The regression model for assessing drivers of development outcome and the predictive model for FY16 factored in project size and other known influencers of the development outcome, specifically:  Outcome variable: Development outcome based on a six-point scale (1 = highly unsatisfactory; 2 = unsatisfactory; 3 = moderately unsatisfactory; 4 = moderately satisfactory; 5 = satisfactory; and 6 = highly satisfactory)  Project size: Measured by a natural logarithm of net commitment (in millions of dollars)  Real sector projects refer to IFC projects from two industries: (i) Infrastructure and Natural Resources and (ii) Manufacturing, Agribusiness and Services; Financial and Bank sector projects refer to IFC projects from the industry: Financial Institutions Group  Three internal indicators of IFC work quality (coded as binary variables) ◦ Screening, appraisal, and structuring ◦ Supervision and administration ◦ Role and contribution.  External risk factors, project level (coded as binary variables) ◦ Profit margin for real sector projects ◦ Management quality for real sector and bank projects ◦ Corporate governance for bank projects.  External risk factor, country level: Risks captured by changes in the Institutional Investor Country Credit Risk ratings (IICCR) between project approval year and evaluation year.  World Bank Group regional fixed effects: Included to control for regional- level variations in development outcome not captured by work quality and external factors. 2. The following regression analysis was conducted using ordered probit model to identify the primary drivers of development outcomes: 92 APPENDIX F ANALYSIS OF ADDITIONAL INFORMATION ON IFC PORTFOLIO Table F.1. Regression Analysis (Dependent Variable: Development Outcome Rating) Real sector projects Financial and bank sector projects Explanatory variables (1) (2) (3) (4) Project size (net commitment in log) 0.403*** 0.343* −0.101 −0.024 (0.127) (0.202) (0.191) (0.256) Mandate-to-disbursement in log 0.228 0.104 −0.380 −0.470 (0.217) (0.288) (0.368) (0.506) Work quality: screening, appraisal, and 2.520*** 0.488 structuring (0.423) (0.602) Work quality: supervision 1.603*** 1.666** (0.586) (0.778) Work quality: screening, appraisal, and 1.507*** 2.993*** structuring (0.437) (0.788) Profit margin risk: real sector projects -1.058*** −0.963** (0.290) (0.405) Management risk: real sector projects -1.248*** −1.486*** (0.368) (0.555) Corporate governance risk: financial −0.406 0.004 sector projects (0.582) (0.871) Management risk: financial sector projects −1.847** −2.046** (0.668) (0.963) Repeat client 0.142 −0.255 0.910* 1.102 (0.312) (0.605) (0.579) (0.854) IDA projects -0.169 0.633 −0.555 −0.869 (0.346) (0.605) (0.579) (0.854) Change in IICCR 0.020 −0.005 0.007 −0.027 (0.021) (0.035) (0.027) (0.044) Regional fixed effects No Yes No Yes Observations 261 234 106 104 Source: IFC, IEG. Note: Columns (2) and (4) control for IFC work quality measures and region fixed effects; columns (1) and (3) do not and are used as prediction for non-XPSR or future projects with unknown work qualities. All coefficients are estimated using logistic regressions. Robust standard errors are in parentheses. IICCR = Institutional Investor Country Credit Risk. *p < 0.1 **p < 0.05 ***p < 0.001 93 APPENDIX F ANALYSIS OF ADDITIONAL INFORMATION ON IFC PORTFOLIO Predictive Models for FY16 (IFC Investment Project’s Development Outcomes) 3. For exploring the directional implication of FY15 and FY16 projects (not rated), the same set of regressions were run by attaching more weight to the recent years, FY12–FY15, to the benchmark sample. The regressions of the predictive model were qualitatively similar to the regression model results for the current year. For FY15 and FY16 Expanded Project Supervision Report (XPSR) projects not evaluated (combined), the predicted ratings are lower than the evaluated XPSR sample for real sector projects, and the predicted ratings are higher for banking sector projects. Real Sector (INR, MAS and CTT Investment Projects) Table F.2. Predictive model for real sector projects Variable Definition log_netcomm: natural logarithm of net commitment (in millions of U.S. dollars) log_m2d: natural logarithm of “Mandate to Disbursement” (using without log does not change results) realmkt: indicator variable for high profit margin risk in real sector (1 if risky, 0 otherwise) 94 APPENDIX F ANALYSIS OF ADDITIONAL INFORMATION ON IFC PORTFOLIO realmgt: indicator variable for high management risk in real sector (1 if risky, 0 otherwise) repeat_client: indicator for repeat client (1 repeat, 0 otherwise) iiccr_start: Institutional investor country credit risk (IICCR) score at project approval diiccr: difference in IICCR score (2014—approval, use close—approval still no significance) cpia: country policy and institutional assessment score Financial Sector (FIG Investment Projects) Table F.3. Predictive model for financial sector projects bmmgm: indicator variable for high management risk in banking sector bkcghl: indicator variable for high corporate governance risk in banking sector 4. Both models feature the following variables: project credit risk ratings (CRR) at end of project, repeat client, IDA project, Mandate-to-Disbursement (M2D), log project size, IICCR at start of project, change in IICCR (approval to close year). 95 Appendix G. Analysis of Factors Affecting World Bank Group Country Development Outcome Ratings 1. IEG analyzed data from its reviews of Country Assistance Strategy Completion Reports (CASCRs) and completion and learning reviews (CLRs) to identify factors that might influence World Bank Group country development outcome ratings. IEG used two approaches:  Regression analysis to identify associations between country and World Bank program characteristics and country development outcome ratings  Review of thematic objectives articulated in the strategies to determine whether the set of objectives differs between countries with higher and lower Country Policy and Institutional Assessment (CPIA), given the relationship between country capacity and country development outcomes identified through the regression. Multiple Regression Analysis 2. IEG conducted a regression analysis to explore factors associated with World Bank Group country development outcome ratings.1 As a beginning hypothesis, IEG assumed that four, broad factors influenced country development outcomes: the level of economic development, government capacity, overall economic performance, and World Bank Group performance. The regression included country 1The analysis considered past IEG analyses. IEG (2014c) found that the quality of country assistance strategy (CAS) results frameworks is the most important ratings driver, based on analysis of 138 CAS Completion Reports (CASCRs) reviewed in FY06–13. IEG (2015o) found that selectivity is positively correlated with country development outcome ratings. Selectivity was measured by a dummy variable (1 if the number of objectives or sector boards is smaller than average) and activity size (total commitments divided by the total number of lending projects and knowledge products [analytical and advisory activities]). The previous analysis developed its own ratings for elements of World Bank Group performance, such as realism, ownership, and the quality of results frameworks, because the CASCR reviews discuss these topics, but do not provide separate ratings (unlike the country development outcome, which is based on the ratings of the objectives). This analysis builds on the previous findings to assess the influence of different factors on country development outcomes using data from World Development Indicators, the CASCR reviews, and other internal World Bank sources. 96 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS development outcome ratings (based on a six-point scale)2 as the dependent variable and the following independent variables:  Level of economic development o GDP per capita (constant 2010 US$) as of the end of the CASCR FY  Government capacity o CPIA overall score as of the end of the CASCR FY  Overall economic performance o Average GDP growth rate over the country assistance strategy (CAS) review period  World Bank Group performance o Percentage of projects at risk—average over the CASCR period o Percentage of projects that exited during the CASCR period with project outcome rated moderately satisfactory or above (MS+) by IEG o Differences between planned and actual lending amounts (1 = increase, 0 = decrease) o Lending program size (log of the number of ongoing projects at the start of the CASCR period plus newly approved projects) o Log of the number of objectives per strategy o World Bank expenditures per activity: the total World Bank budget and World Bank–executed trust funds for the CASCR period divided by the number of World Bank ongoing and exited projects and advisory services and analytics activities. 3. The analysis covered 160 CASCR and CLR reviews during FY07–16. The regression model explained a little more than 18 percent of the variation in World Bank Group country development outcome ratings. 4. The regression analysis found that country development outcomes are positively correlated with CPIA, average GDP growth rates, and country program size, and the coefficients are statistically significant at the 5 percent confidence level (table G.1). Per capita GDP and portfolio quality (projects at risk and projects rated MS+ by IEG at exit) also affect country development outcomes, but these were statistically significant at about the 10 percent confidence level. IEG found that changes to lending amounts, the number of objectives per strategy, and the World Bank budget do not factor into country development outcomes. 2The country development outcomes ratings scale is as follows: highly satisfactory = 6, satisfactory = 5, moderately satisfactory = 4, moderately unsatisfactory = 3, unsatisfactory = 2, and highly unsatisfactory = 1. 97 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS Table G.1. Correlations with Country Development Outcome Variable Coefficient GDP per capita .1180* CPIA 1.7307** Average GDP growth 0.1573** Projects at risk −2.1853* Projects MS+ at exit (IEG) 1.3021* Difference between planned and actual lending 0.3874 Lending program size (log of number of active projects) 0.5566** Log of number of objectives per strategy −0.2051 Average World Bank expenditures per activity −0.5164 Number of observations 160 Pseudo R-squared 0.1754 Source: IEG CLR ratings data; World Bank CPIA; World Bank Business Intelligence; World Development Indicators. Note: CLR = completion and learning review; CPIA = Country Policy and Institutional Assessment; GDP = gross domestic product; MS+ = moderately satisfactory or better (rating). *p<0.1 **p<0.5 ***p<0.001 Analysis of Country Objectives 5. In a CASCR or CLR review, the rating of country development outcome is based on the collective ratings of achievement of the objectives in the country program.3 Since FY14, 41 CASCR and CLR reviews are available with objective-level ratings. 6. The 41 CASCR and CLR reviews analyzed contained 618 objectives, for an average of 15.1 objectives per country (table G.2).4 CAS and country partnership 3Country strategies are divided into pillars or focal points, which are broken down into individual objectives. The World Bank Group and IEG began rating the achievement of CAS and country partnership framework (CPF) objectives in FY14 and started using these achievement ratings to determine the overall rating for country development outcome. Ratings of achievement of CAS and CPF objectives are, therefore, available for 41 CASCR and CLR reviews completed in FY14–16. Among the 41 reviews, Haiti is counted twice because the CLR review consisted of separate assessments for the CAS (FY09–12) and the Interim Strategy Note (CY12–FY14). Of the 20 CASCR reviews completed in the transition year FY14, eight contained ratings of achievement of objectives. Although the CASCR reviews for Georgia and Jamaica included some assessment of objectives, the ratings for individual objectives could not be determined, and therefore IEG excluded those two CASCR reviews. 4The CASCR and CLR reviews included 625 objectives. For this analysis, IEG dropped two because they were added ex post at the CASCR or CLR stage (after the country program 98 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS framework results frameworks varied in the number of objectives they contained, from 49 in Morocco to four in Guyana. Some country programs listed several objectives for each sector or theme, and other country programs consolidated these into one sectorwide objective.5 The number of objectives per country decreased from 18.9 in FY14 to 13.4 in FY16.6 Table G.2. Number of Countries and Objectives Covered in CASCRs and CLRs, FY14–16 FY Countries Objectives Avg. objectives per country FY14 8 151 18.9 FY15 11 173 15.7 FY16a 22 294 13.4 Total 41 618 15.1 Source: IEG analysis. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review. a. Haiti includes a separate assessment for the CAS and the Interim Strategy Note, and therefore is counted twice in FY16. 7. The achievement of each objective is rated based on evidence presented in the CASCR or CLR.7 IEG rated 51 percent of objectives achieved or mostly achieved during FY14–16 (figure G.1). was completed), and IEG assessed five objectives as not rated because the indicators were duplicated under other, similar objectives. 5For example, “improving access to water” and “institutional strengthening of the water utility” could appear as separate objectives in one country program and as one objective in another country program. 6IEG did not assess the statistical significance of differences for the analyses described in this appendix. 7 Normally this includes information on indicators, baselines, and targets presented in the results framework, along with information on actual values achieved. Sometimes other evidence presented in the CASCR or CLR complemented this information. IEG rates achievement of objectives on a four-point scale: (a) achieved (the objectives are fully met); (b) mostly achieved (good progress toward meeting the objective); (c) partially achieved (limited progress); and (d) not achieved (little progress). If the CASCR or CLR presents insufficient evidence to assess achievement of the objective, then IEG rated the objective as not verified. For this analysis, objectives rated not verified are included under not achieved. 99 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS Figure G.1. CASCR and CLR Ratings on Achievement of Objectives by Fiscal Year, FY14–16 About half of the objectives were achieved or mostly achieved. Not Achieved Partially Achieved Mostly Achieved Achieved FY14 13% 35% 19% 33% N=151 FY15 25% 22% 17% 35% N=173 FY16 19% 32% 22% 27% N=294 Total 19% 30% 20% 31% N=618 Source: IEG CLR ratings data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; N = number of objectives. 8. At the country level, ratings of achievement of objectives vary to some extent across country types. On average, 54 percent of the objectives were achieved or mostly achieved in IBRD countries, and a smaller share of 50 percent were achieved in IDA and blend countries (figure G.2).8 Figure G.2. Country Level Average of CASCR and CLR Ratings on Achievement of Objectives by Country Classification, FY14–16 IRBD countries performed better than IDA and Blend countries. Number of Countries Not achieved Partially achived Mostly achieved Achieved IBRD 18% 28% 22% 32% N=22 IDA 15% 36% 27% 23% N=15 Blend 22% 29% 21% 28% N=4 Total 17% 31% 24% 28% N=41 Source: IEG CLR ratings data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; N = number of objectives. 9. The degree of achievement of objectives decreases as the country development outcome rating deteriorates (figure G.3), which is expected because the 8 IBRD-IDA blend countries are included under IDA for this analysis. 100 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS objective ratings determine the country development outcome rating.9 The percentage of objectives achieved at the country level and its CPIA is also significantly positively correlated. Figure G.3. Percentage and Number of Country Program Objectives Achieved or Mostly Achieved in CASCRs and CLRs by Country Development Outcome Rating, FY07–16 Country development outcome ratings are reflected in achievement of objectives. Percent Number of objectives Total Number of Country outcome rating achieved or mostly achieved Countries Satisfactory 73% N=24 3 Moderately satisfactory 57% N=233 26 Moderately unsatisfactory 37% N=37 7 Unsatisfactory 27% N=20 5 Source: IEG CLR ratings data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; N = number of objectives. 10. For the second part of the analysis, IEG categorized objectives into themes and sectors.10 Some objectives were broad and, consequently, categorized into more than one theme or sector. Therefore, the total number of ratings of objectives by theme (641) and by sector (669) is higher than the number of objectives (618). Figure G.4 shows the share and rating of objectives across themes in four scenarios defined by CPIA scores and country development outcome ratings. Figure G.5 shows the ratings of objectives across themes and sectors in detail. 11. IEG’s review of text related to the 120 objectives rated not achieved suggests that lack of realism (or too much optimism) and deficiencies in CAS results frameworks might be reasons why some country program objectives are not achieved. Thirty-two percent of the objectives rated not achieved cited reasons that might reflect the World Bank Group country teams’ lack of realism about what the programs could accomplish during the country strategy period. The reasons included delays in the 9A country development outcome rating of satisfactory corresponds to a country program in which either the majority of objectives were rated achieved or mostly achieved, or the majority of objectives were rated achieved or mostly achieved and the program had exceptional development outcome or impact in one or more areas. A country development outcome rating of unsatisfactory corresponds to a country program in which either the majority of objectives were rated not achieved or partially achieved, or the majority of objectives are rated at least partially achieved, but the program had major shortcomings. 10The World Bank Group changed its themes and sectors in July 2016. IEG used the new themes and sectors to categorize objectives. 101 APPENDIX G ANALYSIS OF FACTORS AFFECTING WORLD BANK GROUP COUNTRY DEVELOPMENT OUTCOME RATINGS operation supporting the objective or dropping it (21 percent), operations taking longer than originally anticipated (11 percent), and weaknesses in the implementing agency (2 percent) (figure G.6). Insufficient evidence was cited as a reason for 23 percent of the objectives receiving ratings of not achieved (lack of information or weaknesses in the results framework, such as lack of baselines and targets), which made assessment difficult. The CASCR and CLR review did not provide the reason for the progress or lack of progress in 33 percent of the objectives rated not achieved, basing the rating on the extent of progress made toward the CAS results framework targets. 102 Figure G.4. Distribution and Ratings of Country Program Objectives in CASCRs and CLRs by Theme, FY14–16 a. CLR reviews (13) of lower CPIA countries with CDO rated MS+ b. CLR reviews (16) of higher CPIA countries with CDO rated % of objs MS+ % of objs across % of objs rated achieved or mostly across % of objs rated achieved or mostly Theme Theme achieved in high CPIA countries themes achieved in low CPIA countries themes Urban and rural development 21% 51% Urban and rural development 18% 56% Environment and natural resource mgmt 19% 40% Public sector management 18% 38% Human development and gender 15% 60% Human development and gender 17% 49% Public sector management 14% 30% Environment and natural resource… 15% 53% Social development and inclusion 11% 55% Private sector development 12% 57% Private sector development 11% 48% Social development and inclusion 7% 59% Finance 6% 47% 7% Finance 82% Economic policy 3% 38% Economic policy 5% 72% c. CLR reviews (8) of lower CPIA countries with CDO rated MU- d. CLR reviews (4) of higher CPIA countries with CDO rated MU- % of objs % of objs across % of objs rated achieved or mostly across % of objs rated achieved or mostly Theme achieved in low CPIA countries Theme themes achieved in high CPIA countries themes Urban and rural development 32% 51% Urban and rural development 20% 56% Public sector management 20% 30% Environment and natural resource… 18% 53% Human development and gender 18% 60% Public sector management 17% 38% Environment and natural resource mgmt 11% 40% Human development and gender 16% 49% Private sector development 9% 48% Private sector development 15% 57% Social development and inclusion 6… 55% Economic policy 7% 72% Finance 2% 47% Social development and inclusion 6% 59% Economic policy 1% 38% Finance 2% 82% Source: IEG CLR ratings data. Note: The CPIA score is categorized as lower or higher based on the median CPIA scores of the end year of the CASCR periods for all countries in the sample. Higher CPIA score is above the median CPIA score, and lower CPIA score is equal to or below the median score. CASCR = Country Assistance Strategy Completion Report; CDO = Country Development Objective; CLR = completion and learning review; CPIA = Country Policy and Institutional Assessment; mgmt. = management; MS+ = moderately satisfactory or better; MU− = moderately unsatisfactory or lower; objs = objectives. 103 Figure G.5. Achievement Ratings of Country Program Objectives in CASCRs and CLRs by Themes and Sectors, FY14–16 The achievement of objectives varied across sectors and themes. Not achieved Partially achieved Mostly achieved Achieved Theme Finance 6% 29% 38% 26% N=34 Economic policy 19% 19% 19% 42% N=26 Social development and inclusion 18% 24% 20% 37% N=49 Human development and gender 13% 32% 23% 32% N=100 Private sector development 11% 35% 24% 30% N=74 Urban and rural development 19% 28% 21% 32% N=145 Environment and natural resource management 30% 24% 16% 31% N=101 Public sector management 25% 40% 16% 19% N=112 Total 19% 30% 21% 30% N=641 Sector Social protection 20% 27% 23% 42% N=53 Financial sector 16% 32% 37% 24% N=49 Agriculture, fishing and forestry 30% 33% 23% 31% N=71 Industry, trade and services 14% 24% 25% 28% N=64 Education 8% 40% 22% 30% N=69 Health 9% 26% 16% 36% N=50 Information and communications technologies 16% 31% 25% 25% N=16 Transportation 25% 25% 29% 20% N=41 Public administration 21% 33% 19% 27% N=146 Water, sanitation and waste management 20% 32% 18% 27% N=49 Energy and extractives 29% 27% 10% 28% N=61 Total 19% 31% 22% 29% N=669 Source: IEG CLR ratings data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; N = number of objectives. Figure G.6. Reasons Objectives Were Rated Not Achieved Among 120 country program objectives of CASCRs and CLRs reviewed in FY07-16 No Clear Reason Given 33% Lack of realism (or excess of optimism ): Support Operation Delayed 18% Support Operation Dropped 3% Taking Longer than Expected 11% Implementing Agency Weaknesses 2% Insufficient Evidence/No Data 23% No Follow-up 3% Changing Priorities 7% Source: IEG analysis. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review. 104 Appendix H. Additional Analysis of Country Completion and Learning Review Data Figure H.1. CLRs Completed in FY07–16, Grouped by CLR Country Development Outcome Rating and Project-Level Percent MS+ Country development outcome rating in CLR Moderately Moderately Unsatisfactory unsatisfactory satisfactory Satisfactory Projects with outcome  (100 percent) rated MS+   High during the   CASCR   period   (80–99 percent) Medium-high       (60–79 percent) Medium-low       (up to 59 percent)   Low      Region of CLR: Sub-Saharan Africa East Asia and Pacific Europe and Central Asia Latin America and the Caribbean Middle East and North Africa South Asia Source: IEG CLR data. Note: Each square represents one CLR. CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review. 105 Appendix I. Regional Results and Performance Africa Region Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 Africa Region Update, 2015 economic growth, at 3.4 percent, was at the lowest rate in the Region since 2009, although it was expected to recover to 4.2 in 2016 (World Bank 2016e). The sharp decline since an average 6.8 percent growth 2003-08 reflects the challenging economic conditions in the Region’s largest economies and commodity exporters as they face headwinds from lower commodity prices, tighter financing conditions, and droughts, while economic growth shows resilience in about a quarter of the countries. Slower growth deepens the challenge of reducing poverty. The World Bank Group’s strategic framework for Africa builds on opportunities for growth and poverty reduction to support structural transformation, economic diversification and inclusion. It seeks to advance on the Sustainable Development Goals within the new development finance framework emphasizing human capital, governance, agriculture, resilience, domestic resource mobilization, affordable and reliable energy, and promoting private enterprise (World Bank 2016e). 2. The World Bank’s new lending commitments in the Region for FY16 have declined from $10.5 billion in FY14 (26 percent of all commitments) and $11.6 billion in FY15 (27 percent of all commitments) to $9.4 billion, or 20 percent of all World Bank commitments. The commitment emphasis in FY16 was in Social Protection and Labor (25 percent), Energy and Extractives (17 percent), and in Health, Nutrition, and Population and Social, Urban, Rural, and Resilience, which respectively absorbed 10 percent of FY16 commitments. The most significant FY16 shift in emphasis in U.S. dollar terms is toward Social Protection and Labor—for the four previous years, commitments under that practice averaged 10 percent against a current 25 percent (table 8). 3. The operational performance of the World Bank’s portfolio in the Africa Region remains below the World Bank average (figure 1). The gap in performance between the Africa Region and the World Bank average narrowed FY12-14 largely because of deteriorating performance in other World Bank Regions, and has remained stable FY13-15. The Independent Evaluation Group (IEG) rated development outcome as moderately satisfactory or better (MS+) for 67 percent of 106 APPENDIX I REGIONAL RESULTS AND PERFORMANCE the Africa projects (by number of projects) tAhat exited the World Bank portfolio in FY13–15 compared with the World Bank average of 72 percent. Performance improvement for the Africa Region between FY10–12 and FY13–15 is statistically significant. The portfolio performance of projects in the Africa Region from the Social Protection and Labor Global Practice performed particularly well, as did projects from other practices such as Health, Nutrition, and Population; Water; and Social, Urban, Rural, and Resilience did well relative to corresponding Bank-wide averages. Energy and Extractives, Environment and Natural Resources, Governance, Finance and Markets, and Poverty had lower average ratings, noting that the number of projects is often small in many cases (table 1). 4. During FY13-15, the success rate of 191 International Development Association (IDA) projects in the Africa Region was 69 percent; nine International Bank for Reconstruction and Development (IBRD) projects had a success rate of 89 percent; and the success rate of 21 blend operations was 48 percent. Performance was particularly strong in Mauritius (an IBRD country), where all five projects were rated MS+. As shown in table 5, all projects in the following IDA countries (in which more than one or two projects were rated) were also rated MS+: Burkina Faso (nine projects), Côte d’Ivoire (seven projects), Lesotho (four projects), Mozambique (six projects), Rwanda (five projects), and Togo (three projects). 5. A review of Implementation Completion Report (ICR) reviews for projects in the Africa Region portfolio suggests design issues, including complex and overambitious projects, overestimation of implementation capacity, and optimistic assessment of government commitment and engagement, are major weaknesses in quality at entry. 6. IEG produced 13 project performance assessment reports (PPARs) for the Africa Region in FY16. Major messages from across the PPARs included, for example, issues arising from inadequate quality at entry—complex designs, unrealistic expectations, failure to pay full attention to lessons from previous operations of the same type, or poorly articulated links with other World Bank operations. IEG also found that rapid scaling up can pose capacity challenges for countries and this can hamper planned project outcomes. The PPAR for a development policy operation (DPO) in Ghana noted that adequate macroeconomic management is a critical prerequisite for budget support, and that DPOs deployed in an extremely weak public financial management environment are unlikely to be successful. Another PPAR (Mozambique) noted that a decision to use country systems before they are adequately developed could result in mismanagement. Finally, the PPAR for the Uganda Millennium Science Initiative project noted that it is possible to establish and implement a world-class science research fund in a low- 107 APPENDIX I REGIONAL RESULTS AND PERFORMANCE income country through an integrated approach, but sustainability requires an appropriate cost-sharing agreement at entry. The project demonstrated a competitive funding facility that promoted science and technology and was implemented consistently and with a high degree of transparency. The PPAR noted that an appropriate cost-sharing agreement is needed at entry to facilitate sustainability, and that a gender focus is warranted given the low numbers of domestic scientists. 7. World Bank support financed a comprehensive effort to improve transportation services in the Lagos metropolitan area. A key factor in the MS rated project was support for the Lagos Metropolitan Transport Authority (then newly established), which was empowered to plan an integrated transportation system specifically focusing on implementing and regulating mass transit systems. A transport fund would be established to eventually cover the costs of infrastructure maintenance in the metropolitan area. A franchising program for buses was implemented for specific corridors, and it later attracted commercial funds with no government subsidy. The PPAR found that relevance of project objectives and design was substantial, though progress was hindered by the failure to implement a results framework in the first few years of the project, although the World Bank performed well during supervision to resolve issues collaboratively. The project showed that setting up a strong institutional basis for coordinated planning and regulation is crucial for the success of urban transport projects, and that the challenges cities face with urban transport and its governance are long lasting and not quickly fixed. Steady progress with institutional reforms and the establishment of the transport fund were crucial to this project’s success. 8. The PPAR for the Mozambique Poverty Reduction Support Credits 3–5 reviewed three single-tranche policy development operations that were part of a longer program. The program’s objectives were highly relevant to the challenges facing the country’s economy and were complementary to the concurrent International Monetary Fund (IMF) policy support. The efficacy of two of the three objectives (macroeconomic management through institutional reforms and reforms in governance through supporting decentralization and deconcentration) were rated modest—the program relied too much on public financial management reforms to achieve the broader macroeconomic objectives, excluding macroeconomic stability indicators. The third objective (economic development through improving the business environment, removing constraints to growth, and promoting agricultural growth) was rated substantial because it was becoming easier to start a business and the national infrastructure was improving. These operations and developments in the country also showed that the political economy context can have critical impacts 108 APPENDIX I REGIONAL RESULTS AND PERFORMANCE on policy reform achievement, and understanding this context can be crucial to assessing the likelihood that reforms will take place. 9. The PPAR for the Malawi Financial Management, Transparency, and Accountability Project found that the effective and accountable use of public resources and sound economic management are important to improve economic and social conditions in poor countries. At the time of the intervention, the country was one of the poorest in the world—75 percent of the population lived on less than $1.25 per day, and human development indicators were dismal. The project aimed to improve the accountable use of public resources through investing in financial management information systems (FMISs), strengthening public financial management processes and accountability institutions, and developing the civil service’s capacity. The project achieved considerable progress during its lifetime. However, the underlying, supporting public finance environment was not conducive for effectively using the FMIS, which led to disappointing results. FMIS users often ignored or bypassed processes. As a result, the project outcome was rated moderately unsatisfactory (MU), with high risks for the sustainability of the development outcomes actually achieved. The operation became effective just before elections, and borrower performance initially suffered from lack of commitment. A major lesson from this operation is that a functioning FMIS is not sufficient to achieve good public financial management, and changing an FMIS platform carries substantial risks. Furthermore, recurrent investments are required in both software and hardware to keep the FMIS fully functional, safe, and updated. 10. The PPAR for the Zambia Public Sector Management Program Support Project also notes that although an FMIS was implemented in that country, the legacy system prevails at the point of service delivery, and large numbers of idle balances remain outside the treasury’s purview. This raises efficiency concerns and undermines the system’s functionality as a comprehensive expenditure management tool. 11. Monitoring and evaluation (M&E) in project implementation in the Africa Region remains weak, hindered by several institutional challenges, although the improvement between FY10-12 and FY13-15 is statistically significant. For FY13–15, IEG rated M&E for projects in the Africa Region as 30 percent substantial compared with 32 percent for the World Bank as a whole (table 6)—an improvement from 26 percent for the FY12–14 period for Africa. The M&E challenges include project design complexities; lack of capacity, baseline, and other data; and the low priority the World Bank and the borrowers gave to M&E. 109 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 12. The success rate for the International Finance Corporation (IFC) Sub-Saharan Africa Region, by number of investment projects, was the same as IFC’s overall averages for 2013–15. IEG validated 46 investment projects in the Region during that period, of which 25 (54 percent) were rated mostly successful or better compared with an IFC average of 54 percent. Measured by investment commitments, the success rate for the Region was 68 percent, which was higher than the IFC average of 64 percent (table 2). The main drivers of success in the projects were conducive regulatory environment, financially strong project sponsors, and sound project structuring. Financial sector projects dominated the cohort of projects. 13. IFC identified several priority challenges for the Region, including addressing the infrastructure gap. Complex infrastructure projects generally require management teams that have a relevant record and an appropriate mix of skills and hands-on knowledge. IEG reviews of IFC investments also show the importance of IFC ensuring that sponsors have sufficient financial reserves to meet possible initial performance problems. IFC must be satisfied at appraisal that a borrower has the requisite technical expertise and practical knowledge to address technical and performance problems and develop appropriate cost-effective solutions. Without such technical and financial resources, early underperformance could have serious implications for a project and its investors. 14. Agribusiness and manufacturing productivity (often tied to issues arising from the infrastructure gap) is another priority challenge for IFC in the Region. An IEG PPAR for a grain milling project in Rwanda found that the company’s labor productivity is among the highest in the Rwandan agroprocessing sector. However, the company had to develop much of its own infrastructure (roads and electricity, such as transformers) because of the lack of basic infrastructure services, even in the special economic zone where the investment was made. This experience illustrates the general need for investors in the Region to have well-prepared contingency plans. 15. IFC expects to rely on Advisory Services projects to address these priority challenges, along with making investments. Generally, IEG found that requiring a proper baseline assessment and identifying a control group to demonstrate productivity outcomes and impacts on beneficiary incomes is an important requirement of Advisory Services projects addressing productivity issues. 16. That said, IFC Advisory Projects in the Sub-Saharan Africa Region performed poorly. IEG validated 38 IFC Advisory Services projects during the review period and assigned ratings of mostly successful or higher to 18 projects (47 percent), which 110 APPENDIX I REGIONAL RESULTS AND PERFORMANCE is well below the 61 percent overall average performance for IFC Advisory Services projects (table 3). 17. In this Region, IEG evaluated and validated 12 Multilateral Investment Guarantee Agency (MIGA) investment projects during FY13–15. Seven projects (59 percent) were rated satisfactory or higher, which is slightly higher than the MIGA average of 50 percent (table 4). To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating. The success rates for the FY13–15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Findings from Thematic, Global, and Corporate Evaluations 18. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence references relevant Bank Group support in Uganda and Nigeria (IEG 2016q). The World Bank Group’s comparative advantage in such circumstances is in supporting countries in tackling long-term development challenges, including early engagement, sustained presence, and continuous dialogue. The community-driven development model often provided the World Bank Group with a useful point of entry in conflict-affected areas, establishing a presence and contextual learning to support efforts that are more ambitious once peace is restored. The evaluation finds the World Bank Group generally pursued two broad objectives related to serious organized violence: addressing the impact of violent conflict drivers, or mitigating them. Mitigation activities at the local level (supporting communities and local institutions) were often the starting point and often based on a community-driven development approach (such as in Northern Uganda). Such early engagement (through the Northern Uganda Social Action Fund) was relevant and useful for laying the foundation for post-conflict engagement. 19. The evaluation also finds that comprehensive analytical work generally supported the World Bank Group strategies, most commonly diagnostic studies that identified the main conflict and violence drivers for ongoing conflicts, or comprehensive post-conflict needs assessments. However, the World Bank Group country strategies did not focus on gender-based challenges in a conflict context (with few exceptions), and that some critical gender-related issues (such as forced marriages by the Lord’s Resistance Army in Uganda) were not mentioned in the strategies throughout the evaluation period. Conversely, the evaluation finds that the 2007 Public Expenditure Review for Northern Uganda provided a comprehensive picture of overall public financial flows (both off- and on-budget) 111 APPENDIX I REGIONAL RESULTS AND PERFORMANCE disbursed to the Northern region and was considered to be a crucial input to finalizing the Peace, Recovery, and Development Plan (the government’s post- conflict strategy). 20. In Nigeria, the evaluation finds that public financial management provided an entry point for engagement. A general failure to provide adequate governance was one of the core conflict drivers in the Niger Delta. The 2005 country partnership strategy (CPS) for Nigeria linked the possibility of direct state-level lending to demonstrated commitment to the reform through World Bank–led public financial management diagnostics, and state governments’ commitment to address identified weaknesses. For the Delta states, the CPS emphasized actions to strengthen the demand side of governance reforms—supporting public-private dialogue, and growing coalitions around oil resources management within the framework of the Extractive Industries Transparency Initiative. The World Bank’s public expenditure management and financial accountability reviews, on which the states prepared reform action plans, created enabling conditions for direct World Bank lending to the Delta states, though implementation was slow in the last decade because the state commitment build-up was slower than expected. However, the 2014 CPS Completion Report acknowledged some progress on state-level public financial management, especially in public procurement. The evaluation concludes that the main causes of fragility in Nigeria were at the heart of the country’s developmental challenges and often beyond the World Bank’s reach. However, the World Bank’s understanding of discourse on violence in Nigeria has evolved over the years, reflecting how the World Bank is gradually becoming more comfortable focusing on the political economy of subnational violence and on developing analytical and operations instruments to address it. In that regard, community-driven development was used for information-gathering and contextual learning. As in Uganda, community-driven development projects allowed the World Bank to maintain a presence in the conflict regions even when security conditions were highly unsettled. Stakeholders perceived these projects as serving local communities instead of governments and ensured relative security for World Bank staff working on them. However, the projects were not the forum for addressing the insurgencies’ causes and consequences. 21. IEG’s 2016 learning product Lessons from Land Administration Projects: A Review of Project Performance Assessments synthesizes cross-cutting findings and lessons from IEG’s assessments of World Bank–supported projects, including a project in Ghana, dealing with often complex and politically fraught land administration issues. IEG compared World Bank Group experience across projects using different institutional structures to manage two important elements of a land administration system—registry, which records the rights to land, and the cadaster, 112 APPENDIX I REGIONAL RESULTS AND PERFORMANCE which provides information on the location, boundaries, use, and values of land parcels. The comparison suggests that no single model is best for performing these functions. The review also emphasizes that making land tenure more secure is a process, not a single event, and the concept of land tenure and associated rights is highly context-specific, meaning there are no absolute, generally applicable standards for defining tenure security (IEG 2016j). 22. The Ghana project included in the sample on which the report is based, noted a merger of six land sector agencies that, in practice, continued to work as separate entities with unconnected accounting procedures. Interventions to enhance tenure security tended to have better results when sufficiently tailored to match a number of local conditions, and incrementally strengthening the legal and policy framework was integral to the process. In this instance, an insufficient appreciation of domestic political economy factors hindered the Ghana project’s effectiveness. Project experience shows that social impacts need to be monitored and not assumed, yet most of the reviewed projects did not explicitly target the poor or vulnerable groups or reflect social inclusion in their objectives. This also occurred in the Ghana project, with a net effect of leaving the chiefs’ control over land unchallenged (which, on balance, might have reduced tenure security for many land users). 23. Managing Environmental and Social Risks in Development Policy Financing assesses the application of World Bank Operational Policy (OP 8.60) elements governing development policy financing related to implementation of the policy’s environmental and social risk management requirements to identify lessons and good practices. Examples of environmental risks not identified included an economic recovery operation in Togo, in which the program document failed to note any possible negative effects from increased fertilizer and pesticide use. Examples of unidentified social risks included a Mali poverty reduction support credit series that did not identify risks from changes in social services delivery, and a Togo DPO that cited only potential positive aspects for a number of actions, with no reference to possible social risks (IEG 2015i). 24. Transformational engagements are a crucial pillar of the World Bank Group’s current strategy. IEG’s 2016 learning product Supporting Transformational Change for Poverty Reduction and Shared Prosperity identified a sample of such engagements from 2000–14 and found that although it may be difficult to identify transformational engagements ex ante, some salient characteristics can be identified ex post. The World Bank–IFC Lighting Africa project in Kenya was among the identified engagements. The project aimed to increase the access of poor households to better, cleaner, and safer off-grid lighting. This engagement enabled transformation at two levels: the project transformed the solar lamp market in Kenya by making modern, 113 APPENDIX I REGIONAL RESULTS AND PERFORMANCE good quality, affordable lighting products available to the very poor, and it showed the commercial viability and sustainability of the approach to address the needs of the population pyramid’s base. The project considered constraints along the entire supply chain to achieve this transformation (IEG 2016p). 25. According to the IEG evaluation Financial Inclusion: A Foothold on the Ladder toward Prosperity, Tanzania is among the least financially inclusive countries in the Africa Region. The evaluation found that most banks focused on the commercial sector or, when they catered to retail banking, serving salaried workers. By contrast, for an IFC microfinance institution (MFI) investee in the country, the cost of resource mobilization (collecting savings), high operational costs, and low loan officer productivity posed major challenges. The MFI eventually broke even and recovered all its losses, though two years later than expected. The MFI now reaches the poor and microenterprises, even though average loan volumes remain higher than originally expected. One of the lessons from various IFC investments in greenfield MFIs in the Region is that they might simply take longer to turn profitable than originally anticipated, and therefore could need a longer preimplementation phase (IEG 2015b). 26. Certain IEG evaluations referenced in the Africa Region Update as part of Results and Performance of the World Bank Group 2015 (IEG 2016n) are again noteworthy given the particular, ongoing importance of relevant subject matter from the Africa Region perspective. IEG’s 2015 Evaluation World Bank Group Support to Electricity Access, FY2000–2014: An Independent Evaluation cited good practice, national access scale-up experiences worldwide that are relevant for Africa, including experiences in Africa itself –Kenya and Rwanda –and experiences from further afield, such as Bangladesh, Indonesia, the Lao People’s Democratic Republic, and Vietnam. With World Bank support, Kenya and Rwanda prepared the first sectorwide programs in the electricity sector with national electricity access rollout plans using least-cost combinations of grid and off-grid electrification. After long periods of stagnation, access levels increased from 6 percent to 15 percent in Rwanda and from 23 percent to 30 percent in Kenya. Good practice experiences in off-grid solar home systems (Bangladesh and Mongolia, for example) are especially relevant for several Sub-Saharan Africa countries in fragile situations, with dispersed populations, or whose sector conditions are not ready for systematic and rapid scale-up (IEG). 27. IEG’s 2015 evaluation World Bank Support to Early Childhood Development: An Independent Evaluation refers to relevant World Bank support in Mozambique and Senegal. According to the evaluation, a positive correlation exists between analytical work on early childhood development (ECD) and subsequent successful World 114 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Bank operations. IEG cites Mozambique where the Minister of Education requested support for an ECD program within a week after a presentation of dismal child development indicators from an impact evaluation of an ECD pilot. Furthermore, in Senegal, three nutrition-related knowledge products and one impact evaluation were implemented before the World Bank supported five interventions in nutrition. IEG rated development outcome and borrower performance on each of two nutrition interventions in Senegal highly satisfactory and satisfactory (IEG 2015s). 28. Although poverty and economic diversification were central to development strategies and planning in the resource-rich countries in Africa, such as Zambia and Nigeria, the results were disappointing. Sustained growth and economic and export diversification proved elusive, with persistent reliance on the dominant extractive industry for government revenues and exports. Progress in reducing poverty, containing income inequality, and creating employment was limited. Zambia was one of four countries included in an in-depth IEG evaluation of World Bank Group engagement in resource-rich developing countries (IEG 2015p). The Zambia CPS focus on macroeconomic and public financial management, investing in infrastructure, improving the business climate, and encouraging private investment was generally consistent with the evaluation’s findings on the key drivers of progress on shared growth and diversification. However, results were limited because the program implementation was less than effective. IEG rated the Zambia CPS FY08–12 as MS on both outcome and World Bank performance. 29. The section on cross-regional findings also presents summary findings from the learning product, World Bank Group Support for Housing Finance (IEG 2016s). Evaluation Findings from Country Program Performance in the Africa Region 30. Analysis for Results and Performance of the World Bank Group 2016 shows that for the 20 countries in the Africa Region for which IEG validated at least two country strategies during FY07–16, the rating for overall development outcome improved from MU to moderately satisfactory (MS) in one case, remained at MS for five countries, remained at MU for eight countries, and declined from MS to MU for six countries.1 31. IEG validated five completion and learning reviews (CLRs) for countries in the Africa Region during FY 16 (Botswana, Côte d’Ivoire, Lesotho, Mali, and Uganda) and 1The 20 countries included in this analysis are: Benin, Burkina Faso, Cabo Verde, Ethiopia, The Gambia, Ghana, Kenya, Lesotho, Malawi, Mali, Mauritania, Mozambique, Niger, Nigeria, Rwanda, Senegal, South Africa, Tanzania, Uganda, and Zambia. 115 APPENDIX I REGIONAL RESULTS AND PERFORMANCE one country program evaluation (CPE) for Zambia, which is also part of a CPE for resource-rich developing countries. IEG rated development outcome MS in only one of the CLRs (Côte d’Ivoire) and rated World Bank performance as good in that instance. Two countries (Lesotho and Uganda) are rated MU for development outcome (World Bank performance was rated fair in both instances). For the final two countries (Botswana and Mali), IEG rated development outcome unsatisfactory and World Bank performance fair (table 7). Several common themes are apparent across the five CLRs, although to varying degrees between countries:  Focus and selectivity in line with country capacities is important. The CLR validation for Côte d’Ivoire noted that the World Bank Group’s program ambitiously sought to cover almost all elements of the government program. In the case of Lesotho, the originally planned program was overambitious and complex, with proposed interventions covering a wide range of sectors. The Uganda CLR validation stated that the country assistance strategy (CAS) program was based on an optimistic reading of the country’s capacity.  Results frameworks were inadequate or incomplete. The Mali CLR validation noted that the results frameworks did not identify the policies needed to achieve the CAS objectives. Uganda’s indicators were not always chosen appropriately, and Botswana’s results matrix as a whole was overloaded, often imprecise, and unduly process oriented.  Achieving institutional reforms is difficult. This was the main shortcoming for the Mali program. The pillar for the Uganda program on strengthening governance and institutions was only partly achieved. The envisioned outcomes of fiscal consolidation, reduction in wage bill, and strengthening public financial management in Lesotho were not achieved. 32. The Côte d’Ivoire CLR validation noted IFC’s links between investments and CPS outcomes were sometimes not clear. Thus, for Pillar 2 (improving the performance of the agricultural sector), IFC provided financing to major cocoa trading companies, but it was not clearly articulated how this would lead to a significant increase in rural incomes. In other cases, IFC’s program and its outcomes were well linked to the country program’s objectives. For example, IFC’s investments in the power sector (supported by MIGA guarantees) were expected to contribute to Pillar 4 (renewing infrastructure and basic services) through investments in generation capacity and natural gas supply development. 33. The Uganda CLR validation found that the emphasis of IFC’s program was on focus area 1 (promote shared and sustainable economic growth) and focus area 2 (enhance public infrastructure) and led to advances in the ease of doing business through support for extending financial institutions’ reach to small and medium 116 APPENDIX I REGIONAL RESULTS AND PERFORMANCE enterprises. IFC-supported projects in roads, railways, and hydropower and financed the Rift Valley Railways to rehabilitate the Kenya-Uganda railway. MIGA’s program was also oriented toward focus areas 1 and 2 through support for the energy sector and foreign direct investment in agribusiness and manufacturing. Furthermore, IEG specifically noted that the IFC and MIGA CAS results framework should integrate their programs more fully and should select indicators that will clearly articulate the links between all interventions and progress toward CAS objectives. 34. Risk assessment with relevant mitigation measures facilitates program design and appropriate responses to unanticipated developments. The five CLR reviews found that the CPSs generally identified the programs’ main risks appropriately, but the Lesotho CPS did not anticipate political instability. The Mali CPS correctly identified the risks, but it underestimated the level of political instability—in reality, the conflict in the northern areas exploded. In these cases, the identified mitigating measures were not sufficiently robust and implementation was weak. These examples also emphasize the need for a realistic understanding of political economy issues. Figure I. Africa Region: Percentage of Projects Rated MS+ on Outcome FY01–15 100 95 Bank-wide Africa 90 83 Non-Africa Regions 85 81 81 78 78 78 Percentage 80 77 75 75 74 74 73 72 75 70 65 70 70 67 68 67 60 64 63 64 64 63 63 65 62 55 50 Fiscal year, three-year rolling Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 117 Table 1. Africa Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13–15) Africa Region Other Regions Bank-wide Net Net Net Number of Number of Number of commitments commitments commitments projects projects projects (US$, millions) (US$, millions) (US$, millions) Global practice Rated Rated Rated Rated Rated Rated Total Total Total Total Total Total MS+ MS+ MS+ MS+ MS+ MS+ number amount number amount number amount (%) (%) (%) (%) (%) (%) Agriculture 25 76 1,135 91 48 81 2,948 89 73 79 4,083 89 Education 25 72 1,485 66 55 75 5,090 90 80 74 6,576 85 Energy and Extractives 24 58 1,016 48 49 69 6,247 84 73 66 7,264 79 Environment and Natural Resources 16 31 294 39 34 76 1,316 90 50 62 1,610 81 Finance and Markets 8 50 171 33 26 81 4,830 98 34 74 5,001 96 Governance 16 44 717 51 39 62 3,989 78 55 56 4,706 74 Health, Nutrition, and Population 30 77 1,510 91 37 70 3,836 80 67 73 5,345 83 Macroeconomics and Fiscal Management 17 76 808 85 32 84 10,043 98 49 82 10,850 97 Poverty 2 50 20 51 5 80 1,155 100 7 71 1,175 99 Social Protection and Labor 11 100 1,659 100 29 86 4,760 97 40 90 6,419 98 Social, Urban, Rural, and Resilience 20 75 1,245 83 76 70 7,337 81 96 71 8,582 81 Trade and Competitiveness 11 64 505 86 6 67 510 22 17 65 1,015 54 Transport and ICT 21 71 1,161 75 57 82 8,478 91 78 79 9,639 89 Water 16 75 932 91 56 66 5,137 71 72 68 6,069 74 Total 242 68 12,658 79 549 74 65,675 88 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; total number refers to the total number of projects rated by IEG. 118 Table 2. Sub-Saharan Africa: IEG Outcome Ratings of IFC Investment Projects, FY13–15 Sub-Saharan Africa IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 11 55 334 47 34 38 904 56 and Funds Financial Markets 15 53 312 73 70 59 1,705 62 Infrastructure and Natural 5 60 281 91 41 63 1,518 79 Resources Manufacturing, Agribusiness, 15 53 245 66 84 51 2,105 59 and Services Total 46 54 1,172 68 229 54 6,232 64 Source: IEG data. Note: IFC’s categorization of Region uses Sub-Saharan Africa instead of Africa. MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. Sub-Saharan Africa: IEG Outcome Ratings of IFC Advisory Projects, FY13–15 Sub-Saharan Africa IFC overall IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 8 63 43 70 Cross-Cutting Advisory Solutions 1 0 4 75 Cross-Cutting Advisory Solutions: Public-Private 2 50 14 43 Partnerships Environment, Social, and Governance 0 n.a. 1 100 Financial Markets 0 n.a. 5 40 Financial Institutions Group 4 25 17 47 Infrastructure and National Resources 0 n.a. 2 100 Investment Climate 6 67 26 77 Manufacturing, Agribusiness, and Services 1 0 3 33 Public-Private Partnerships 3 67 17 47 Sustainable Business Advisory 11 36 38 66 Telecom, Media, Technology, and Venture Capital 0 n.a. 1 0 Trade and Competitiveness 2 50 8 50 Total 38 47 179 61 Source: IEG data. Note: IFC’s categorization of Region uses Sub-Saharan Africa instead of Africa. MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. During FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. 119 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 4. Sub-Saharan Africa: MIGA Development Outcome Ratings by Sector, FY13–15 Sub-Saharan Africa MIGA overall Sector Total number Rated S+ (%) Total number Rated S+ (%) Agribusiness 1 0 1 0 Financial 1 0 8 38 Infrastructure 4 50 9 44 Manufacturing 3 67 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 1 0 2 0 Services 2 100 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 12 50 34 47 Source: IEG data. Note: MIGA’s categorization of region uses Sub-Saharan Africa instead of Africa. S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Table 5. Africa Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank IFC IEG XPSR IFC IEG PCR Projects Ratings Ratings Total Rated Total Rated Total Rated Country number MS+ (%) number MS+ (%) number MS+ (%) Angola 1 100 Botswana 1 100 Gabon 1 0 IBRD Mauritius 5 100 1 0 Namibia 1 100 Seychelles 1 0 South Africa 1 100 3 67 IBRD Total 9 89 5 60 1 0 Cabo Verde 3 67 Cameroon 5 20 1 100 Blend Congo, Rep. of 3 67 Nigeria 10 50 4 75 6 50 Blend Total 21 48 5 80 6 50 Benin 3 33 Burkina Faso 9 100 IDA Burundi 7 57 Central African Republic 2 0 120 APPENDIX I REGIONAL RESULTS AND PERFORMANCE World Bank IFC IEG XPSR IFC IEG PCR Projects Ratings Ratings Total Rated Total Rated Total Rated Country number MS+ (%) number MS+ (%) number MS+ (%) Chad 7 29 1 100 Comoros 4 75 Congo, Dem. Rep. of 6 83 1 100 1 0 Cote d'Ivoire 7 100 Ethiopia 13 77 2 50 Gambia, The 4 75 1 0 Ghana 8 63 4 75 1 100 Guinea 13 46 Guinea-Bissau 2 50 Kenya 5 80 3 33 7 43 Lesotho 4 100 1 100 Liberia 1 100 1 0 2 0 Madagascar 8 50 Malawi 8 75 1 0 1 0 IDA Mali 5 40 1 100 Mauritania 7 43 Mozambique 6 100 1 100 1 100 Niger 4 75 1 0 Rwanda 5 100 3 100 2 100 Sao Tome and Principe 1 100 Senegal 9 89 1 100 1 0 Sierra Leone 5 80 1 0 Somalia 1 100 South Sudan 8 50 1 0 Sudan 5 80 Tanzania 7 43 2 0 Togo 3 100 Uganda 9 67 2 50 1 100 Zambia 5 80 2 50 3 33 IDA Total 191 69 23 57 27 44 Africa Region 21 62 7 43 3 67 East Africa Region 1 100 1 100 Other South Africa Region 1 100 Western Africa Region 1 100 1 0 Other Total 22 64 10 50 4 75 TOTAL 243 67 43 58 38 47 Source: IEG data. Note: IFC’s categorization of Region uses Sub-Saharan Africa instead of Africa. The data excludes projects not mapped to a Region. MS+ = moderately satisfactory or better (World Bank rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number refers to the total number of projects rated by IEG. 121 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 6. Africa Region: IEG Ratings of Project M&E Quality, FY13–15 Africa Region Other Regions Bank-wide Rated Rated Rated Total number substantial Total number substantial Total number substantial (%) (%) (%) IBRD 10 40 326 33 336 33 Blend 21 14 86 29 107 26 IDA 185 30 124 35 309 32 Other 22 41 13 46 35 43 Total 238 30 549 33 787 32 Source: IEG data. Note: The data excludes projects not mapped to a Region. M&E = monitoring and evaluation; total number refers to the total number of projects rated by IEG. Table 7. Africa Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 World Bank IFC MIGA FY of CASCR/CLR Outcome Country performance performance performance review review period rating rating rating rating Burkina Faso FY10–12 MS MS Ghana FY08–12 MU MU Reviewed & Reviewed & Guinea FY04–13 not rated not rated Kenya FY10–14 MS Fair Liberia FY09–12 MS MS 2014 Mauritania FY07–12 U U Nigeria FY10–13 MS MS Rwanda FY09–13 S Superior São Tomé and FY06–12 MS MS Príncipe South Africa FY08–12 U MU 2015 Cabo Verde FY09–12 MU Fair Botswana FY09–13 U Fair Côte d'Ivoire FY10–14 MS Good 2016 Lesotho FY10–14 MU Fair Mali FY08–15 U Fair Uganda FY11–15 MU Fair Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory; MU = moderately unsatisfactory; S = satisfactory; U = unsatisfactory.Table 8. Africa Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16 122 Table 8. Africa Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16. 2012 2013 2014 2015 2016 % of % of % of % of $, $, % of Region Global Practice $, millions Region $, millions Region millions Region $, millions Region millions total total total total total Agriculture 480 6 850 10 602 6 1,120 10 589 6 Education 127 2 225 3 705 7 670 6 623 7 Energy and Extractives 1,441 19 1,132 14 1,772 17 1,020 9 1,589 17 Environment and Natural 560 7 57 1 71 1 250 2 0 0 Resources Finance and Markets 95 1 40 0 330 3 622 5 225 2 Governance 297 4 237 3 335 3 312 3 258 3 Health, Nutrition, and 534 7 419 5 439 4 2,041 18 920 10 Population Macroeconomics and Fiscal 1,200 16 623 8 856 8 2,038 18 530 6 Management Poverty 30 0 0 0 5 0 9 0 136 1 Social Protection and Labor 761 10 1,244 15 684 7 1,011 9 2,305 25 Social, Urban, Rural, and 641 9 654 8 1,229 12 497 4 970 10 Resilience Trade and Competitiveness 70 1 225 3 360 3 150 1 277 3 Transport and ICT 326 4 1,874 23 1,471 14 1,282 11 554 6 Water 963 13 666 8 1,591 15 548 5 391 4 % of total % of total % of total % of total $, $, % of total Region $, millions IBRD and $, millions IBRD and millions IBRD and $, millions IBRD and millions IBRD and IDA IDA IDA IDA IDA Africa Region 7,525 21 8,245 26 10,451 26 11,569 27 9,365 20 Other Regions 27,731 79 23,302 74 30,231 74 30,926 73 36,534 80 Total 35,256 31,547 40,681 42,495 45,899 Source: World Bank Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. 123 East Asia and the Pacific Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 East Asia Pacific Regional Update, the Region’s client base is exceptionally diverse in income, resources, population size, and fragility. Extreme poverty has been falling rapidly, from almost 60 percent in 1990 to about 7 percent in 2012, and prosperity was shared, but vulnerabilities persist. The World Bank’s East Asia and Pacific Regional Update for 2016 notes that growth in China continued to moderate, but held steady in the rest of developing East Asia and Pacific (World Bank 2016f). The establishment of the Association of Southeast Asian Nations Economic Community in December 2015 helped bolster trade and the flow of foreign direct investment. Going forward, the region faces rapid middle class growth accompanied by intensifying urbanization, an aging population, and managing the potentially conflicting energy requirements of expanding economies and the impact of climate change. Against that broad backdrop, the World Bank Group’s strategy is designed to address inclusion and empowerment, jobs and private sector–led growth, enhanced governance, infrastructure and urbanization, and climate change and disaster risk management (World Bank 2016f). 2. The U.S. dollar amount of the World Bank’s new lending commitments in the East Asia and Pacific Region trended slightly upward in recent years: $6.2 billion in FY13, $6.3 billion in FY14, $6.3 billion in FY15, and $7.5 billion in FY16. FY13 commitments in the Region represented 20 percent of all World Bank commitments, and FY16 commitments represent 16 percent (table 8). 3. The performance of World Bank operations in the East Asia and Pacific Region was 73 percent moderately satisfactory or better (MS+) in FY13–15, slightly above the average performance (72 percent MS+) for the World Bank as a whole (figure 1). 4. About 60 percent of the 143 projects rated for East Asia and Pacific in FY13–15 are concentrated in four Global Practices: Education (16), Energy and Extractives (15), Social, Urban, Rural, and Resilience (28), and Transport and Information and Communication Technologies (20). The Region outperforms the World Bank average in all four cases based on rated projects and commitments except for performance by commitment in Energy and Extractives, which dips slightly below the World Bank average (table 1). 124 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 5. IEG rated 26 IDA projects and 41 blend projects in the Region between FY13 and FY15, of which 85 percent of IDA projects and 76 percent of blend projects were rated MS+—better than the performance of the 76 IBRD projects rated during the same period (68 percent MS+). At the individual country level, performance was particularly strong in two IDA countries (Cambodia and the Solomon Islands), in China (IBRD), and in two blend countries (Mongolia and Vietnam). Table 5 shows that performance was well below the regional and IBRD average in Indonesia (63 percent), and the Philippines (47 percent). During this period only three projects were rated in Thailand, two of which were rated MS+. 6. IEG evaluations found that successful projects reflect a strong results framework defined through a clear, logical sequence between project activities, output, outcomes, and development objectives, along with a sound reading of the political economy and the application of relevant learning from past projects. However, common weaknesses include overambitious objectives relative to project components or time frame, complex project design involving multiple components and implementing agencies (lack of strategic selectivity), and overestimating client buy-in and system capacity. 7. IEG produced project performance assessment reports (PPARs) for projects in Indonesia and Vietnam in FY16. Development outcome for the Indonesia Climate Change Development Policy Loan was rated moderately unsatisfactory (MU). The PPAR notes that Indonesia is both the third largest emitter of greenhouse gases in the developing world and one of the most vulnerable countries to the adverse effects of climate change and extreme weather events. To support efforts to deal with climate change, the World Bank joined other development partners in the second stage of an ongoing, multiyear policy-based loan. The PPAR notes, despite the relevance of the project’s objectives and its substantially relevant design, its efficacy with regard to both its low carbon and climate resilience objectives was modest, and its overall outcome is rated moderately unsatisfactory (MU). An important lesson identified by the evaluation is that, even where a DPL is unsuccessful with reference to its own objectives, it may play a positive and strategically important role as part of longer-term Bank-borrower relationships and this can, in turn, open the way to effectively addressing an emerging complex development challenge (climate change). 8. The PPAR for the Second Higher Education Project in Vietnam rates development outcome as satisfactory. Project objectives—increase the quality of teaching to improve the employability of graduates, and increase the quality of research to improve the relevance of research—were considered particularly relevant. Design was strong, based on years of detailed technical work and informed 125 APPENDIX I REGIONAL RESULTS AND PERFORMANCE by lessons from an earlier project and stakeholder engagement. The PPAR describes World Bank supervision as responsive, proactive, efficient, and supportive. 9. IEG rated 33 percent of East Asia and Pacific Region projects during FY13–15 substantial or better on monitoring and evaluation (M&E) (table 6). Weaknesses in M&E reflect a series of perennial issues, including poor results chains, overreliance on output indicators and a lack of outcome indicators, using indicators not directly relevant to project objectives or not directly attributable to project activities, and a lack of baseline data that precludes measurement of progress (if any). 10. As noted in IEG’s East Asia and Pacific Regional Update section of Results and Performance of the World Bank Group 2015: An Independent Evaluation, management in the East Asia and Pacific Region is acting to address performance challenges, particularly related to results and M&E (IEG 2016n). Work is ongoing in this regard after analysis and desk review of 42 projects across the Region’s country management units (CMUs). The Region, and particularly the Vietnam CMU, are working with Operations Policy and Country Services (OPCS) to implement a results and M&E operational program for operational staff that included three complementary activities: a results and M&E training workshop, results and M&E operational clinics, and an M&E framework assessment at the CMU level. The impact of that training and investment will be seen in future projects. For now, the Region’s M&E performance is average, although the performance of individual countries noted in the previous paragraph is relatively impressive. 11. IEG validated 28 IFC investment projects for FY13–15 and assigned ratings of mostly successful or higher to 15 projects (54 percent), which was the same as the IFC average. The Region’s success rate by investment commitment was 63 percent, dipping slightly below the 64 percent average IFC success rate (table 2). 12. IFC’s regional strategy emphasizes addressing climate change and sustainable infrastructure. In this context, a recent PPAR assessed an investment to support the growth of a medium-size private player in the water and wastewater industry, and the lesson on environmental and social safeguards from this PPAR is particularly relevant. It suggests that IFC environmental and social teams should be more thorough in portfolio supervision visits and should continue to record gaps or deviations in implementation, which would strengthen IFC’s overall work quality. An important lesson from an assessment of a highly successful IFC investment in a high- risk country in the region suggests that partnerships with committed sponsors who have proven operating records are key, particularly during tough economic cycles. 126 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 13. IEG validated 27 IFC Advisory Services projects during the review period and assigned ratings of mostly successful or higher to 17 projects (63 percent), which is slightly higher than the 61 percent overall average performance for IFC Advisory Services projects (table 3). A lesson from a recent IEG Evaluative Note (EvNote) for a Pacific Island Regulatory Simplification and Investment Policy and Promotion Advisory Services Project suggests that performance reporting systems need to be simple. Furthermore, reporting systems need to provide useful information to the client as a basic performance management tool. This could mean sacrificing data richness and complexity in favor of pragmatism. A lesson from an IEG Evaluative Note (EvNote) for an Advisory Services project in Vietnam notes that when major policy changes are required (the tax environment, in this case), IFC should conduct a thorough scoping exercise to better comprehend the complexity of undertaking the reforms and set a more realistic time frame and targets. Setting priorities for and sequencing activities is crucial to ensuring quick wins to create and maintain momentum while working on longer-term reform initiatives. 14. IEG evaluated and validated four Multilateral Investment Guarantee Agency (MIGA) investment projects in the Region during FY13–15 of which none were rated satisfactory or higher (table 4). To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Findings from Thematic, Global, and Corporate Evaluations 15. IEG’s 2016 evaluation World Bank Group Engagement in Situations of Fragility, Conflict, and Violence: An Independent Evaluation references relevant support in the Philippines and Indonesia (IEG 2016q). In the Philippines, the Bangsamoro Development Agency (BDA) was created as part of an agreement between the government and the Mindanao Islamic Liberation Front to determine, lead, and manage relief, rehabilitation, and development programs in the conflict-affected areas in Mindanao. The World Bank began working with the BDA in 2005 with the tacit approval of the Philippine government, and this collaboration encouraged other donors to get involved, helping to cement its capacity and ongoing legitimacy. With support from the World Bank–administered Mindanao Trust Fund, the BDA has contributed to economic recovery in conflict-affected areas, benefiting more than 500,000 people in 75 municipalities across Mindanao. As of 2015, more than 300 BDA staff members operate with a central management office that oversees six regional offices across Mindanao. 127 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 16. Country programs for Indonesia began paying more attention to the province of Aceh, although most World Bank Group activities did not officially relate to the conflict there. However, scaled-up field presence and expertise facilitated the implementation of key programs (often funded by other donors), which were essential to sustain the end of the conflict. The World Bank’s program in Aceh wound down almost as swiftly as it built up. The World Bank Group left Aceh soon after the tsunami reconstruction was complete and closed its office in 2012 (IEG 2016q). 17. Managing Environmental and Social Risks in Development Policy Financing identifies the stand-alone Tonga energy development policy loan as an example of good practice in addressing several environmental aspects. The main action was to support development of an energy road map, which consisted of a set of policy measures and investments intended to address major energy sector weaknesses. Environmental sustainability was included as a core design principle of the operation in its support for renewable energy development and energy efficiency. The project accounted for the possibility that infrastructure development, if not well designed, could lead to negative environmental effects, and it built several mitigation measures into the project design. The project screened renewable energy investments for environmental risks and discarded any options with significant environmental risks it could not mitigate or avoid. The road map included requirements for environmental impact assessment and mitigation plans for new energy investments, and the goal was to make the assessment and mitigation plans routine requirements for other infrastructure sectors in Tonga. The operation also included significant public consultation (including on environmental aspects), linking this to the government’s intention to establish ongoing communication and consultation with citizens and public stakeholders (IEG 2015i 18. The Quality of Results Frameworks in Development Policy Operations, a 2015 IEG learning product, presents the results of a review of the World Bank’s DPF portfolio based on a sample of operations from across the World Bank Regions, including a project in Thailand. The review found that the Thailand Public Sector Development Loan, which focuses on public financial management and service delivery, did not address public investment management in prior actions, although this was an area identified in the program document as one of the most challenging reform areas. IEG also found teams are often better equipped to identify output-oriented results indicators instead of outcome-oriented indicators because of intrinsic difficulties in measuring policy and institutional change. Although flexibility is a necessary component of the programmatic approach, it can sometimes be excessive (as in the Vietnam Poverty Reduction Strategy Credit), resulting in significant weakening caused by frequent program adjustments. However, recent changes that shifted the 128 APPENDIX I REGIONAL RESULTS AND PERFORMANCE preparation of investment completion reports for DPOs from six to 12 months after project closure are a positive step that will enhance the World Bank’s ability to capture development results in line with stated objectives (IEG 2015l). 19. Working with a sample of World Bank and IFC projects from across the regions, IEG’s 2016 learning product World Bank Group Support for Housing Finance sought to generate knowledge and identify lessons regarding investment, knowledge, advisory, and other support for housing finance. The review included an IFC intervention in Vietnam providing a local currency credit line of approximately $50 million to a commercial bank was based on the use of a long-term cross-currency swap. When the swap market collapsed in 2008, IFC could not disburse $20 million. The project evaluation concluded that a long-term cross- currency swap, which was the fulcrum for IFC’s provision of a long-term local currency loan, was untenable in Vietnam, which does not have a substantive market for such swaps, and the financial system essentially operates on a short-term basis. IEG’s review also underscores the importance of having technical assistance in place (IEG 2016s). 20. IEG’s 2016 learning product Lessons from Land Administration Projects: A Review of Project Performance Assessments synthesizes cross-cutting findings and lessons from IEG’s assessments of World Bank–supported projects dealing with often complex and politically fraught land administration issues. The Thailand Land Titling Program successfully administered a systematic titling program that contributed to several development outcomes, such as enhancing access to credit. The Indonesia Land Management and Policy Development Project was part of a series of land titling projects in the East Asia and Pacific Region modeled on the Thai program design. Assuming that the Thai model could be successful in other countries resulted in a design that emphasized land titling and the cadastral survey. Broader issues regarding the regime of land rights reform were treated as ancillary and were relegated to a series of policy studies. However, the land rights context in Indonesia was different from that in Thailand (for example, a more complicated mix of land tenure traditions and a registration process that itself does not guarantee ownership). A land titling project in Lao Peoples Democratic Republic exceeded its titling targets, but the absence of impartial and efficient enforcement of the new land titles undermined tenure security. The project included components for policy and regulatory reform, but they were not sufficient, and their implementation was not well supported. Capacity-building efforts aimed at improving the land administration institutions’ efficiency and transparency got little traction, and the project assessment found, among other things, that too many land transactions continued to be conducted informally to avoid high transaction costs, threatening to make titles that had been issued obsolete (IEG 2016j). 129 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 21. Furthermore, the review found the overall level of political commitment in support of a long-term program for land administration was weaker in both Lao Peoples Democratic Republic and Indonesia than in Thailand. This affected the rate of progress on institutional reforms and the sustainability of gains from the titling components. Assessments of those projects found that land titling projects might not achieve or sustain benefits if the government is not willing to commit to a long-term program of land administration and allocate the necessary budget. Project assessments also show, despite the exercise of a systematic process, the poor may still be neglected due, for example, to the informal practice of making side payments to officials, resulting in higher costs than legally stipulated, thus reducing the demand for titles by those of lesser means (IEG 2016j). Evaluation Findings from Country Program Performance in the Region 22. Analysis undertaken for Results and Performance of the World Bank Group 2016 shows that for the three countries—Indonesia, Philippines, Vietnam—in the East Asia and Pacific Region for which IEG validated two or more country strategies during FY07–16, the rating for overall development outcome remained moderately satisfactory (MS). 23. IEG completed two completion and learning reviews (CLRs) for the East Asia and Pacific Region in FY16. Implementation of the Country Partnership Strategy for Indonesia (FY13–15) was rated MS, and World Bank performance was rated good. The CLR did not provide a rating for Cambodia (FY05–15) because of data limitations (table 7). 24. Key lessons from the CLR for Indonesia reference the importance of gauging the political economy (good technical design alone does not ensure the success of a project or a program) and developing local capacity to underpin prospects of success (for example, the ability to effect prompt procurement and the issuance of appropriate environmental permits). Another important lesson is the need to be more strategically selective and focused—the three-year program had 15 objectives, eight of which were considered achieved or partially achieved. The CLR found that the results chain was robust, but commented on the lack of quality indicators. The indicators sometimes captured process and activity and, when focused on outcome, tended to refer to the country level instead of program-derived outcomes. The CLR suggests more time could have been devoted to the implementation of infrastructure projects, in which long implementation delays are evident. IFC’s program had substantial relevance, with emphasis on the focus area Pro-Growth—Promoting Prosperity, and aimed at addressing issues of connectivity, competitiveness, financial sector, and infrastructure. 130 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 25. The CLR for Cambodia reemphasizes lessons identified in the CLR for Indonesia, particularly the need to understand the political economy and the extent of buy-in, and the need to develop and ensure local capacity. In that regard, the CLR suggests using technical assistance projects to perform advance, comprehensive, and in-depth assessment of borrower commitment to implement reforms. It also suggests that making safeguard supervision plans during project preparation and clarifying respective roles and responsibilities will make safeguard violations less likely. The CLR draws attention to specific safeguard and fiduciary issues that affected the World Bank program and resulted in the suspension of new IDA financing. In that regard, the Inspection Panel found that the Forest Concession Management and Control Pilot Project in Cambodia failed to comply fully with the World Bank’s Indigenous Peoples, Natural Habitats, and Environmental Assessment safeguards. The panel also found that in the Land Management and Administration Project, the World Bank did not adequately follow up on strengthening public awareness and community participation and was slow to respond to evictions. An Integrity Vice Presidency investigation found instances of fraud, collusion, and bid manipulation. An action plan resulted in remedial measures, including the establishment of an independent procurement agent. Despite the wide range of issues encountered during the extended period (FY05–15) and the suspension of IDA financing after 2011 (after investigation by the Integrity Vice Presidency), the World Bank remained engaged with the government using nonlending services. These included analytical and advisory work (health, capacity building on poverty analysis, technical assistance on social protection, water and sanitation review, and investment climate analysis), and support through trust-funded activities. IFC investments helped strengthen institutions and promote financing to rural and micro businesses, especially in the agricultural sector, and improve infrastructure and manufacturing sectors. On advisory activities, IFC helped with the adoption of regulations for establishing a credit bureau and mobile banking. 131 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Figure 1. East Asia and Pacific Region: Percentage of Projects Rated MS+ on Outcome FY01– 15 100 Bank-wide 95 East Asia & Pacific Non-East Asia & Pacific Regions 90 84 85 85 85 79 81 80 80 Percentage 76 75 74 72 73 75 78 79 70 70 75 76 70 74 73 74 71 71 72 65 70 67 60 65 55 50 Fiscal year, three-year rolling Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 132 Table 1. East Asia and Pacific Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13–15) East Asia and Pacific Region Other Regions Bank-wide Number of Net commitments Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) projects (US$, millions) Global Practice Total Rated Total Rated Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Agriculture 9 78 615 82 64 80 3,468 91 73 79 4,083 89 Education 16 88 1,889 97 64 70 4,687 80 80 74 6,576 85 Energy and Extractives 15 73 1,922 72 58 64 5,342 81 73 66 7,264 79 Environment and Natural Resources 10 70 593 87 40 60 1,017 78 50 62 1,610 81 Finance and Markets 3 67 154 71 31 74 4,847 97 34 74 5,001 96 Governance 11 45 1,008 27 44 59 3,699 86 55 56 4,706 74 Health, Nutrition, and Population 8 63 259 88 59 75 5,086 83 67 73 5,345 83 Macroeconomics and Fiscal Management 7 86 1,434 99 42 81 9,416 97 49 82 10,850 97 Poverty 0 n.a. 0 n.a. 7 71 1,175 99 7 71 1,175 99 Social Protection and Labor 1 100 48 100 39 90 6,371 98 40 90 6,419 98 Social, Urban, Rural, and Resilience 28 79 3,961 93 68 68 4,622 71 96 71 8,582 81 Trade and Competitiveness 3 33 407 2 14 71 608 89 17 65 1,015 54 Transport and ICT 20 80 1,998 83 58 79 7,641 90 78 79 9,639 89 Water 12 67 1,117 75 60 68 4,952 73 72 68 6,069 74 Total 143 73 15,403 81 648 72 62,930 87 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. 133 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 2. East Asia and Pacific Region: IEG Outcome Ratings of IFC Investment Projects, FY13– 15 East Asia and Pacific IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 5 20 156 51 34 38 904 56 and Funds Financial Markets 9 89 177 92 70 59 1,705 62 Infrastructure and Natural 3 67 47 68 41 63 1,518 79 Resources Manufacturing, Agribusiness, 11 36 256 48 84 51 2,105 59 and Services Total 28 54 636 63 229 54 6,232 64 Source: IEG data. Note: MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. East Asia and Pacific Region: IEG Outcome Ratings of IFC Advisory Projects, FY13–15 East Asia and Pacific IFC overall IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 10 70 43 70 Cross-Cutting Advisory Solutions 1 100 4 75 Cross-Cutting Advisory Solutions: Public-Private 1 100 14 43 Partnerships Environment, Social, and Governance 1 100 1 100 Financial Markets 2 0 5 40 Financial Institutions Group 0 n.a. 17 47 Infrastructure and National Resources 0 n.a. 2 100 Investment Climate 2 50 26 77 Manufacturing, Agribusiness, and Services 1 100 3 33 Public-Private Partnerships 3 33 17 47 Sustainable Business Advisory 3 100 38 66 Telecom, Media, Technology, and Venture Capital 0 n.a. 1 0 Trade and Competitiveness 3 33 8 50 Total 27 63 179 61 Source: IEG data. Note: MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Please note that during FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public-Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. 134 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 4. East Asia and Pacific Region: MIGA Development Outcome Ratings by Sector, FY13 –15 East Asia and Pacific MIGA overall Sector Total Total Rated S+ (%) Rated S+ (%) number number Agribusiness 0 n.a. 1 0 Financial 2 0 8 38 Infrastructure 1 0 9 44 Manufacturing 0 n.a. 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 0 n.a. 2 0 Services 1 0 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 4 0 34 47 Source: IEG data. Note: S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Table 5. East Asia and Pacific Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank IFC IEG XPSR IFC IEG PCR projects ratings ratings Total Total Rated Rated MS+ Total Rated Country numb number MS+ (%) (%) number MS+ (%) er China 26 88 9 22 4 75 Indonesia 32 63 4 75 4 25 IBRD Philippines 15 47 3 100 2 100 Thailand 3 67 1 100 IBRD total 76 68 17 53 10 60 Mongolia 10 90 2 100 Papua New Guinea 2 50 2 50 Blend Timor-Leste 4 25 Vietnam 25 80 1 0 8 88 Blend total 41 76 Cambodia 8 88 1 0 2 50 Kiribati 1 0 Lao Peoples Democratic 10 70 2 50 2 0 Republic IDA Myanmar 1 100 Samoa 1 100 1 100 Solomon Islands 4 100 Tonga 2 100 Vanuatu 1 0 IDA total 26 85 3 33 7 29 Other East Asia and Pacific Region 1 0 Other total 1 0 TOTAL 143 73 26 50 15 25 Source: IEG data. 135 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Note: The data excludes projects not mapped to a Region. MS+ = moderately satisfactory or better (World Bank rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number’ refers to the total number of projects rated by IEG. Table 6. East Asia and Pacific Region: IEG Ratings of Project M&E Quality, FY13–15 East Asia and Pacific Other Regions Bank-wide Rated Rated Rated Total Total Total substantial substantial substantial number number number (%) (%) (%) IBRD 76 30 260 34 336 33 Blend 40 33 67 22 107 26 IDA 26 42 283 31 309 32 Other 0 n.a. 35 43 35 43 Total 142 33 645 32 787 32 Source: IEG data. Note: The data excludes projects not mapped to a Region. M&E = monitoring and evaluation; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Table 7. East Asia and Pacific Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 CASCR/CLR World Bank IFC MIGA Outcome FY of review Country review performance performance performance rating period rating rating rating 2014 Philippines FY10–13 MS Good Good Reviewed & Reviewed & Cambodia FY05–15 not rated not rated 2016 Indonesia FY13–15 MS Good Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory.Table 8. East Asia and Pacific Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16. 136 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 2012 2013 2014 2015 2016 % of % of % of % of % of Global Practice $ million region $ million region $ million region $ million region $ million region total total total total total Agriculture 17 0 80 1 360 6 1,234 19 25 0 Education 50 1 345 6 438 7 127 2 95 1 Energy & Extractives 490 7 676 11 810 13 537 8 1,500 20 Environment & Natural Resources 280 4 219 4 134 2 119 2 1,050 14 Finance & Markets 0 0 130 2 20 0 500 8 0 0 Governance 150 2 300 5 430 7 0 0 0 0 Health, Nutrition & Population 100 2 150 2 126 2 126 2 30 0 Macro Economics & Fiscal Management 2,411 36 1,012 16 835 13 322 5 1,127 15 Poverty and Equity 0 0 0 0 0 0 0 0 0 0 Social Protection & Labor 0 0 180 3 60 1 0 0 450 6 Social, Urban, Rural, and Resilience 1,873 28 1,909 31 1,054 17 1,303 21 932 12 Trade & Competitiveness 0 0 166 3 300 5 0 0 0 0 Transport & ICT 776 12 531 9 1,541 24 974 15 1,093 15 Water 481 7 550 9 205 3 1,100 17 1,199 16 % of % of % of % of % of total total total total total $ million $ million $ million $ million $ million IBRD IBRD IBRD IBRD IBRD and IDA and IDA and IDA and IDA and IDA East Asia and Pacific Region 6,628 19 6,247 20 6,313 16 6,342 15 7,500 16 Other Regions 28,629 81 25,301 80 34,369 84 36,152 85 38,399 84 Grand Total 35,256 31,547 40,681 42,495 45,899 Source: World Bank Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. 137 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Europe and Central Asia Region Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 Europe and Central Asia Regional Update, poverty reduction gains are reversing in many countries in the Region. More than half of the Region’s countries experienced an increase in poverty between 2005-15, with shared prosperity stagnating and reversing in several countries. The Region is operating at 60 percent of precrisis growth levels, resulting in high unemployment rates and particularly high concentrations of female and youth unemployment. Remittances, an important source of income for many of the poorest, fell substantially with the decline in host country growth, led by a decline in transfers from the Russian Federation that is shedding some of its migrant workforce—7 percent of Central Asia’s working-age population work in Russia (World Bank 2016g). The Region faces several long-standing and ongoing challenges, such as political uncertainty, and challenges associated with the refugee crisis and with climate vulnerability (climate change compounding legacy issues such as inefficient infrastructure, and unsustainable land and water management). Changes in commodity prices are also a challenge, one that adversely affects the eastern part of the Region, exerting pressure on key oil exporters (Azerbaijan, Kazakhstan, and Russia) where government policy buffers are eroding (World Bank 2016b). The Region also lags on competitiveness indicators of technological readiness and capacity to innovate. 2. The World Bank Group believes it can respond to the challenges and to client demand with tailored approaches that use its comparative advantage—the ability to take on various partner roles in country, private, and global development, offering financial, global knowledge, and convening services (World Bank 2016g). 3. The World Bank’s new lending commitments in the Region trended slightly upward in recent years: $5.3 billion in FY13, $5.5 billion in FY14, $7.2 billion in FY15, and $7.3 billion in FY16. FY13 commitments in the Region represented 17 percent of all World Bank commitments, and FY16 commitments, though a larger amount, represent 16 percent of total. More than three-quarters of all FY16 commitments to the Region are concentrated in three Global Practices: Macroeconomics and Fiscal Management ($2.55 billion, 35 percent); Transport and ICT ($2.07 billion, 28 percent); and Finance and Markets ($1.03 billion, 14 percent) (table 8). 4. The overall performance of World Bank operations in Europe and Central Asia is better than the World Bank performance as a whole: outcome ratings at exit 138 APPENDIX I REGIONAL RESULTS AND PERFORMANCE FY13–15 were 79 percent moderately satisfactory or better (MS+) compared with a World Bank average of 72 percent (figure 1). 5. Performance by Global Practice in the Region portfolio is better than the World Bank average in all cases except Social Protection and Labor, which has a marginal performance difference, and in Agriculture and Education, in which the Region performs less well than the World Bank average. Note that Agriculture had only nine rated projects and Education had seven in Europe and Central Asia during FY13–15 (table 1). 6. Average performance across rated projects in IDA and IBRD countries in FY13–15 (82 and 80 percent MS+, respectively) was stronger than the average of 62 percent MS+ for the smaller number (two) of blend countries in the Europe and Central Asia Region. Performance in IBRD countries was particularly strong in many instances—nine countries, hosting a total of 29 projects, registered a 100 percent MS+ rating (table 5). Noting the small number of projects in each instance, a performance rating of 67 percent MS+ in each of Croatia, Albania, Poland, and Azerbaijan was well below the IBRD regional average, as was the rating for Turkey (57 percent MS+). 7. Across the portfolio, common weaknesses in less successful projects include overambitious objectives relative to project components and time frame, complex project design involving multiple components and implementing agencies, and overestimating client buy-in and system capacity. However, a robust results framework defined through a clear, logical sequence between project activities, output, outcomes, and development objectives is a common feature among successful projects. The Results and Performance of the World Bank Group series and other IEG evaluations have found that successful projects reflect the project team’s willingness to learn from past projects, and they must always tailor learning to the local context to add value. 8. IEG produced two project performance assessment reports (PPARs) in FY16 for projects in the Europe and Central Asia Region. The PPAR for the Private and Financial Sector Policy-Based Guarantee in Serbia (approved in 2011 and due to expire 2017) was rated satisfactory for development outcome. The project sought to support access to commercial loans on better terms and longer duration; improve the business environment; support an enhanced, more stable and efficient financial sector; and support financial discipline, focusing particularly on bankruptcy. The PPAR found that the World Bank team worked hard to maintain policy dialogue and to act as facilitator in discussion between government and prospective external financiers. Objectives closely aligned with country strategy and associated plans to accelerate progress toward a market-driven economy. The government managed to 139 APPENDIX I REGIONAL RESULTS AND PERFORMANCE double maturity and substantially lower the cost of borrowing by using the guarantee, which also supported financial sector reforms, despite adverse conditions. 9. A PPAR in Uzbekistan covered two projects (the Uzbekistan Water Supply, Sanitation, and Health Project, and the Bukhara and Samarkand Water Supply Project), both of which IEG rated moderately satisfactory (MS). Both projects were extended—the water, sanitation, and health project operated during 1997–2008, and the water supply project during 2002–10. The primary focus of the water, sanitation, and health project was rehabilitating water supplies in three major cities and rural areas, and the second focus was providing demand-driven rural sanitation and a program of hygiene education. The relevance of project objectives was high, focused on service provision and capacity building toward sustainable infrastructure. IEG rated the project design as substantial, supported by a robust results framework and a pilot phase. Weaknesses associated with underestimating the community self-help capacity needed to make the community-driven development approach effective resulted in dropping the hygiene and rural sanitation components. The water supply project sought to rehabilitate and improve the efficiency of existing water supply facilities and infrastructure (which attracted most of the spending) and to strengthen institutional and financial capacity. IEG assigned positive ratings for both relevance and project design. However, unrealistic assumptions were made about the availability of baseline data and stakeholder buy-in. The World Bank’s performance rating (unsatisfactory) suffered as a result, and high supervisory staff turnover affected the rating. 10. IEG rated 38 percent of projects in the Europe and Central Asia Region during FY13–15 substantial or better on monitoring and evaluation (M&E), which is above the World Bank average (32 percent) and indicates relative success, though overall levels of satisfaction with M&E design, implementation, and utilization are still low (table 6). However, M&E ratings for projects in two IDA countries (the Kyrgyz Republic and Tajikistan) are promising. As earlier regional updates noted, a series of perennial issues are evident relating to M&E, including poor results chains, overreliance on output indicators and a lack of outcome indicators, using indicators not directly relevant to project objectives or not directly attributable to project activities, and a lack of baseline data that precludes measurement of progress. 11. In the case of IFC, IEG assigned ratings of mostly successful or higher to 23 (44 percent) of the 52 IFC investment projects for FY13–15, which is less than the IFC average (54 percent). The Region’s success rate by investment commitments was 57 percent, which is less than the 64 percent IFC average success rate (table 2). 140 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 12. The regional strategy for Europe and Central Asia notes that countries need to embrace new opportunities because past growth patterns are no longer possible. New growth opportunities may come from provision of financial services to underserved sectors, primarily small and medium enterprises (SMEs). A lesson from an IEG PPAR that assessed an SME loan to a bank in Turkey is particularly relevant in that regard. Cost considerations may lead such clients to favor shorter maturity or foreign currency loans and expose the client (a bank in this case) and its SME borrowers to greater risk if a devaluation should occur. A similar lesson from an IEG Evaluative Note on an IFC investment suggests running stress tests at the appraisal stage to analyze and understand results under pessimistic scenarios, then try to mitigate risks. Another PPAR emphasizes the importance of sensitivity analysis (a form of stress test) when working with the nonfinancial sector and recommends such analyses for all possible scenarios, specifically including scenarios for price shocks, oversupply, fuel supply price, load factors, and foreign exchange risks. 13. In the case of IFC Advisory Services projects, IEG assigned ratings of mostly successful or higher to 18 (72 percent) of the 25 projects validated FY13–15, which is higher than the 61 percent IFC average (table 3). IFC Advisory Services projects often engage the public sector (on investment climate reform, for example). An IEG review of such an initiative in Tajikistan identified the drivers for consideration to ensure implementation of reforms. These include clearly identifying the entities responsible for implementation; providing adequate information and training for officials at the local level; providing the necessary tools and resources to implementing agencies at the local level; and establishing an incentive structure. 14. IEG validated 15 MIGA projects in the Europe and Central Asia Region during FY13–15 and rated eight projects (53 percent) satisfactory or higher, which was better than the 47 percent average success rate (table 4). To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Findings from Thematic, Global, and Corporate Evaluations, and Learning Notes 15. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence focused on situations of fragility, conflict, and violence, which are predominantly, though not exclusively, found in middle-income countries. The evaluation, which references the Kyrgyz Republic, found that when countries experience situations of fragility, conflict, and violence, the World Bank Group and the governments may have differing views on how and when to tackle the 141 APPENDIX I REGIONAL RESULTS AND PERFORMANCE underlying issues, which makes providing effective support particularly challenging (IEG 2016q). 16. In the Kyrgyz Republic, the World Bank co-led (with the Asian Development Bank and the IMF) a multidonor assessment, The Kyrgyz Republic Joint Economic Assessment: Reconciliation, Recovery, and Reconstruction, that identified interpersonal ethnic animosities and national government policies as conflict drivers. The World Bank adopted a conflict filter as a screening tool to make sure that any World Bank activity distributed benefits among ethnic and social groups in a way that was seen to be fair and that would not exacerbate conflict situations. The goal was to provide information and guidance to improve project design to mitigate the risk of violence, and strengthen social cohesion. The objectives of the conflict filter were to make task teams more cognizant of their operating context and help them identify and manage conflict and fragility risks. The conflict filter sought to identify and then avoid doing activities (lending and nonlending) in ways that might do harm by stirring up ethnic animosities. According to the evaluation, the Kyrgyz Republic case shows that the World Bank can gather information (at a reasonable cost) about the fault lines of potential conflict and design activities to bridge those fault lines instead of exacerbating them. This fairly simple and inexpensive tool seems to be most useful in the aftermath of relatively short and recent conflicts. The country teams found its application in other contexts to be less useful, such as in Sri Lanka, where the conflict was much more protracted and stayed unresolved for a long time (ADB, IMF, and the World Bank 2010). 17. Managing Environmental and Social Risks in Development Policy Financing assesses the application of World Bank Operational Policy (OP 8.60) elements governing development policy financing related to implementation of the policy’s environmental and social risk management requirements to identify lessons and good practices (IEG 2015i). Projects in Tajikistan and Georgia were part of the review sample. The review found that in some instances environmental risks were included in the provision of support for public-private partnerships (PPPs), but other operations supporting similar reforms (including a project in Tajikistan) identified no such risk. The review finds the Georgia Poverty Reduction Support Operation series does well in dealing with social risk at several levels, especially regarding the role of the poverty and social impact assessment (PSIA). In this case, the prior action was the satisfactory implementation of an energy sector medium- term strategic action plan. The program document succinctly describes the PSIA conducted before the series of four operations that addressed the distributional impact of electricity tariff reform. The PSIA determined that the reform would affect poor people outside of the capital. A mitigation measure was proposed and implemented to establish a block tariff—a separate lower rate for low levels of 142 APPENDIX I REGIONAL RESULTS AND PERFORMANCE electricity consumption. The section on across-regional findings summarizes the findings from this evaluation. 18. IEG’s learning product, World Bank Group Support for Housing Finance, found that IFC’s €12 million credit line to a commercial bank in Bosnia and Herzegovina financed more than 1,300 housing loans (€21.3 million). The report also found IFC staff was also asked to bring expertise to a bank to improve overall mortgage laws and regulations in the country. IFC’s long-term loan to a bank in Romania enabled significant growth in the client’s mortgage and home improvement loan portfolio during 2004–08 and contributed to greater competition that led to increased mortgage tenor—from 13 years at the end of 2005 to 23 years at the end of 2008— and a decrease in mortgage rates. Referencing IFC’s provision of a long-term credit line to a first (greenfield) specialized housing finance company in Romania, the review also found that pursuing ambitious development objectives through a single long-term loan can lead to unsatisfactory results. The project’s aim was to develop and expand the mortgage market and ultimately create a secondary mortgage market in a country with low mortgage penetration (0.3 percent of gross domestic product in 2001). The supported entity became the country’s largest specialist mortgage lender, achieving a market share of 1.7 percent by 2007 with good portfolio quality; however, its presence in the market was small, and operations were unprofitable. The company did not securitize its portfolio and did not become the important secondary market institution envisioned in the project (IEG 2016s). 19. IEG’s review of housing finance also finds that capital market interventions require extensive knowledge of relevant country constraints. World Bank upstream support can be crucial to support the development of the housing finance systems and capital markets in cases that lack preconditions for success. An example is the enactment of the new law on residential real estate in Turkey, which included sections enabling both covered bonds and domestic securitization facilitating Turkey’s capital markets development. The review also found that extending foreign exchange loans to unhedged borrowers earning in local currency puts affordability at risk for vulnerable segments of the population. An IFC-supported project in Ukraine intended to provide more affordable housing finance through foreign exchange loans and to stimulate sustainable and healthy market competition. Providing a loan in U.S. dollars to borrowers earning domestic currency was highly risky in general, and the low-income group was the least likely to have protection from a devaluation because they did not have access to foreign exchange resources. In Russia, dollar loans to borrowers predominately earning in local currency funded by IFC U.S. dollar loans put borrowers at risk when the local currency (ruble) devalued in 2008 and 2009 (IEG 2016s). 143 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 20. Lessons from Land Administration Projects: A Review of Project Performance Assessments synthesizes cross-cutting findings and lessons from IEG’s assessments of World Bank–supported projects dealing with often complex and politically fraught land administration issues. IEG compared World Bank Group experience across projects using different institutional structures to manage two important elements of a land administration system—registry, which records the rights to land, and the cadaster, which provides information on the location, boundaries, use, and values of land parcels. The comparison suggests that no single model is best for performing these functions. Projects in Georgia, the Kyrgyz Republic, Romania, and Tajikistan used a single-agency model effectively. However, Bulgaria and Slovenia use a system in which different agencies manage the cadaster and registry, which requires coordination and data integration. In Bulgaria, a project implementation unit was established that bridged cadaster and registration services across two separate agencies and introduced a unified information technology system that allowed rapid data sharing. In Slovenia, establishment of a project coordinating unit, ongoing political support through the higher-level Program Council, and the use of digitized information technology reduced coordination inefficiencies across separate agencies. Digitized information technology helped ensure seamless coordination between the registry and cadaster (IEG 2016j). 21. The review also emphasizes that making land tenure more secure is a process, not a single event, and the concept of land tenure and associated rights is highly context-specific, meaning there are no absolute, generally applicable standards for defining tenure security. Projects in Azerbaijan, Bulgaria, Georgia, the Kyrgyz Republic, Romania, and Slovenia improved the implementation capacity, legal framework, and effectiveness of land administration systems, providing a foundation for better-functioning land markets and other outcomes. In some instances, deficiencies or inadequate land administration systems affected tenure security. Responsibilities were fragmented across different agencies, records were inaccurate, and the boundaries of parcels were unclear. When several of these projects ended, functioning land institutions were in place with reference to effectiveness of land administration (for example, less time to register and fewer steps in the process). Supportive legal frameworks backed the institutional and technical measures. Strong political commitment facilitated the projects’ progress. In some countries (Bulgaria, Romania, and Slovenia), the prospect of accession to the European Union enhanced the political commitment (IEG 2016j). 22. Most of the projects reviewed did not explicitly target the poor or vulnerable groups, reflect social inclusion in their objectives, or sufficiently report on social impacts across projects. Experience shows that social impacts need to be monitored and not assumed, even when laws and procedures are nominally the same for all 144 APPENDIX I REGIONAL RESULTS AND PERFORMANCE potential beneficiaries. Many of the projects in the Europe and Central Asia Region did not specifically address the needs of the poor on the basis that land legislation and procedures have general application regardless of social status and, as such, relevant projects tended not to monitor social impacts. The project in Bulgaria tried to address the needs of the vulnerable by providing free legal assistance to registry and cadaster clients who needed it, but IEG’s assessment found these actions were limited, and evidence of their social impact was inconclusive. Although project M&E did not assess social inclusiveness, some of the beneficiary surveys conducted by the land registry in the Kyrgyz Republic showed that some groups (women among them) had more difficulty accessing land administration services than others. A cross-cutting lesson based on experience in the Europe and Central Asia Region highlights the need for better information to improve knowledge of how land administration programs affect different groups, and how the welfare of vulnerable groups might be protected and improved (IEG 2016j). Evaluation Findings from Country Program Performance in the Region 2. Analysis undertaken for Results and Performance of the World Bank Group 2016 shows that for the 15 countries in the Europe and Central Asia Region for which IEG validated two country strategies for FY07–16, the rating for overall development outcome remained MS+ for seven countries. The analysis also shows that the rating improved from moderately unsatisfactory (MU) to MS+ for five countries, and declined from MS to MU for three countries. For the five countries for which IEG rated their country strategies three times during the FY07–16 period, the ratings for two remained MS. One country’s rating improved from MU to MS and remained so after the third validation. In two countries, ratings first declined from MS to MU before regaining an MS rating after the third validation.1 23. Overall outcomes of World Bank Group country programs in Europe and Central Asia remain largely positive for FY16. Country outcomes were rated MS or higher for all seven IEG completion and learning reviews (CLRs) conducted in FY14, four reviews completed in FY15, and three of four CLR reviews (Uzbekistan, Bosnia and Herzegovina, and Montenegro) completed in FY16 – the fourth CLR review undertaken in FY16 rated the program for Bulgaria unsatisfactory (table 7). 1The 15 countries included in this analysis are: Albania, Armenia, Azerbaijan, Belarus, Bosnia and Herzegovina, Bulgaria, Croatia, Georgia, the Kyrgyz Republic, the former Yugoslav Republic of Macedonia, Moldova, Montenegro, Poland, Romania, the Russian Federation, Serbia, Tajikistan, Turkey, Ukraine, and Uzbekistan. 145 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 24. A country program evaluation (CPE) of Kazakhstan (FY04-13) conducted as part of a clustered CPE on resource-rich countries found that impressive economic growth and rising hydrocarbon prices helped the country make steady progress on poverty reduction and social development during the review period. However, some systemic issues still need to be addressed (lack of progress on economic diversification and anticorruption, the state’s dominant role in the economy, low skill levels in the labor force, and a legacy of environmental issues). The World Bank Group performed exceptionally well in its engagement with government, providing prompt and trusted high-quality technical and policy advice, though this engagement’s demand-driven nature provides little opportunity for the World Bank to push the boundaries in defining strategic priorities. Looking forward, the World Bank Group will need to advance transparency and accountability through engagement with a wider range of stakeholders, bringing them deeper into the conversation about ongoing progress. In line with its global development mandate, and to counteract possible limitations on defining priorities related to the current emphasis on Reimbursable Advisory Services, the World Bank should consider (re-) introducing standard, nonreimbursable work on country diagnostics (such as governance, anticorruption, Public Expenditure Reviews, and poverty assessments). 25. The four FY16 CLRs’ key lessons highlight the importance of the following:  Generating buy-in and maintaining ongoing dialogue with government and other stakeholders  The need to improve World Bank–IFC cooperation and coordination  The value of building flexibility into program design, particularly in middle- income countries  The need to improve the design of results frameworks regarding the quality and number of indicators, and the need to enhance M&E. 26. The CLR for the FY12–15 country partnership strategy (CPS) for Bosnia and Herzegovina notes that the strategy was based on three focus areas designed to address some of the country’s main challenges, concentrating on areas of the World Bank Group’s comparative advantage. The first focus area sought to support competitiveness and economic growth through tackling key bottlenecks. The second aimed to promote inclusion through improvements in public services for the vulnerable and rationalization of social services. The third focus aimed to promote environmental sustainability through better use of water and forest resources, climate change adaptation, and sustainable development of municipal water and sanitation services. 146 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 27. IEG’s review reinforces lessons set out in the CLR itself, including the need for the following:  Continue to build flexibility into CPS programming while clear focusing on long-term, transformative reform objectives  Take the care and time needed to build a broad base of support among internal and external stakeholders, especially for cross-border projects or those involving more than one entity government  Finalize institutional arrangements for implementation during preparation to reduce loss of momentum beyond the largely unavoidable effectiveness delays arising from political tensions in state and entity legislatures. 28. IEG also recommends continuing to build on the strong partnerships established with a number of EU institutions to both leverage limited World Bank resources and take advantage of the influence of these institutions on policy continuity and governance more broadly. The review endorses the practice adopted in Bosnia and Herzegovina of setting explicit, quantitative goals for IFC investment activity as part of a clear and transparent effort to achieve the CPS objectives and the World Bank Group corporate goals. Furthermore, the review also endorses the country team practice of preparing joint business plans that specify the areas, forms of engagement, and milestones for joint World Bank Group cooperation. 29. IEG’s CLR review for the Montenegro CPS for FY11–15 notes the close alignment of the country strategy’s two focus areas (support for EU accession, and environmental management) with the government program’s core priorities. The review finds that the CPS was not a joint World Bank–IFC document and that potential synergies between IBRD and IFC were underused as a result. The review finds that two main IFC investments during the CPS period were unrelated to the CPS objectives. It also finds that the original CPS results framework showed significant weaknesses affecting program implementation, and that the World Bank’s flexibility in program implementation perhaps weakened the relevance of the program—for example, dropping a public expenditure management objective at the time of the CPS progress report. The review notes that the approval of a policy- based guarantee without ensuring an adequate fiscal policy framework may have increased the World Bank portfolio’s exposure risk. During program implementation, however, the World Bank managed to improve its portfolio quality while making significant adjustments to the original program to reflect changes in the government’s priorities, ensuring the quality of the current portfolio remains broadly satisfactory. The CPS progress report significantly modified the results framework, which generally increased the evaluability of the program, although some original weaknesses remained unaddressed. The review finds the program 147 APPENDIX I REGIONAL RESULTS AND PERFORMANCE helped to strengthen certain institutions, but the most significant progress against the CPS objectives was in sustainable land management, reforming the government system to deliver assistance to farmers, strengthening local research capacity, improving the reliability of the power supply, and strengthening solid waste management. IEG largely agrees with the CLR’s lessons, particularly regarding the importance of selectivity for a small portfolio, the World Bank’s complementary role in supporting the EU accession process, and the detrimental effects of shortcomings in the results framework. Furthermore, IEG notes that the World Bank Group effectively used regional projects to generate both economies of scale and additional policy insights, and that there is room for increasing IBRD and IFC integration in the forthcoming Community Support Framework. IEG states that a clear justification is needed when dropping a crucial policy objective from the program (referring to the dropped public expenditure management objective) and emphasizes the importance of calibrating program design to factor in government ownership to achieve a robust program that is less vulnerable to sudden changes in government sentiment. 30. The CLR for Bulgaria (rated unsatisfactory) assesses the World Bank Group’s innovative approach to support Bulgaria in its efforts to further integrate into the EU, to which it accessed in 2007. Given the significant funding available to Bulgaria through EU funds, the CPS pioneered a new model of knowledge-based country engagement building on the Reimbursable Advisory Services program. The idea was to engage in selective, fee-based policy support enable the government to leverage the World Bank’s knowledge and experience better to accelerate investment projects and more effectively absorb available EU funding. The program was organized into three focus areas: knowledge and advisory services for policy and program development in innovation, education, public finance, competitiveness (business regulatory environment), green growth, and social inclusion; advisory services and ongoing operations to support the preparation of large projects to secure EU support; and financing for priority, results-based projects ineligible for EU funding. The CLR finds the strategy to be relevant, innovative, and selective in its approach, but overoptimistic in its assessment of the durability of key conditions (including political continuity and stability and country ownership) that would be required to support the efficacy of the main instrument (the Reimbursable Advisory Services program). In that regard, a smaller, less ambitious program might have fared better. Generally, IFC was expected to focus on business regulation advisory services, agribusiness (including land consolidation), energy, transport, and banking. However, these activities appeared to be add-ons to the World Bank program, and the results framework did not reflect the expected outcomes of IFC activities. IEG generally concurred with the lessons in the CLR. However, IEG added that greater attention to the potential role of closer cooperation between the 148 APPENDIX I REGIONAL RESULTS AND PERFORMANCE World Bank and IFC should be pre-planned—especially in areas of mutual expertise and experience, such as energy, banking, and business climate—to both improve the quality of World Bank Group assistance and to have greater impact in the field. 31. The FY16 CLR reviews highlight lessons regarding the substantial overall program of analytic and advisory activities in Europe and Central Asia. For example, the CLR for Bulgaria endorses using analytic and advisory activities to keep current with developments in strategic sectors in which the World Bank is not lending. The CPS for Uzbekistan planned significant analytic and advisory support for the competitiveness and diversification thematic areas, and the program followed up in implementation (60 percent of all analytic and advisory activities in Uzbekistan). Therefore, support to develop the Uzbekistan Vision 2030 program (business climate reforms and improvement in the financial infrastructure) was a major activity. In Bosnia and Herzegovina, a series of core diagnostic reports supported the CPS, and further work during the CPS period was undertaken to prepare for the next stage of strategic planning. Furthermore, when a development policy loan series was dropped, the World Bank maintained open dialogue with the client through its analytic and advisory activities program (including support for debt management, and analytical work on social spending and smart safety nets). 32. The CLRs report on a range of IFC Advisory Services support, including work that is regional in scope. Advisory support focused on business climate and the energy sector in Montenegro. IFC faced challenges in the provision of advisory support for PPPs in Bulgaria (water sector) and Uzbekistan (health sector). In Bosnia and Herzegovina, IFC participation in the CPS process and World Bank-IFC cooperation were particularly strong, and the CLR review noted important lessons from this experience. Setting explicit, quantitative goals for IFC investment activity should be part of a transparent effort to achieve the CPS objectives and the World Bank Group corporate goals. Furthermore, preparing joint business plans specifying the areas, forms of engagement, and milestones for cooperation between the World Bank Group organizations was a useful tool for joint CPS implementation. 33. The Montenegro CLR noted that IFC program relevance to the CPS objectives was low. IFC advisory services in the investment climate and energy sector and a relatively small credit line in the financial sector contributed to attaining the CPS objectives. However, IFC made major investments during the review period in road infrastructure, which were not included in the CPS objectives. IFC’s other achievements as presented in the CLR are mostly unrelated to specific CPS objectives. Potential synergies between IBRD and IFC remained underused in the original program design and its subsequent modification. The CLR review for Uzbekistan noted that IFC interventions were designed to complement World Bank 149 APPENDIX I REGIONAL RESULTS AND PERFORMANCE activities. IFC lending through a commercial bank improved SME access to finance, particularly in the agriculture sector, and IFC advisory work supported improvements in the financial infrastructure. Figure 1. Europe and Central Asia Region: Percentage of Projects Rated MS+ on Outcome FY01–15 100 Bank-wide 95 Europe & Central Asia Non-Europe & Central Asia Regions 90 86 85 83 84 85 80 79 81 78 79 78 79 80 Percentage 74 75 75 78 77 76 70 73 73 75 72 71 65 70 69 69 67 68 60 55 50 Fiscal year, three-year rolling Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 150 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 1. Europe and Central Asia Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13–15) Europe and Central Asia Other Regions Bank-wide Region Number of Net commitments Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) projects (US$, millions) Global Practice Rated Rated Rated Rated Rated Rated Total Total Total Total Total Total MS+ MS+ MS+ MS+ MS+ MS+ number amount number amount number amount (%) (%) (%) (%) (%) (%) Agriculture 9 78 227 75 64 80 3,855 90 73 79 4,083 89 Education 7 43 105 28 73 77 6,470 86 80 74 6,576 85 Energy and Extractives 13 85 1,052 89 60 62 6,212 77 73 66 7,264 79 Environment and Natural Resources 6 67 113 86 44 61 1,496 81 50 62 1,610 81 Finance and Markets 9 100 1,990 100 25 64 3,011 93 34 74 5,001 96 Governance 9 67 241 86 46 54 4,465 73 55 56 4,706 74 Health, Nutrition, and Population 9 78 390 67 58 72 4,955 84 67 73 5,345 83 Macroeconomics and Fiscal Management 12 83 5,603 99 37 81 5,248 95 49 82 10,850 97 Poverty 1 100 42 100 6 67 1,133 99 7 71 1,175 99 Social Protection and Labor 7 86 300 68 33 91 6,119 99 40 90 6,419 98 Social, Urban, Rural, and Resilience 11 73 388 89 85 71 8,194 81 96 71 8,582 81 Trade and Competitiveness 2 100 90 100 15 60 925 49 17 65 1,015 54 Transport and ICT 11 82 1,830 83 67 79 7,809 90 78 79 9,639 89 Water 13 85 628 87 59 64 5,441 72 72 68 6,069 74 Total 119 79 12,999 92 672 71 65,334 85 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; total number refers to the total number of projects rated by IEG. 151 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 2. Europe and Central Asia Region: IEG Outcome Ratings of IFC Investment Projects, FY13–15 Europe and Central Asia IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 3 0 67 0 34 38 904 56 and Funds Financial Markets 20 50 761 55 70 59 1,705 62 Infrastructure and Natural 10 70 545 86 41 63 1,518 79 Resources Manufacturing, Agribusiness, 19 32 528 37 84 51 2,105 59 and Services Total 52 44 1,901 57 229 54 6,232 64 Source: IEG data. Note: MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. Europe and Central Asia Region: IEG Outcome Ratings of IFC Advisory Projects, FY13– 15 Europe and Central IFC overall Asia IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 8 88 43 70 Cross-Cutting Advisory Solutions 0 n.a. 4 75 Cross-Cutting Advisory Solutions: Public-Private 2 0 14 43 Partnerships Environment, Social, and Governance 0 n.a. 1 100 Financial Markets 0 n.a. 5 40 Financial Institutions Group 2 0 17 47 Infrastructure and National Resources 0 n.a. 2 100 Investment Climate 3 100 26 77 Manufacturing, Agribusiness, and Services 0 n.a. 3 33 Public-Private Partnerships 4 50 17 47 Sustainable Business Advisory 5 100 38 66 Telecom, Media, Technology, and Venture Capital 0 n.a. 1 0 Trade and Competitiveness 1 100 8 50 Total 25 72 179 61 Source: IEG data. Note: MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Please note that during FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. 152 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 4. Europe and Central Asia Region: MIGA Development Outcome Ratings by Sector, FY13–15 Europe and Central MIGA overall Asia Sector Total Total Rated S+ (%) Rated S+ (%) number number Agribusiness 0 n.a. 1 0 Financial 4 75 8 38 Infrastructure 3 33 9 44 Manufacturing 3 100 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 1 0 2 0 Services 4 25 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 15 53 34 47 Source: IEG data. Note: S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. 153 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 5. Europe and Central Asia Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank IFC IEG XPSR IFC IEG PCR projects ratings ratings Total Rated Total Rated Total Rated Country number MS+ (%) number MS+ (%) number MS+ (%) Albania 6 67 1 100 1 100 Armenia 9 100 1 0 3 100 Azerbaijan 3 67 2 0 2 100 Belarus 1 100 2 50 Bosnia and Herzegovina 2 100 1 0 1 100 Bulgaria 2 100 4 25 1 0 Croatia 6 67 1 0 Georgia 2 100 2 50 4 50 IBRD Kazakhstan 3 100 3 33 Macedonia, FYR 5 100 1 0 1 0 Montenegro 3 100 1 0 Poland 3 67 Romania 8 75 2 50 Russian Federation 2 100 13 62 2 50 Serbia 7 71 2 50 Turkey 7 57 8 75 Ukraine 7 71 2 0 2 100 IBRD total 76 80 45 47 18 67 Moldova 11 73 1 100 Blend Uzbekistan 2 0 1 100 Blend total 13 62 1 100 1 100 Kosovo 1 100 1 100 IDA Kyrgyz Republic 16 75 1 100 1 100 Tajikistan 11 91 1 0 3 67 IDA total 28 82 Eastern Europe Region 2 0 South Eastern Europe and Other Balkans 2 100 Southern Europe Region 1 0 Other total 2 100 3 0 TOTAL 119 79 51 45 24 71 Source: IEG data. Note: The data excludes projects not mapped to a region. MS+ = moderately satisfactory or better (WB rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number’ refers to the total number of projects rated by IEG. 154 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 6. Europe and Central Asia Region: IEG Ratings of Project M&E Quality, FY13–15 Europe and Central Asia Other Regions Bank-wide Rated Rated Rated Total number substantial Total number substantial Total number substantial (%) (%) (%) IBRD 76 38 260 32 336 33 Blend 13 31 94 26 107 26 IDA 28 43 281 31 309 32 Other 2 0 33 45 35 43 Total 119 38 668 31 787 32 Source: IEG data. Note: The data excludes projects not mapped to a region. M&E = monitoring and evaluation; total number refers to the total number of projects rated by IEG. Table 7. Europe and Central Asia Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 CASCR/ World Bank IFC MIGA FY of CLR Outcome Country performance performance performance review review rating rating rating rating period Armenia FY09–13 MS S 2014 Georgia FY10–13 S Good Good Fair Kyrgyz Republic FY07–13 MS MS Moldova FY09–13 MS MS 2014 Poland FY09–13 MS S Romania FY09–13 MS Good Tajikistan FY10–14 MS Good Albania FY11–14 MS Good Azerbaijan FY11–14 S Good 2015 Macedonia, FYR FY11–14 MS Fair Good Serbia FY12–15 MS Fair Bosnia and Herzegovina FY12–15 MS Good Bulgaria FY11–13 U Fair 2016 Montenegro FY11–15 MS Good Uzbekistan FY12–15 MS Good Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory; S = satisfactory; U = unsatisfactory. 155 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 8. Europe and Central Asia Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16 2012 2013 2014 2015 2016 % of % of % of % of % of $, $, $, $, $, Global Practice millions Region millions Region millions Region millions Region millions Region total total total total total Agriculture 18 0 50 1 239 4 27 0 0 0 Education 17 0 57 1 54 1 358 5 103 1 Energy and Extractives 1,320 20 391 7 852 15 1,362 19 30 0 Environment and Natural Resources 0 0 44 1 60 1 121 2 102 1 Finance and Markets 368 6 638 12 300 5 750 10 1,027 14 Governance 0 0 122 2 86 2 110 2 158 2 Health, Nutrition, and Population 10 0 145 3 554 10 265 4 214 3 Macroeconomics and Fiscal Management 3,170 49 2,479 47 2,356 43 1,828 25 2,550 35 Poverty 0 0 10 0 0 0 0 0 0 0 Social Protection and Labor 61 1 0 0 21 0 426 6 0 0 Social, Urban, Rural, and Resilience 221 3 145 3 400 7 152 2 351 5 Trade and Competitiveness 10 0 76 1 50 1 433 6 162 2 Transport and ICT 1,248 19 890 17 45 1 1,099 15 2,069 28 Water 73 1 273 5 510 9 277 4 504 7 % of % of % of % of % of $, total $, total $, total $, total $, total Region millions IBRD millions IBRD millions IBRD millions IBRD millions IBRD and IDA and IDA and IDA and IDA and IDA Europe and Central Asia Region 6,516 18 5,320 17 5,527 14 7,207 17 7,271 16 Other Regions 28,740 82 26,228 83 35,154 86 35,288 83 38,628 84 Total 35,256 31,547 40,681 42,495 45,899 Source: Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. 156 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Latin America and the Caribbean Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 Latin America and the Caribbean Regional Update, the slowing pace of global activity and commodity prices that remain weak—with prospects that they will remain lower for longer—provide the backdrop for the Region in 2016. Economic activity in the region was hard hit and is likely to contract for the second consecutive year (IMF 2016). However, differences in individual country growth outcomes resulted from external and domestic factors that have affected countries differently. The ongoing U.S. recovery continues to support activity in Mexico, Central America, and the Caribbean, but China’s manufacturing- based slowdown reduced the demand for exports from South America, and further declines in commodity prices added to the accumulated terms-of-trade shock for commodity exporters. Adjustment to the more fragile external conditions was smooth in countries where macro policy frameworks had improved during the past two decades, but policy missteps, domestic imbalances, and rigidities led to sharp declines in private demand in a few countries. Lending broadly reflects the World Bank Group’s strategic framework for the region, which sets the stage for economic recovery, strengthening infrastructure services, investing in human capital, and protecting the poor (World Bank 2016g). 2. The World Bank’s new lending commitments in the Latin America and the Caribbean Region trended upward in recent years: $5.2 billion in FY13, $5.1 billion in FY14, $6.0 billion in FY15, and $8.2 billion in FY16. Commitments in FY16 represented 18 percent of all World Bank commitments, almost reaching the same proportionate share as FY12 ($6.6 billion, 19 percent). The Region’s share of overall commitments had diminished in the intervening years (table 8). By World Bank Global Practice, FY16 commitments are concentrated in Macroeconomic and Fiscal Management (25 percent), Governance (18 percent), Environment and Natural Resources (11 percent), and 10 percent in Transport and Information and Communication Technologies (ICT). Commitments under Macroeconomic and Fiscal Management, and Transport and ICT are part of a pattern of significant support for projects under those practices during FY12–16 (table 8). 3. IEG rated 152 projects in the Region, of which 73 percent were rated MS+— just above the World Bank average of 72 percent MS+ for FY13–15 by number of projects (figure 1). By commitments, IEG rated 89 percent of projects as MS+, which is slightly better than the 86 percent MS+ World Bank average by commitments (figure 1). 157 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 4. Allowing for small numbers of projects in most instances, projects under the Education, Environment and Natural Resources, Governance, Poverty, Trade and Competitiveness, and Transport and ICT Global Practices performed better than the relevant Global Practice average across the World Bank. Projects in the Water sector (54 percent MS+, 13 projects rated) and the Energy and Extractives sector (43 percent MS+, 7 projects rated) performed below average for FY13–15 (table 1). 5. IEG rated 111 projects across 15 IBRD countries, of which 75 percent were rated MS+ for FY13–15—better than the performance of the 13 projects rated across four blend countries (62 percent MS+) and the 23 projects rated across four IDA countries (65 percent MS+). In six IBRD countries, two blend countries, and one IDA country, IEG rated all 22 projects as MS+. Relatively poor performance for the 27 projects rated in Brazil (IBRD) during FY13–15 affected the Region’s overall performance because only 56 percent of those projects were rated MS+, and 2 (29 percent) of the seven projects rated in Honduras (IDA) during the period were rated MS+ (table 5). 6. Taking a closer look at the poor performance of projects rated for the Water practice FY13-15 (table 1), Implementation and Completion Report (ICR) reviews suggest Water projects are generally challenging during implementation because they may involve engagement with different levels of government, significant infrastructure development that is technically difficult, and institutional change at both the national and subnational levels. The projects evaluated during FY13–15 included several at the subnational level (e.g., Brazil), or projects in which subnational levels of government had significant responsibility. Some of the projects rated moderately unsatisfactory (MU) for outcome (for example, the Rio Grande Do Norte Integrated Water Resources Management Resources had an implementing agency that showed limited capacity in leading and coordinating the project’s participating actors. The project management units also had limited capacity for managing and overseeing contractors, which in many cases produced poor-quality outputs (activities were unfinished or reduced in scope). A lesson from the region’s water sector is that streamlined projects with focused activities might be easier for low- capacity borrowers to implement because many diverse activities at different government levels can create an implementation burden that is too heavy for the borrower. Another lesson is that adopting a subnational lending strategy requires balancing both significant benefits and drawbacks. Lending directly to subnational governments can contribute to greater project intervention relevance, narrower project scope that helps in measuring the project’s impact, and strengthened institutional capacity. However, the subnational borrower might be unaware of its implementation weaknesses and therefore have limited capacity to procure the right technical assistance to address weaknesses or to absorb it once procured. 158 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Furthermore, the need to clarify loan guarantee terms between the province and the national government might cause delays. 7. Projects in the Energy and Extractives practice also performed well below the average for the Region, noting only seven such projects were validated FY13-15 (table 1). ICR reviews suggest implementation delays were common across projects rated moderately unsatisfactory and below. In the case of the Renewable Energy Resource Markets project in Argentina (rated moderately unsatisfactory), three of the four objectives were not achieved. This was largely associated with the underestimation, at entry, of risks that ultimately materialized in the form of the economic crisis of 2001. This resulted in insufficient budget allocations to the project at the federal level, and reduced budget allocations for rural electricity subsidies at the provincial level leading to major delays in project execution and in the realization of anticipated economic benefits. Haiti’s Electricity Project was rated unsatisfactory (U). IEG’s review found administrative inefficiencies led to cost overruns and implementation delays (even prior to the earthquake). Furthermore, modest design meant that the outcome for the first objective—achieving sustainable improvement in the quality of electrical services to customers—could not be attributed to the project. There was a negative outcome for the second objective— strengthening the financial and operational performance of the power utility company. The Power Sector Efficiency Project in Honduras was rated unsatisfactory. In this case, considerable operational inefficiencies resulted in delays and the need for major reallocation of IDA resources. Finally, the Bolivia Decentralized Electricity for Universal Access project is rated moderately unsatisfactory (MU) based on modest efficiency and modest design relevance. 8. Lessons from ICR reviews suggest that projects rated satisfactory for development outcomes highlight some well-known ingredients for success: government commitment, dialogue and consensus building among beneficiaries, and focused interventions that work in partnership with existing institutions in the private and public sectors. The Environmental Services Project in Mexico enhanced the provision of nationally and globally significant environmental services, and secured the long-term sustainability of environmental services by supporting projects that were part of the government’s larger Pro-Árbol Program, and thus received substantial commitment as shown in the level and consistency of government funding allocations. In Nicaragua, the Land Administration Project helped develop a legal, institutional, and participatory framework for property rights administration, and showed the feasibility of a systematic land rights regularization program. Building broad social and political commitment with a strong focus on local government investment in cadaster and registry modernization successfully addressed these problematic social issues. The project is an example of 159 APPENDIX I REGIONAL RESULTS AND PERFORMANCE alternative conflict resolution mechanisms facilitating cadastral and regularization processes. The Peru Rural Electrification Project showed that seeking to engage actively with distribution companies from the early stages of program design is essential for gaining ownership, adequate financing, and overall effective design. The productive use of electricity—especially in middle-income countries with a critical mass of entrepreneurs—can be key to achieving long-term development impacts from rural electrification programs. 9. The six project performance assessment reports (PPARs) disclosed in FY16 provide additional insights on project performance drivers. For example, productive partnership projects need to pay attention to both farmers and buyers. The PPAR for Colombia’s Productive Partnership Project notes that the project paid more attention to the productive capacity of smallholders than of commercial entities. This adversely affected commercial entities’ engagement in project opportunities because they lacked awareness. Generally, such projects are more likely to succeed when they are part of an integrated rural development approach. Farmer alliances in Colombia suffered the consequences of poor rural infrastructure (storage, roads, and power) and weak extension services. The lack of cold storage facilities and the high cost of transport significantly constrained perishable goods producers’ earning opportunities. The PPAR for Nicaragua’s Education Project (FY05) suggests that World Bank Group support needs to go beyond technical aspects. In developing the education project, the World Bank provided strong technical support in designing the school autonomy model. However, it appears that the World Bank did not provide sufficient support to the reform’s logistics elements, such as piloting an information system or building local capacity. Similarly, the government and the World Bank did not consider the political economy behind the reform. 10. IEG rated the Latin America and the Caribbean Region’s performance for monitoring and evaluation (M&E) as substantial or better for 27 percent of projects, which is below an already low World Bank average (32 percent), indicating particularly low overall satisfaction with M&E design, implementation, and utilization during FY13–15. Common gaps in M&E include poor results chains, overreliance on output indicators, a lack of outcome indicators, using indicators not directly relevant to project objectives or not directly attributable to project activities, and a lack of baseline data that precludes measuring progress (table 6). 11. IEG validated 53 IFC investment projects for FY13–15 and assigned ratings of mostly successful or higher to 33 projects (62 percent), which was better than the IFC average (54 percent). The Region’s success rate by investment commitments was 70 percent, also better than the 64 percent IFC average success rate (table 2). 160 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 12. IEG validated 30 IFC Advisory Services projects during the review period and assigned ratings of mostly successful or higher to 20 projects (67 percent), which is better than the 61 percent IFC average for Advisory Services projects (table 3). 13. IFC investment or advisory support to microfinance institutions in the Region jointly includes 39 percent of the global micro loan volume, signifying IFC’s leadership role in this area (IEG 2015b). The portfolio analysis, which includes many countries in the Latin America and the Caribbean Region, shows the importance of IFC advisory interventions in establishing a strong enabling environment. Relevant projects, often executed in collaboration across the World Bank Group, frequently benefited from high-quality analytical work and stakeholder assessments. 14. A lesson from an IEG PPAR that assessed an IFC housing finance technical assistance project (IFC Advisory Services) in Mexico is that including mid-term corrections in projects targeting untested financial markets can help enable the creation of a mortgage loan product that reaches the poor. Market-making projects carry unknowns that this project identified and embedded as part of the implementation process. IFC support training provision, the creation of mortgage units, and the launch of a mortgage loan product. Afterward, a series of visits to the financial institutions to test procedures, methodology application, and the product itself led to significant project corrections. One of the main findings was that the number of potential clients for the product and service innovations that IFC’s client could identify was constrained significantly by the proof of formal income requirements. The project team suggested a socioeconomic study to explore the situation. Furthermore, the project team helped the institution define a solution that engaged potential clients in a six-month savings program that helped build trust, which allowed clients to access housing loans. This product increases the potential for low-income clients with no formal proof of income to access housing. Such clients typically have difficulty demonstrating their willingness and ability to pay. 15. IEG evaluated and validated one Multilateral Investment Guarantee Agency (MIGA) investment project in the Region during FY13–15, and it was not rated satisfactory or higher (table 4). To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. 161 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Findings from Thematic, Global, and Corporate Evaluations 16. A lesson from IEG’s 2016 learning product World Bank Group Support for Housing Finance is that when expanding affordable housing is an intervention’s key objective, the intervention needs to be targeted and the results measured adequately (IEG 2016s). In Honduras, an IFC joint advisory and investment project supported a commercial bank with the objective of moving down-market by increasing housing segments as part of the client bank’s growth strategy. The housing finance expansion outperformed its targets, for example, for value and numbers of outstanding loans in its portfolio and its targeted average loan, which likely enhanced access to finance for first-time housing finance borrowers from the middle- and upper-income segments. However, IEG found little evidence that the project fully supported its objective to serve low- and medium-income households. Because of the high average loan size and minimum and maximum income requirements ($500–$1,500 per month), the bulk of expanded mortgages likely went to borrowers in the upper-middle and above income categories instead. Furthermore, if there is no significant mortgage financing in the target country, the low-income segment should not be prioritized, but should be introduced gradually as the mortgage financing market develops. In Mexico, the World Bank supported the Sociedad Hipotecaria Federal (SHF) to stabilize and develop the mortgage market with two minor components to expand access to low-income housing. Although the project allowed SHF to inject liquidity into the sector and prevented the slowdown of the housing market in the aftermath of the crisis, expansion of products toward lower-income segments was modest because the goal of market stabilization and strengthening SHF’s financial position was the priority. 17. IEG’s Lessons from Land Administration Projects: A Review of Project Performance Assessments suggests that technical integration and interagency coordination and collaboration need to be part of a project when different agencies manage the cadaster and registry functions (IEG 2016j). The report notes that an insufficient understanding of tenure insecurity in the region hindered the Guatemala Land Administration project. The project assessment found that the project pilot was in a region where lawlessness was so pervasive that even regularizing the peasant farmers’ tenure status would not protect them from subsequent acts of coercion intended to make them give up their land. To enhance tenure security, it would have been important for this type of project in that particular region to embed the regularization process into a wider program to strengthen law and order and reduce the scope of physical coercion of property owners. This wider commitment was missing when the project was prepared. 162 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 18. IEG’s 2016 learning product Supporting Transformational Change for Poverty Reduction and Shared Prosperity identified a sample of such engagements from 2000– 14 and found that although it may be difficult to identify transformational engagements ex ante, some salient characteristics can be identified ex post (IEG 2016p). The report references a project in Peru that sought to provide a rural roads system to help alleviate rural poverty through increasing access to markets, income- generating opportunities, and basic social services. The program supported systemic change by developing the institutional capacity for rural road decentralization at several levels. It also included specific pro-poor components, such as a mechanism to provide funds for local development that helped small associations of rural producers to develop business plans, obtain funding, and capitalize on opportunities the new infrastructure provided. The program’s maintenance activities create job opportunities for poor, rural men and women who live alongside the rehabilitated roads—a concept replicated in Bolivia, Ecuador, Guatemala, Haiti, and Honduras. The program in Peru was eventually scaled up to cover the entire country, and an impact evaluation for Peru shows the rural roads interventions had a large impact on reducing extreme poverty (Macroconsult- Cuanto 2014). 19. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence (IEG 2016q) notes that the World Bank experience during the past 20 years in Colombia shows that interventions generally aimed to mitigate the impact of conflict and violence on households, farmers, and communities, and to address some conflict drivers. These two goals were interlinked, and one or other was prioritized, subject to circumstance and need over time. The World Bank started with mitigation activities at the local level by supporting communities under extreme duress, and as security gradually improved, it began working with local institutions, emphasizing education sector support. Eventually the World Bank turned its focus to facilitating broader regional or national-level policies focused on land titling and restitution. A Evaluation Findings from Country Program Performance in Latin America and the Caribbean 20. Analysis undertaken for Results and Performance of the World Bank Group 2016 shows that for the 21 countries in the region for which IEG validated two country strategies during FY07-16—including three OECS countries rated together—overall development remained moderately satisfactory (MS) or above for 11 countries, 163 APPENDIX I REGIONAL RESULTS AND PERFORMANCE improved from MU to MS for Colombia, Dominican Republic, Guyana, Jamaica, the OECS countries, and remained MU for Bolivia, Honduras, and Paraguay.1 21. IEG completed six completion and learning reviews (CLRs) for the Latin America and the Caribbean Region in FY16. Overall development outcome at the country level is rated satisfactory for Uruguay, MS for Colombia and Guyana, MU for Bolivia and Honduras, and unsatisfactory for Haiti noting the implementation of the CAS FY09-12 was interrupted by the catastrophic earthquake of January 2010. It is also notable that two post-earthquake interim strategy notes (ISNs) for Haiti, CY12 and FY12-FY13, were rated moderately satisfactory (MS) World Bank performance is rated good or fair in all six cases (table 7). 22. The main lessons in the six FY16 CLR reviews highlight that responding to country priorities and being flexible, open, and selective will likely lead to a substantive strategy. Furthermore, project preparation that fails to assess institutional capacity and identify potential gaps and related mitigating measures will likely to lead to subpar project implementation that can affect country-level outcomes. The need to pay more attention to what happens after project closing is a lesson on intervention sustainability—when preparing and appraising a project, the World Bank should incorporate prospects and plans for sustainability on project completion. In some operations (Colombia), the World Bank did not conduct the extensive due diligence required or maintain the fluid dialogue needed with the government to anticipate adverse outcomes and facilitate decision making by both the World Bank and the authorities. 23. Many of the countries in the Region for which a CLR review was conducted in FY16 moved up the income ladder over the course of their country strategies, strengthened their policy frameworks, and became more mature and demanding in their relations with the World Bank Group. Clients have a clear idea of what they need from the World Bank Group and how the World Bank Group can contribute to their country development goals. The positive effect of this new reality is more government ownership of country programs, better results in areas of World Bank involvement, and a new relationship in which country officials and private and public institutions are more closely involved in the design of World Bank interventions and more committed to the follow-through with policy and project commitments. An example is Uruguay, which led program preparation and implementation, underscoring its ownership of the program. However, this 1The 21 countries included in this analysis were: Argentina, Bolivia, Brazil, Chile, Colombia, Costa Rica, Dominican Republic, El Salvador, Guatemala, Guyana, Honduras, Jamaica, Mexico, Nicaragua, OECS (3) countries, Panama, Paraguay, Peru, and Uruguay. 164 APPENDIX I REGIONAL RESULTS AND PERFORMANCE approach can potentially leave the World Bank Group out of areas in which it has significant comparative advantage, experience, and much to contribute—for example, secondary and tertiary education in Uruguay.2 24. The discrepancy between the development outcome ratings and World Bank performance ratings in country programs relate to a number of factors of World Bank Group program design and supervision, and response to exogenous changes. Although development outcomes generally improved, the design of results frameworks had weaknesses. In several instances country program results frameworks did not incorporate IFC and MIGA activities. In others, they relied on outcome indicators that poorly reflected the targeted outcomes, were based mostly on outputs instead of outcomes, or did not quantify baselines and target values for indicators, which made program assessment difficult. These shortcomings also had adverse effects on supervision because teams could not use the results framework to closely monitor program performance or restructure interventions when needed. In some cases, when indicators were redefined at the progress report stage, the level of aspiration, or reach, was lowered while World Bank financing was increased, particularly through DPLs to support the budget. For example, the Honduras country level results framework was poorly designed, and it failed to incorporate IFC and MIGA activities even at the progress report stage when such activities were more certain. In other cases, such as Colombia, the original program design was unrealistic because of unanticipated or unaddressed implementation constraints and other implementation issues (including fiduciary) many of which were associated with predictable inadequacies in institutional capacity. IEG’s review found that revisions at progress report stage curtailed the overall ambition of the strategy, and made the results framework more precise in light of scaled down Bank interventions. Outcomes were cut from nine to eight, and certain outcomes were revised to better reflect Bank interventions. Of the original 26 indicators, 15 were revised, seven dropped for improved measurement, and five were added to account better for results, although, in some instances, the redefinition of outcomes led to a significant lowering of the bar with indicators set as outputs rather than outcomes, despite a significant increase in Bank financing. Generally, the new reality of enhanced country program ownership by the authorities and private and public institutions and their greater involvement in implementation outweighed some 2Although other development partners (Inter-American Development Bank) were involved in these areas, the completion and learning review recognized that a more comprehensive education strategy would probably have been more effective to improve education results in Uruguay, which significantly lags peers of similar per capita income in education achievement. 165 APPENDIX I REGIONAL RESULTS AND PERFORMANCE shortcomings of World Bank Group performance to produce better development outcomes and help countries achieve their development goals. 25. IEG conducted its country program evaluation (CPE) for Bolivia (FY05–13) as part of the clustered CPE for resource-rich developing countries. The Bolivia CPE found that the government must focus on improving agricultural productivity and living standards in rural areas to achieve the goal of substantially reducing extreme poverty. It also recommends that the World Bank Group collaborate with the government to examine the interventions it supported, evaluate their impact on poverty, and distinguish between their social assistance and productivity-enhancing aspects. This would enable the World Bank Group and the government to identify the main problems, diagnose their causes, and set priorities for a development strategy. The CPE also found that assistance provided in urban development is likely to be more effective and its scope more easily expanded if the assistance is embedded in a government program. To ensure smooth project execution, the World Bank must devote resources to help its Bolivian counterparts learn about and comply with World Bank procedures, especially the procurement and contracting aspects. Regarding governance, the World Bank emphasized staying engaged and now maintains connections in public financial management, decentralization, and transparency and institutional strengthening. The evaluation finds that although the approach could have been appropriate for a transitional period, it may no longer be appropriate because the World Bank could incur reputational risk from supporting politicized projects. The evaluation recommends that the World Bank use analytic work to stay current on development in relevant areas and respond to the government’s requests for assistance on specific topics. 166 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Figure 1. Latin America and the Caribbean Region: Percentage of Projects Rated MS+ on Outcome FY01–15 100 Bank-wide Latin America & Caribbean 95 Non-Latin America & Caribbean Regions 90 85 81 82 81 79 78 78 79 78 80 77 77 76 Percentage 76 73 75 78 79 75 76 70 73 73 74 72 65 70 70 68 68 68 60 55 50 Fiscal year, three-year roliing Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 167 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 1. Latin America and the Caribbean Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13–15) Latin America and the Caribbean Other Regions Bank-wide Number of Net commitments Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) projects (US$, millions) Global Practice Rated Rated Rated Rated Rated Rated Total Total Total Total Total Total MS+ MS+ MS+ MS+ MS+ MS+ number amount number amount number amount (%) (%) (%) (%) (%) (%) Agriculture 15 67 524 69 58 83 3,559 92 73 79 4,083 89 Education 14 86 976 95 66 71 5,600 83 80 74 6,576 85 Energy and Extractives 7 43 465 73 66 68 6,799 79 73 66 7,264 79 Environment and Natural Resources 11 73 504 93 39 59 1,106 75 50 62 1,610 81 Finance and Markets 3 67 1,021 99 31 74 3,980 95 34 74 5,001 96 Governance 13 77 2,447 98 42 50 2,259 47 55 56 4,706 74 Health, Nutrition, and Population 12 67 2,270 90 55 75 3,076 78 67 73 5,345 83 Macroeconomics and Fiscal Management 11 82 2,930 96 38 82 7,921 98 49 82 10,850 97 Poverty 3 100 1,109 100 4 50 66 78 7 71 1,175 99 Social Protection and Labor 10 80 3,712 99 30 93 2,707 96 40 90 6,419 98 Social, Urban, Rural, and Resilience 22 68 1,850 52 74 72 6,732 89 96 71 8,582 81 Trade and Competitiveness 1 100 13 100 16 63 1,002 53 17 65 1,015 54 Transport and ICT 17 88 3,329 98 61 77 6,310 84 78 79 9,639 89 Water 13 54 1,437 47 59 71 4,631 82 72 68 6,069 74 Total 152 73 22,584 89 639 72 55,749 85 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; total number refers to the total number of projects rated by IEG. 168 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 2. Latin America and the Caribbean Region: IEG Outcome Ratings of IFC Investment Projects, FY13–15 Latin America and the Caribbean IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 6 67 115 95 34 38 904 56 and Funds Financial Markets 16 56 229 64 70 59 1,705 62 Infrastructure and Natural 13 69 409 68 41 63 1,518 79 Resources Manufacturing, Agribusiness, 18 61 337 70 84 51 2,105 59 and Services Total 53 62 1,090 70 229 54 6,232 64 Source: IEG data. Note: MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. Latin America and the Caribbean Region: IEG Outcome Ratings of IFC Advisory Projects, FY13–15 Latin America and IFC overall the Caribbean IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 7 71 43 70 Cross-Cutting Advisory Solutions 1 100 4 75 Cross-Cutting Advisory Solutions: Public-Private 4 25 14 43 Partnerships Environment, Social, and Governance 0 n.a. 1 100 Financial Markets 0 n.a. 5 40 Financial Institutions Group 3 67 17 47 Infrastructure and National Resources 2 100 2 100 Investment Climate 5 80 26 77 Manufacturing, Agribusiness, and Services 0 n.a. 3 33 Public-Private Partnerships 1 100 17 47 Sustainable Business Advisory 7 57 38 66 Telecom, Media, Technology, and Venture Capital 0 n.a. 1 0 Trade and Competitiveness 0 n.a. 8 50 Total 30 67 179 61 Source: IEG data. Note: MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Please note that during FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public-Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. Table 4. Latin America and the Caribbean Region: MIGA Development Outcome Ratings by Sector, FY13–15 169 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Latin America and MIGA overall the Caribbean Sector Total Total Rated S+ (%) Rated S+ (%) number number Agribusiness 0 n.a. 1 0 Financial 1 0 8 38 Infrastructure 0 n.a. 9 44 Manufacturing 0 n.a. 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 0 n.a. 2 0 Services 0 n.a. 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 1 0 34 47 Source: IEG data. Note: S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Table 5. Latin America and the Caribbean Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank IFC IEG XPSR IFC IEG PCR projects ratings ratings Rated Rated Total Total Total Rated Country number MS+ number MS+ number MS+ (%) (%) (%) Argentina 21 71 7 71 Belize 1 100 Brazil 27 56 9 56 6 83 Chile 3 100 Colombia 14 86 10 90 4 100 Costa Rica 3 33 2 0 Dominican Republic 3 100 1 100 1 100 Ecuador 1 100 1 100 IBRD El Salvador 2 100 1 100 1 0 Guatemala 6 83 Jamaica 4 75 3 33 Mexico 10 90 6 50 2 50 Panama 5 60 1 0 Paraguay 1 100 3 67 Peru 7 86 1 0 2 50 Uruguay 4 100 1 100 IBRD total 111 75 45 62 18 78 170 APPENDIX I REGIONAL RESULTS AND PERFORMANCE World Bank IFC IEG XPSR IFC IEG PCR projects ratings ratings Rated Rated Total Total Total Rated Country number MS+ number MS+ number MS+ (%) (%) (%) Bolivia 8 50 1 100 Grenada 2 50 1 0 Blend St. Lucia 2 100 St. Vincent and the 1 100 Grenadines Blend total 13 62 1 100 1 0 Guyana 2 100 1 0 Haiti 6 67 2 50 IDA Honduras 7 29 5 60 Nicaragua 8 88 1 100 2 50 IDA total 23 65 2 50 9 56 Andean Countries 1 100 Barbados 1 100 Caribbean Region 1 0 Other Central America Region 1 100 2 50 Latin America Region 1 100 3 67 1 100 OECS Countries 1 100 Other total 5 100 5 60 2 50 TOTAL 152 73 53 62 30 67 Source: IEG data. Note: The data excludes projects not mapped to a Region. MS+ = moderately satisfactory or better (World Bank rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number refers to the total number of projects rated by IEG. Table 6. Latin America and the Caribbean Region: IEG Ratings of Project M&E Quality, FY13–15 Latin America and Other Regions Bank-wide the Caribbean Rated Rated Total Rated Total Total number substantial substantial number substantial (%) number (%) (%) IBRD 113 32 223 34 336 33 Blend 13 0 94 30 107 26 IDA 23 22 286 33 309 32 Other 5 20 30 47 35 43 Total 154 27 633 33 787 32 Source: IEG data. Note: The data excludes projects not mapped to a Region. M&E = monitoring and evaluation; total number refers to the total number of projects rated by IEG. 171 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 7. Latin America and the Caribbean Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 CASCR/CLR World Bank IFC MIGA FY of Outcome Country review performance performance performance review rating period rating rating rating Jamaica FY10–13 MS Good 2014 Mexico FY08–13 MS MS Argentina FY10–14 MS Fair Good Costa Rica FY12–15 MS Fair Dominican FY10–13 MS Fair Good Republic 2015 El Salvador FY10–14 MS Good OECS countries FY10–14 MS Good Fair Panama FY11–14 MS Fair Paraguay FY09–14 MU Fair Bolivia FY12–15 MU Good Colombia FY12–16 MS Fair Guyana FY09–12 MS Good 2016 Haiti FY09–14 U Good Honduras FY12–15 MU Fair Uruguay FY11–15 S Good Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory; MU = moderately unsatisfactory; S = satisfactory; U = unsatisfactory. 172 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 8. Latin America and the Caribbean Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16 2012 2013 2014 2015 % of % of % of %o $, $, $, $, Global Practice millions region millions region millions region millions regi total total total tot Agriculture 388 6 580 11 290 6 203 3 Education 626 9 290 6 523 10 751 12 Energy and Extractives 50 1 110 2 50 1 200 3 Environment and Natural Resources 710 11 0 0 0 0 59 1 Finance and Markets 100 2 0 0 0 0 0 0 Governance 128 2 1,100 21 286 6 60 1 Health, Nutrition, and Population 80 1 220 4 10 0 410 7 Macroeconomics and Fiscal Management 1,747 26 796 15 1,215 24 1,830 30 Poverty 500 8 0 0 623 12 0 0 Social Protection and Labor 75 1 615 12 72 1 875 15 Social, Urban, Rural, and Resilience 427 6 714 14 307 6 650 11 Trade and Competitiveness 480 7 20 0 350 7 50 1 Transport and ICT 1,015 15 701 13 932 18 730 12 Water 304 5 58 1 410 8 208 3 % of % of % of %o $, total $, total $, total $, tot millions IBRD millions IBRD millions IBRD millions IBR and IDA and IDA and IDA and I Latin America and the Caribbean Region 6,629 19 5,204 16 5,068 12 6,024 14 Other Regions 28,627 81 26,343 84 35,613 88 36,470 86 Total 35,256 31,547 40,681 42,495 Source: World Bank Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. Middle East and North Africa Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 Middle East and North Africa Regional Update, a combination of factors threatens the strong achievements in reducing extreme poverty and boosting shared prosperity, including refugees and displacement, violence and conflict, low commodity prices, and climate vulnerability. These factors increase the region’s fragility as a whole and the need for strategic World Bank Group support while making the operational environment riskier. Along with these pronounced factors, countries in the Middle East and North Africa Region still face many long-standing development challenges and structural issues. Voice and accountability remain low across the region, and labor force participation rates for 173 APPENDIX I REGIONAL RESULTS AND PERFORMANCE women are the lowest in the world (in 17 of 19 countries, women’s participation is below the world average of 50 percent in 2016). The region’s level of financial inclusion is among the lowest in the world, particularly for women (in 2014, 19 percent of men and 9 percent of women had a Bank account compared to the next highest Region, Africa, where, respectively, 34 and 30 percent had bank accounts), and youth unemployment, particularly among educated youth is the world’s highest (39.5 percent of those educated to tertiary level were unemployed in 2015). The region is the world’s most water-scarce area, its droughts driving fragility and displacement. Middle East and North Africa is more generally vulnerable to climate change impacts, especially regarding water scarcity and food security. To address these challenges, the Middle East and North Africa Regional Update 2016 proposed a new strategy, Economic and Social Inclusion for Peace and Stability: A New Strategy for the World Bank Group. The strategy builds on four mutually reinforcing pillars: renewing the social contract, regional cooperation, resilience to shocks associated with internally displaced persons and refugees, and recovery and reconstruction (World Bank 2016i). 2. The World Bank’s new lending commitments in the Region trended upward in recent years: $2.1 billion in FY13, $2.8 billion in FY14, $3.5 billion in FY15, and $5.2 billion in FY16. Commitments in FY16 represented 11 percent of all World Bank commitments compared with 4 percent in FY12 and 7 percent in FY13 (table 8). More than 80 percent of all FY16 commitments are concentrated in the following four Global Practices: Macroeconomics and Fiscal Management ($1.7 billion, 33 percent); Energy and Extractives ($1.25 billion, 24 percent); Water ($0.70 billion, 14 percent); and Social, Urban, Rural, and Resilience ($0.55 billion, 11 percent). 3. Performance of World Bank operations in the Region is below the World Bank performance as a whole. Outcome ratings at exit (FY13–15) were 60 percent moderately satisfactory or better (MS+) for the Region compared with a World Bank average of 72 percent (figure 1). 4. IEG rated all projects under five Global Practices MS+ for outcome during the period: Agriculture (three projects), Environment and Natural Resources (five projects), Finance and Markets (two projects), Health, Nutrition, and Population (one project), and Macroeconomics and Fiscal Management (one project). However, performance is relatively weak for the Education and Water Global Practices. In Education, IEG rated five of 11 projects (45 percent) as MS+, and for Water, three of eight projects (38 percent) are rated MS+. For the Social, Urban, Rural, and Resilience Global Practice, only four of 10 projects (40 percent) are rated MS+ for FY13–15, which is lower than performance for projects in other Global Practices in the Region (table 1). 174 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 5. Average ratings for projects across IBRD and IDA countries in the Region during FY13–15 (56 percent and 55 percent MS+, respectively) was below the Region’s average performance of 60 percent MS+. Five projects in the West Bank and Gaza (which is not categorized as IDA or IBRD) were all rated MS+, which raised the Region’s average performance. Two IBRD countries—the Arab Republic of Egypt and Morocco—are relatively high performers in the region in FY13–15: four of five projects in Egypt (80 percent) and three of four projects in Morocco (75 percent) are rated MS+. Projects in other IBRD countries did not perform as well—55 percent of projects were rated MS+ in Iraq, 56 percent in Jordan, and 43 percent in Tunisia (table 5). 6. Middle East and North Africa Region performance for monitoring and evaluation (M&E) is rated substantial or better for only 26 percent of projects, which is below an already low World Bank average (32 percent), indicating low overall satisfaction with M&E design, implementation, and utilization during FY13–15 (table 6). 7. IEG did not complete any project performance assessment reports (PPARs) for projects in the region in FY16. 8. IEG validated 21 IFC investment projects in the Middle East and North Africa Region in FY13–15 and assigned ratings of mostly successful or higher to 10 projects (48 percent), slightly below the IFC average of 54 percent. The region’s success rate by investment commitments was 56 percent, also below the IFC average success rate of 64 percent (table 2). 9. The investment climate is particularly challenging in the region. An IEG Evaluative Note (EvNote) for an IFC investment in Egypt stresses the importance of clearly defining a project’s development goals, and emphasizes that ensuring client understanding and commitment to clearly defined developmental goals is necessary for proper interest alignment and successful implementation. A complementary lesson identified from a project in Sub-Saharan Africa indicates that investee management team deficiencies could go undetected during appraisal. Compelling vision and relationships are not enough for success, and solid investee internal processes and controls are also needed, even in small project development companies. 10. IEG validated 21 IFC Advisory Services projects during the review period and assigned ratings of mostly successful or higher to 13 projects (62 percent), which was slightly higher than the IFC average for Advisory Services projects (61 percent). A key lesson from an IEG EvNote for a successful project suggests careful review of 175 APPENDIX I REGIONAL RESULTS AND PERFORMANCE a given company’s capacity to raise enough wholesale funding locally to fund its growth (table 3). 11. IEG evaluated and validated two Multilateral Investment Guarantee Agency (MIGA) investment projects during FY13–15, both of which were rated satisfactory or higher (100 percent), which is higher than the MIGA average of 47 percent (table 4). An IEG PPAR recently assessed a MIGA-issued guarantee to two financial institutions covering their loan to the Tunisian state-owned shipping company for 13 years. A lesson is that MIGA’s non-honoring of sovereign financial obligations (NHSFO) product can have a valuable countercyclical role in helping projects and companies with sound fundamentals to access lending and capital markets when it would otherwise be too costly or out of reach. The PPAR suggests that linking more closely with the World Bank Group in choosing projects to support with the NHSFO cover would allow prudent risk management and help MIGA leverage the overall development impact of its NHSFO interventions. To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, negative and a No Opinion Possible (NOP) rating. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Findings from Thematic, Global, and Corporate Evaluations 12. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence found that when countries experience situations of fragility, conflict, and violence, the World Bank Group and the governments may have differing views on how and when to tackle the underlying issues, which makes providing effective support particularly challenging (IEG 2016q). 13. The evaluation notes that, in some situations, such as helping Lebanon and Jordan with the effects of neighboring crises, finding alternative financing instruments was a challenge for the World Bank Group. The most readily available option of IBRD loans was quickly exhausted (and not particularly embraced by the governments), but the alternative sources and mechanisms (IBRD surplus and trust funds) were not sufficient to meet needs. An important lesson from this experience is that the World Bank Group—with its shareholders’ political and financial support—needs to develop financial mechanisms or fast-response facilities to be used in similar situations, and use its global convening power more effectively while encouraging others to do so. 14. The evaluation draws other lessons about the World Bank’s engagement with Lebanon and Jordan in the context of the post-2011 crises that affected those 176 APPENDIX I REGIONAL RESULTS AND PERFORMANCE countries. World Bank assistance is not likely to alter the fragility profile of both countries fundamentally, though it is appropriate for addressing the short-term effects of the refugee crisis. The probability is high that a short-term crisis will become a serious long-term development challenge. The World Bank Group’s current strategy design is inadequate to deal with it. The experience in Jordan and Lebanon shows the importance of maintaining a credible and robust macroeconomic framework to withstand impacts from unforeseen crises. The World Bank will likely be an important institution (along with the IMF) in providing financial support when a crisis occurs. However, without a sustainable macro framework, the World Bank’s ability to increase its support will be limited. Without drastic structural reforms, Jordan and Lebanon will continue to be vulnerable to events like the crisis in Syria. Annex A provides additional cross-regional findings from the evaluation. 15. Managing Environmental and Social Risks in Development Policy Financing assesses the application of World Bank Operational Policy (OP 8.60) elements governing development policy financing related to implementation of the policy’s environmental and social risk management requirements (IEG 2015i). A project in Morocco was part of the review sample. The review found that in certain instances social risks were included in the provision of support for public-private partnerships (PPPs)—for example, risks associated with establishing an urban code that would affect where people could live. However, other operations supporting similar reforms (including the project in Morocco) identified no such risk. Annex A provides additional cross- regional findings from the learning product. 16. Given the Region’s ongoing challenges, two recent IEG evaluations referenced in Results and Performance of the World Bank Group 2015 (IEG 2016n) continue to be noteworthy. First, IEG’s evaluation The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises, 2006–12 distilled the experience of IFC’s West Bank and Gaza Olive Oil Supply Chain Development Project, in which linking a group of SMEs that “lack knowledge of required skills and Performance Standards to operate effectively” to value chains involving large firms enhanced the SME group’s performance. The evaluation showed the importance of targeting in the context of developing a financial institution (IEG 2014a). Without targets, financial institutions may choose to do more corporate lending. Often, targeting small and medium enterprises (SMEs) also requires the provision of a suite of complementary supports. 17. IEG’s evaluation, Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations, also showed the value of good targeting. In Morocco, the IFC Alternative Dispute Resolution Project’s 177 APPENDIX I REGIONAL RESULTS AND PERFORMANCE awareness-raising campaigns and public outreach efforts used a targeted approach to attract women to alternative dispute resolution. Awareness-raising events integrated gender, and commercial mediation and its implication on women business owners was the focal point of several events, such as a national conference. IFC also trained women mediators and supported a mediation center (IEG 2015d). Evaluation Findings from Country Program Performance in the Region 18. Analysis undertaken for Results and Performance of the World Bank Group 2016 shows that for the three countries—Morocco, Djibouti, and Tunisia—in the Middle East and North Africa Region for which IEG validated two country strategies during FY07–16, the rating for overall development remained moderately satisfactory (MS) for Morocco, improved from moderately unsatisfactory (MU) to MS for Tunisia, and remained MU for Djibouti. 19. IEG completed two completion and learning reviews (CLRs) for countries in Middle East and North Africa in FY16. Development outcome rating for the Egypt CLR (FY06–14) is unsatisfactory, and the Tunisia CLR (FY10–13) is rated MS. World Bank performance in Egypt is rated fair, and in Tunisia the performance is rated good (table 7). 20. The CLR for Tunisia covers FY10–11 (the one year of the country partnership strategy’s operation before the December 2010 revolution), and FY13–14, which is the period covered by the subsequent World Bank Group interim strategy note (ISN). Reflecting the findings of IEG’s Tunisia: World Bank Group Country Program Evaluation, FY05–12 the CLR is critical of the pre–2011 program, which failed to address critical bottlenecks that the World Bank knew about because of its own economic and sector work and other intelligence. Conversely, the CLR concludes that the ISN aimed to address critical constraints through its focus on sustainable growth and job creation, promoting social and economic inclusion, and strengthening governance. However, the CLR finds that the World Bank’s program was overoptimistic about the government’s ability to make and implement decisions, particularly given the two- year ISN period and the volatile political context. The program could have been more realistic and selective, although it did achieve progress in several areas under difficult circumstances, including support for macroeconomic stability, improvements in the labor reforms enabling environment, access to basic services, improved social safety net efficiency, and access to information. Progress was slow for issues such as unemployment and the transparency and accountability of institutions. IEG rated the performance of World Bank projects that closed during the CLR period (some of which were initiated in the early 2000s) as poor—less than 50 percent of 15 projects received a rating of MS+. 178 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 21. IFC worked with the World Bank to help strengthen the financial sector through an investors’ motivation survey, an investment climate assessment, and a pilot project matching job skills and labor demand. IFC also helped support micro, small, and medium enterprises through investment and advisory services, and initiated the Education for Employment Initiative for Arab Youth in Tunisia. IEG did not rate any IFC investments active in the CLR period, but internal self-rating suggests a challenging context for private investment (40 percent performing substandard or worse). 22. The CPF for Tunisia explicitly references and draws on recommendations made in IEG’s Tunisia country program evaluation covering FY05–11. The CPF embraces evaluation recommendations that urge the World Bank Group to do four things:  Conduct a political economy analysis to better manage risk in a volatile environment—risk mitigation scenarios are needed based on an ongoing analysis of risks associated with the political economy and conflict, complemented by specific political economy analysis of reforms in critical sectors, as needed.  Galvanize broad public support for reform, which would help to enhance stakeholders’ capacity to raise awareness and gradually build ownership of the reform agenda, thus helping overcome resistance to change from vested interests.  Selectively and carefully sequence first-order policy reforms (based on the political economy analysis) when designing World Bank Group strategy, considering capacity and other constraints inherent in the transition period.  Build government ownership and capacity on how to roll out the reform agenda. 23. The CLR development outcome rating for Egypt during FY06–14 is unsatisfactory. As in Tunisia, this CLR is unusual because it covers an extended period that included significant political upheaval, and it deals with two distinct periods. The first period covers January 2006 to January 2011 and the pro-growth country strategy that was in operation through the food and financial crises. After the January 2011 revolution, government commitment to the World Bank Group program waned, and a modest ISN was put in place that envisioned a development policy loan for economic management (which was never developed), and an emergency labor-intensive project. IFC activities were included in the framework in the main areas of IFC interventions (PPPs, for example). Generally, no outcome indicators linked directly to IFC activities, which made their contributions more difficult to evaluate, although IFC broadly contributed to key World Bank Group 179 APPENDIX I REGIONAL RESULTS AND PERFORMANCE program objectives and adapted to changing circumstances and priorities. However, IFC lost traction on PPPs because of political uncertainties in Egypt during the ISN period and government reluctance to commit to reforms. Negative perception of the private sector in the post-revolution period resulted in several reversals of public asset privatizations and legal challenges to land allocation or to concessions (in infrastructure, for example), which hindered IFC’s ability to support job creation and developmental projects. Overall, IEG found that 11 of the 13 country assistance strategies and ISN objectives were partially achieved or not achieved. Lessons highlighted in IEG’s CLR review support flexibility in the type of volatile context encountered in Egypt at various stages, but they also warn against setting objectives too broadly. Figure 1. Middle East and North Africa Region: Percentage of Projects Rated MS+ on Outcome FY01–15 100 Bank-wide Middle East & North Africa 90 Non-Middle East & North Africa Regions 79 81 78 Percentage 75 77 76 80 75 74 72 73 71 70 70 80 70 74 70 71 71 60 65 53 63 63 62 63 59 60 50 Fiscal year, three-year roliing Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 180 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 1. Middle East and North Africa Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13–15) Middle East and North Africa Other Regions Bank-wide Number of Net commitments Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) projects (US$, millions) Global Practice Rated Rated Rated Rated Rated Rated Total Total Total Total Total Total MS+ MS+ MS+ MS+ MS+ MS+ number amount number amount number amount (%) (%) (%) (%) (%) (%) Agriculture 3 100 230 100 70 79 3,852 89 73 79 4,083 89 Education 11 45 512 37 69 78 6,064 89 80 74 6,576 85 Energy and Extractives 6 67 185 88 67 66 7,079 78 73 66 7,264 79 Environment and Natural Resources 5 100 60 100 45 58 1,550 80 50 62 1,610 81 Finance and Markets 2 100 46 100 32 72 4,955 96 34 74 5,001 96 Governance 2 50 14 28 53 57 4,693 74 55 56 4,706 74 Health, Nutrition, and Population 1 100 4 100 66 73 5,342 83 67 73 5,345 83 Macroeconomics and Fiscal Management 1 100 40 100 48 81 10,810 97 49 82 10,850 97 Poverty 1 0 5 0 6 83 1,170 99 7 71 1,175 99 Social Protection and Labor 6 83 282 99 34 91 6,137 98 40 90 6,419 98 Social, Urban, Rural, and Resilience 10 40 288 41 86 74 8,294 83 96 71 8,582 81 Trade and Competitiveness 0 n.a. 0 n.a. 17 65 1,015 54 17 65 1,015 54 Transport and ICT 2 50 179 60 76 80 9,460 89 78 79 9,639 89 Water 8 38 436 32 64 72 5,632 77 72 68 6,069 74 Total 58 60 2,280 61 733 73 76,053 87 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; total number refers to the total number of projects rated by IEG. 181 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 2. Middle East and North Africa Region: IEG Outcome Ratings of IFC Investment Projects, FY13–15 Middle East and North Africa IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 3 33 113 66 34 38 904 56 and Funds Financial Markets 4 50 130 21 70 59 1,705 62 Infrastructure and Natural 6 50 173 72 41 63 1,518 79 Resources Manufacturing, Agribusiness, 8 50 289 57 84 51 2,105 59 and Services Total 21 48 705 56 229 54 6,232 64 Source: IEG data. Note: MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. Middle East and North Africa Region: IEG Outcome Ratings of IFC Advisory Projects, FY13–15 Middle East and North IFC overall Africa IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 4 75 43 70 Cross-Cutting Advisory Solutions 0 n.a. 4 75 Cross-Cutting Advisory Solutions: Public-Private 2 0 14 43 Partnerships Environment, Social, and Governance 0 n.a. 1 100 Financial Markets 1 100 5 40 Financial Institutions Group 3 33 17 47 Infrastructure and National Resources 0 n.a. 2 100 Investment Climate 5 60 26 77 Manufacturing, Agribusiness, and Services 0 n.a. 3 33 Public-Private Partnerships 1 0 17 47 Sustainable Business Advisory 5 100 38 66 Telecom, Media, Technology, and Venture Capital 0 n.a. 1 0 Trade and Competitiveness 0 n.a. 8 50 Total 21 62 179 61 Source: IEG data. Note: MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Please note that during FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public-Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. 182 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 4. Middle East and North Africa Region: MIGA Development Outcome Ratings by Sector, FY13–15 Middle East and North MIGA overall Africa Sector Total Total Rated S+ (%) Rated S+ (%) number number Agribusiness 0 n.a. 1 0 Financial 0 n.a. 8 38 Infrastructure 1 100 9 44 Manufacturing 1 100 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 0 n.a. 2 0 Services 0 n.a. 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 2 100 34 47 Source: IEG data. Note: S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. 183 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 5. Middle East and North Africa Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank IFC IEG XPSR IFC IEG PCR projects ratings ratings Total Rated Total Rated Total Rated Country or economy numbe MS+ numbe MS+ numbe MS+ r (%) r (%) r (%) Algeria 1 100 Egypt, Arab Republic 5 80 2 0 Iran, Islamic Republic 1 0 Iraq 11 55 1 0 IBRD Jordan 9 56 3 33 Lebanon 2 50 2 50 Morocco 4 75 1 0 Syrian Arab Republic 1 0 Tunisia 7 43 1 0 1 100 IBRD total 39 56 9 33 4 25 Djibouti 2 0 IDA Yemen, Republic of 11 64 1 0 4 50 IDA total 11 55 1 0 4 50 Kuwait 1 100 Middle East and North Africa Region 1 100 2 50 1 100 Other Saudi Arabia 1 100 West Bank and Gaza 5 100 2 50 3 100 Other total 6 100 5 60 5 100 TOTAL 58 60 15 40 13 62 Source: IEG data. Note: The data excludes projects not mapped to a Region. MS+ = moderately satisfactory or better (World Bank rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number refers to the total number of projects rated by IEG. 184 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 6. Middle East and North Africa Region: IEG Ratings of Project M&E Quality, FY13–15 Middle East and North Other Regions Bank-wide Africa Rated Rated Rated Total number substantial Total number substantial Total number substantial (%) (%) (%) IBRD 39 21 297 35 336 33 Blend 0 n.a. 107 26 107 26 IDA 13 15 296 33 309 32 Other 6 83 29 34 35 43 Total 58 26 729 33 787 32 Source: IEG data. Note: The data excludes projects not mapped to a Region. M&E = monitoring and evaluation; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Table 7. Middle East and North Africa Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 World Bank IFC MIGA CASCR/CLR Outcome FY of review Country performance performance performance review period rating rating rating rating Djibouti FY09–13 MU MS 2014 Morocco FY10–13 MS Good Egypt, Arab Republic 2016 of FY06–14 U Fair Tunisia FY10–14 MS Good Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory. 185 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 8. Middle East and North Africa Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16 2012 2013 2014 2015 2016 % of % of % of % of % of $, $, $, $, $, regi regi regi regi regi Global Practice millio on millio on millio on millio on millio on ns ns ns ns ns total total total total total Agriculture 3 0 203 10 0 0 0 0 7 0 Education 6 0 166 8 0 0 3 0 70 1 Energy and Extractives 445 29 591 29 61 2 1,025 29 1,250 24 Environment and Natural 0 0 0 0 300 11 15 0 300 6 Resources Finance and Markets 100 7 100 5 720 26 550 16 0 0 Governance 16 1 5 0 205 7 0 0 200 4 Health, Nutrition, and 0 0 7 0 10 0 100 3 0 0 Population Macroeconomics and Fiscal 250 17 500 24 500 18 0 0 1,700 33 Management Poverty 0 0 0 0 0 0 0 0 0 0 Social Protection and Labor 305 20 155 8 150 5 645 18 4 0 Social, Urban, Rural, and 388 26 130 6 14 0 480 14 550 11 Resilience Trade and Competitiveness 0 0 160 8 52 2 200 6 0 0 Transport and ICT 0 0 40 2 591 21 0 0 400 8 Water 0 0 0 0 185 7 474 14 700 14 % of % of % of % of % of total total total total total $, $, $, $, $, IBR IBR IBR IBR IBR Region millio millio millio millio millio D D D D D ns ns ns ns ns and and and and and IDA IDA IDA IDA IDA Middle East and North Africa 1,513 4 2,058 7 2,788 7 3,492 8 5,181 11 Region 33,74 29,49 37,89 39,00 40,71 Other Regions 96 93 93 92 89 3 0 4 3 8 35,25 31,54 40,68 42,49 45,89 Total 6 7 1 5 9 Source: World Bank Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. South Asia Regional Challenges and Evaluation Findings from World Bank Group Operations in the Region 1. According to the 2016 South Asia Regional Update, the Region can potentially transition from having the largest number of poor people to a dynamic middle-income area by building on its strengths and overcoming barriers to economic cooperation. Key growth drivers in the region include demographic 186 APPENDIX I REGIONAL RESULTS AND PERFORMANCE momentum, urbanization, private sector development, and governance. The South Asia Region has had one of the highest growth rates in the world in recent years (almost 7 percent in 2014 compared to 2.5 percent globally), which led to declining poverty and impressive gains in human development—the bottom 40 percent fared better than average in all countries in the Region between 2007-2012, except for India. However, overall prosperity was less widely shared in South Asia than in most other regions, and factors such as multiple forms of social and economic exclusion, and susceptibility to natural disasters held the region back. These factors can potentially impede growth and progress on poverty alleviation. The World Bank Group strategy for the region is built on mutually reinforcing pillars to support inclusive growth in South Asia: enhanced support for private sector development, greater social and financial inclusion, and strengthened governance and security (World Bank 2016j). 2. The World Bank’s new lending commitments in the Region peaked in FY14 at $10.5 billion, representing 26 percent of all World Bank lending in FY14. Commitments returned to a more typical 18 percent in FY15 at $7.9 billion, and in FY16 were at $8.4 billion (table 8). Almost 60 percent of all FY16 commitments are concentrated under the following three Global Practices: Energy and Extractives ($2.13 billion, 26 percent); Water ($1.84 billion, 22 percent); and Macroeconomics and Fiscal Management ($0.92 billion, 11 percent). 3. IEG rated the performance of 82 percent of World Bank operations in South Asia as moderately satisfactory or better (MS+), which is better than the World Bank as a whole (72 percent MS+). South Asia performed the best of all Regions during FY13–15, and the difference between outcome ratings (MS+) for projects in South Asia compared with all other projects is statistically significant (figure 1). 4. IEG rated all 27 projects under five Global Practices (representing 43 percent of all rated projects in the Region FY13–15) as MS+: Agriculture (12 projects), Education (seven projects), Social Protection and Labor (five projects), Environment (two projects), and Macroeconomics and Fiscal Management (one project). Performance is marginally below the World Bank average for three Global Practices: five of eight projects (63 percent) for Energy and Extractives are rated MS+, as are six of nine projects (67 percent) for Finance and Markets and two of four projects (50 percent) for Governance (table 1). 5. IEG rated 78 percent of projects across five IDA countries and 79 percent of projects across two blend countries as MS+ during FY13–15, which is slightly below the Region average (table 5). For India (the only IBRD country included in the 187 APPENDIX I REGIONAL RESULTS AND PERFORMANCE analysis), 21 of 22 projects rated were MS+ (91 percent). Within IDA, all seven projects rated for Nepal and all five for Bhutan were rated MS+. 6. Consistent with outcome ratings, South Asia also ranked highest in monitoring and evaluation (M&E) quality in FY13–15 (43 percent of projects rated substantial or above), which was well above the World Bank average (32 percent substantial or above) and represented a statistically significant lead compared with Africa and Latin America and the Caribbean (table 6). 7. IEG produced two project performance assessment reports (PPARs) in the region. The Nepal Education for All Project was rated moderately unsatisfactory (MU) for both development outcome and World Bank performance. Lessons from the PPAR emphasize the need to enhance learning input quality (such as teachers and instructional and learning materials) to improve learning outcomes, have a clear strategy for local capacity development for a decentralized service provision model, and clarify roles and responsibilities at various levels (central, school district, and local) for a decentralized model. The project operated at a total cost of $813 million, of which $703 million was government funding, and $110 million (including $60 million additional financing) was the IDA contribution. The project had three objectives: ensure access and equity in primary education, improve the efficiency and institutional capacity of primary education, and enhance the quality and relevance of basic primary education for children and illiterate adults. At closing, the project had met its access and equity targets (for example, realizing gender parity). Other initiatives, such as increased parental awareness and greater community involvement in education, had been implemented, which contributed to reduced repetition and drop-out rates (school efficiency), though attendance was still a problem, particularly among poorer students. However, evidence of improved learning outcomes was scant, as was evidence to support enhanced institutional capacity, particularly at the local level. The PPAR found that despite positive efforts in project supervision, the World Bank underestimated weaknesses in local capacity and thus mis-calibrated the capacity-building effort required for project implementation. The PPAR is IEG’s first evaluation of an Education for All Initiative financed through a sectorwide approach. 8. IEG produced a PPAR for the Northeast Emergency Irrigated Agriculture Project in Sri Lanka and rated the project MU for development outcome. The project objective was to help conflict-affected communities reestablish at least subsistence- level production and community-based services, and to develop some capacity for sustainable social and economic reintegration. The PPAR assessed the objective as relevant, though it notes that human and social capacity development could have had more emphasis based on the argument that improvements in capacity could have 188 APPENDIX I REGIONAL RESULTS AND PERFORMANCE contributed to better and more sustainable outcomes. IEG also rated the World Bank performance as MU based on the evaluation finding of too much emphasis on hardware rehabilitation and construction at the cost of addressing what it refers to as “fractured societal dynamics.” 9. IEG validated 25 IFC investment projects in the Region during FY13–15 and assigned ratings of mostly successful or higher to 15 projects (60 percent), which was above the IFC average (54 percent). The Region’s success rate by investment commitments was 76 percent, also above the IFC average success rate of 64 percent (table 2) 10. IEG reviews of IFC investments in the power sector, a priority sector in the region, show that power generation projects benefit from conservative tariff assumptions. Associated lessons include the need for contract and financing structures to be flexible enough to absorb intermittent pressure on cash flow that might arise from price fluctuations. The sophistication of IFC investees in this region is highly variable, so environmental and social safeguard work should reflect that variability. A key lesson from an IEG review of such an investment is that IFC should have stricter environmental and social reporting requirements for blue chip companies, especially when IFC has limited leverage in transactions. 11. IEG assigned a mostly successful or higher rating to 18 of the 29 IFC Advisory Services projects reviewed during FY13–15 (62 percent), which was slightly higher than the overall IFC average (61 percent) for Advisory Services projects (table 3). 12. Projects addressing energy efficiency complement investments in power generation. An IEG review of such an initiative in Bangladesh found benefit in focusing on cost competitiveness to demonstrate to clients the business case (based on actual data collection) for undertaking energy efficiency measures. The lesson suggests starting small to let firms experience the benefits from energy efficiency measures and to include local firms in showcasing results. IFC Advisory Services projects engage the public sector typically through its Trade & Competitiveness (previously Investment Climate) and Public-Private Partnerships business lines. An IEG review of such an initiative in a fragile state shows that subproject success depended on the counterparts’ strength. Consequently, a phased approach is more effective in such a risky environment, and build counterpart capacity may have to occur before pursuing other results. 13. There were no Multilateral Investment Guarantee Agency (MIGA) investment projects validated by IEG during FY13–15. To account for MIGA’s entire evaluated portfolio, this report takes into account projects that received a positive, 189 APPENDIX I REGIONAL RESULTS AND PERFORMANCE negative and a No Opinion Possible (NOP) rating. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Findings from Thematic, Global, and Corporate Evaluations 14. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence (IEG 2016q) included Pakistan and Sri Lanka. It notes that World Bank Group engagement and interest in conflict-affected regions of Pakistan, where systemic failures of governance and erosion of state legitimacy raises the risk of violence, varied considerably over time—a heightened focus on specific fragility, conflict, and violence projects and engagement was evident only since 2010. One of the four pillars of the 2010–13 country partnership strategy (CPS) was to improve security and reduce the risk of conflict, focusing on the need to respond to the most vulnerable and marginalized populations in fragility, conflict, and violence situations in the country, and emphasize the lack of employment opportunities and inadequate livelihoods. However, the main causes of fragility were at the heart of the country’s developmental challenges and often beyond the World Bank’s reach, sometimes even in initiating dialogue on sensitive topics such as rents distribution and religious extremism. 15. In reflecting on the link between national and regional social development and community-driven development projects in Pakistan, the evaluation notes that the lack of integration and strategy with how the investments worked in fragile regions prevented the World Bank program from achieving better results. Much of these projects’ focus was on delivering commitments instead of identifying expected changes and using them to determine a way forward. The country team stressed that in the rural areas of northern Pakistan, projects cannot hope to shift the value system simply through the provision of assets and services. Entry points for change exist, but they need to emerge through persistent engagement and committed partnerships. 16. The evaluation also notes that supporting the education sector to deal with issues of fragility is important, particularly relating to psychosocial issues among children, providing safe spaces for integration and community cohesion to develop, countering radicalization, and dealing with the problem of so-called lost generations in more protracted conflicts. The experience in Pakistan showed that even with small regional commitments and relatively high cost, implementing nationwide education programs in fragile and violent environments (adjusted to local realities) can deliver important developmental outcomes and contribute to progress on conflict mitigation. 190 APPENDIX I REGIONAL RESULTS AND PERFORMANCE 17. In Sri Lanka, where subnational conflicts with a regional or identity marker drive conflict, World Bank strategy documents examined the impact of conflict on development to varying extents and from different vantage points. The 2008 country assistance strategy (CAS) underscored the need for projects to remain conflict- sensitive and avoid inadvertently feeding into ethnic tensions. The CAS also warned that the World Bank had to manage reputational risks deftly to avoid the perception that it is a biased development partner in a highly sensitive region. The strategy introduced a conflict filter at the project level in the concept, design, and implementation stages. The CPS for 2012–16 has less emphasis on conflict than earlier strategies, focusing instead on the economic transition and how the World Bank could support growth. The CPS does not unpack or make explicit the sociopolitical dynamics of Sri Lanka after the war, and without this level of analysis, it does not make a clear link between efforts to strengthen economic growth and the need to sustain a durable peace. 18. The national education program in Sri Lanka incorporated features to address fragility by expanding project activities to the northern regions and introducing initiatives such as psychosocial support programs, conflict engagement models, multiethnic classes, development of multiethnic curriculum materials, and multiethnic school link programs. Community-driven development was a mechanism for delivering support to marginal communities (the Second North-East Irrigated Agriculture Project and the Community Livelihoods in Conflict-Affected Areas Project). The agriculture project compiled district social baseline profiles that became analytical insights to work with conflict-affected communities and laid important foundations for the various World Bank projects that followed. A suite of projects delivered in the northeast responded to the needs of post-conflict environments: cash transfers, reconstruction support, housing or shelter, community-based infrastructure, education support to get children back in schools, and capacity development programs to reengage regional authorities. Annex A provides additional cross-regional findings. 19. Managing Environmental and Social Risks in Development Policy Financing includes a project in the Punjab to support its assessment of the application of World Bank Operational Policy elements governing development policy financing (OP8.60) related to implementation of the policy’s environmental and social risk management requirements to identify lessons and good practices. The review presents the Punjab Irrigation Sector Development Policy Loan series as a good example of environmental and social risk screening. The project’s program document contains detailed tables in the annex that assess the environmental consequences of each policy area during the series, noting mitigation measures for identified risks. For example, the series supported increases in the maintenance and repair budgets of 191 APPENDIX I REGIONAL RESULTS AND PERFORMANCE irrigation works. The document noted that better asset maintenance could potentially increase water efficiency and reduce waterlogging and salinization, but also that increased construction activities could have local environmental effects and risks from changes in water flows, particularly to environmentally sensitive wetlands. The operation supported a mitigation measure that prepared environmental guidelines for establishing environmental safeguards for irrigation activities (IEG 2015i). Annex A provides additional cross-regional findings. 20. Working with a sample of World Bank and IFC projects from across the Regions, IEG’s 2016 learning product World Bank Group Support for Housing Finance sought to generate knowledge and identify lessons regarding support for housing finance. In India it found that the banks were prohibited from providing mortgage loans until 1998. To fill the gap prior to 1998, Housing finance companies (HFCs) were created as providers of mortgage finance. IFC invested in the first HFC and in others as they were established. The HFCs showed that they could lend responsibly and profitably by using highly selective credit criteria and personal guarantees to reduce risk, which compensated for the lack of foreclosure legislation. The HFCs in India were more resilient to crises because they had diverse funding sources and adequate, prudent regulations, and they could attract term deposits, borrow from a liquidity facility, and securitize mortgage loans. In response to legislative and market changes, IFC supported several HFCs to retain skilled loan officers, and to establish stringent risk management and pricing, high underwriting standards, and diverse loan products and credit assessment approaches based on local knowledge for different customer segments (both payroll and self-employed). The use of technology and lack of outsourcing collections enabled an IFC-supported HFC to speed up decision making, maintain a high level of customer services, and closely manage collections (IEG 2016s). Annex A provides additional cross-regional findings. 21. Noting the theme of this year’s Results and Performance of the World Bank Group (Management for Results), findings from IEG’s evaluation The Poverty Focus of Country Programs: Lessons from World Bank Experience are worth highlighting. The evaluation notes that the World Bank could have devoted more attention to the challenge of poverty monitoring between the five-year survey rounds, especially for poverty program impact monitoring, short-term interventions, and targeting. It assessed the data quality, diagnosis, focus of country program design, and implementation, and the feedback loops between these four links in the causal chain during the past two decades. The evaluation cites Bangladesh as a good example of how these elements worked reasonably well in World Bank contributions to Bangladesh’s poverty reduction and how it could have worked better. World Bank support to various rounds of the Household Income and Expenditure Survey 192 APPENDIX I REGIONAL RESULTS AND PERFORMANCE (conducted every five years) during a 20-year period created a strong database and effective partnership, but the engagement could have expanded earlier to include other key surveys (such as labor force surveys) and could have benefited from more capacity building at the Bangladesh Bureau of Statistics. Poverty assessments closely followed the survey rounds with strong analytic and research links and consistent messages, though a programmatic approach might have provided more timely inputs into strategy formulation (IEG 2015j). 22. It is also useful to refer to IEG’s evaluation World Bank Group Support to Electricity Access, FY2000–2014: An Independent Evaluation and its evaluation World Bank Support to Early Childhood Development: An Independent Evaluation. Along with Sub-Saharan Africa (591 million), South Asia (379 million) has the largest share of the 1.1 million people with no access to electricity or with inadequate or unreliable service. The Electricity Access evaluation highlights the relative success of the World Bank Group’s Bangladesh engagement supporting off-grid capacity. That experience shows that off-grid solutions—mainly solar home systems, and minigrids and microgrids—are a fast way to provide energy services to rural and remote areas. However, the World Bank Group support to off-grid electrification in the past 15 years represents only 1.5 percent of the electricity portfolio during FY00–14 (IEG 2015q). 23. IEG’s early childhood development (ECD) evaluation highlights the misalignment between country needs in ECD and World Bank engagement (IEG 2015s). Similar to electricity access, the need is greatest in South Asia and Sub- Saharan Africa (for example, in Bangladesh, between one third and one half of children aged less than five suffer from stunting), but World Bank Group engagement was generally weak. Bangladesh had little improvement in nutritional status during 2004–10. More recently, the World Bank Social Protection and Labor team conducted pilots and evaluations of cash transfer systems that improved the nutritional status of children in Bangladesh. In Nepal, the World Bank was actively involved in country dialogue on nutrition and its analytical work focuses heavily on nutrition. In this instance the World Bank recommended addressing malnutrition through a multisector approach, and the government responded by approving a multisector nutrition plan and establishing appropriate coordinating institutions. The evaluation notes that the World Bank Group typically engaged on a fragmented, sector-by-sector basis that is inadequate to address critical multisectoral issues such as nutrition. 193 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Evaluation Findings from Country Program Performance in the Region 24. Analysis undertaken for Results and Performance of the World Bank Group 2016 shows that for the five countries—Bhutan, India, Pakistan, Bangladesh, and Sri Lanka—in the South Asia Region for which IEG validated two country strategies FY07-16, the rating for overall development remained moderately satisfactory (MS) for Bhutan and India, improved from MU to MS for Bangladesh, and remained MU for Pakistan. The Sri Lanka country strategy was rated three times, and its overall development outcome rating improved from MU to MS and remained at that level after IEG’s third validation. 25. IEG completed two completion and learning reviews (CLRs) for the region in FY16. The CLR development outcome rating for both the CPS FY11–15 for Bangladesh and the CPS FY13–16 for Sri Lanka is MS. World Bank performance is rated good for Bangladesh and fair for Sri Lanka (table 7). 26. IEG’s review of the Bangladesh CLR notes the remarkable progress the country made in reaching lower-middle-income status in 2015, showing progress in poverty reduction, gross domestic product growth, exports, and human development. When the CAS was in development, the government had already placed poverty and inequity at the center of its development strategy, which ensured CAS ownership and buy-in. The CAS sought to support government ambitions through accelerated, sustainable, and inclusive growth, supported by stronger governance at the central and local levels. The CAS was organized around four objectives: consolidating human development gains, accelerating growth, reducing environmental degradation and vulnerability, and strengthening governance. IEG’s review notes that the CAS may have been too broad in its scope, and also refers to the cancelation of it calls a cornerstone project (the Padma Bridge Project, planned in cooperation with the Asian Development Bank and the Japan International Cooperation Agency), which had a detrimental effect on the portfolio as a whole. Plans for several new operations to replace the Padma Bridge Project were overambitious given staff constraints and implementation challenges. The CAS progress report tightened the World Bank’s program, which led to dropping some outcomes (from 16 to 11) because of a lack of progress. The CAS eventually showed progress in many areas, including health outcomes, social safety nets, primary and secondary education completion rates, agricultural productivity, and power capacity. However, educational quality did not improve as envisioned, governance improvement was only modest, and the environment for private sector investment remained challenging despite strong IFC commitments of nearly $900 million in 29 projects. The outcome indicator for 2.1 (improved environment for private sector investment, Doing Business Ranking) was not achieved. Unfortunately, this outcome’s 194 APPENDIX I REGIONAL RESULTS AND PERFORMANCE indicator (and the results framework in general) did not integrate IFC, which means that attribution for IFC input cannot be measured. It is useful to note that IFC’s support and signaling effect due to the investment climate does not influence the Doing Business Ranking. The CLR highlights several lessons, including the need for selectivity, and that results-based lending can be effective in advancing policy change and in increasing disbursements. 27. IEG’s review of the CLR for the CPS for Sri Lanka FY13–16 found the World Bank Group’s strategy was relevant to the government’s development vision through its focus on three engagement areas: facilitating sustained private and public investment, supporting structural shifts in the economy, and improving living standards and social inclusion. A fourth focus area was added at the 2014 CPS progress report stage in response to client priorities (improving resilience to climate and disaster risks). Apart the relevance of strategic focus, IEG’s review finds that the World Bank Group could have exercised greater selectivity in setting its development objectives. The CPS included several objectives not backed by sufficient interventions to attribute meaningful results (on public investment and nutrition, for example). Adjustments made through the CPS instilled greater realism to the program, but one important operation (the Investment Policy Reform Development Policy Loan) was delayed, and three operations were dropped (including the Legal and Judicial Reforms Project), indicating weak ownership for policy reforms. However, the review found that IFC’s investments and advisory services complemented the World Bank’s role on investment climate, access to finance, and climate resilience. The overarching lessons from IEG’s review are the need for selectivity, clarity of World Bank Group objectives and outcomes, and a realistic results framework with an explicit and credible results chain. 195 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Figure 1. South Asia Region: Percentage of Projects rated MS+ on Outcome, FY01–15 Bank-wide 100 South Asia 95 Non-South Asia Regions 90 85 82 79 79 80 78 77 78 80 Percentage 76 75 75 74 72 72 75 78 78 77 70 74 74 73 75 71 71 71 65 70 69 69 60 55 50 Fiscal year, three-year rolling Source: IEG data. Note: MS+ = moderately satisfactory or better. Number is the percentage of project outcomes rated moderately satisfactory or better. Projects not mapped to any region are excluded. 196 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 1. South Asia Region: IEG Project Development Outcome Ratings by Global Practice (Closing FY13 –15) South Asia Other Regions Bank-wide Net Net Net Number of Number of Number of commitments commitments commitments projects projects projects (US$, millions) (US$, millions) (US$, millions) Global Practice Rated Rated Total Rated Total Rated Total Rated Total Total Total Rated MS+ MS+ number MS+ (%) amount MS+ (%) number MS+ (%) amount number amount MS+ (%) (%) (%) Agriculture 12 100 1,352 100 61 75 2,731 84 73 79 4,083 89 Education 7 100 1,609 100 73 71 4,967 80 80 74 6,576 85 Energy and Extractives 8 63 2,624 91 65 66 4,639 71 73 66 7,264 79 Environment and Natural Resources 2 100 46 100 48 60 1,564 80 50 62 1,610 81 Finance and Markets 9 67 1,620 97 25 76 3,381 95 34 74 5,001 96 Governance 4 50 280 77 51 57 4,427 74 55 56 4,706 74 Health, Nutrition, and Population 7 71 913 58 60 73 4,432 88 67 73 5,345 83 Macroeconomics and Fiscal Management 1 100 36 100 48 81 10,814 97 49 82 10,850 97 Poverty 0 n.a. 0 n.a. 7 71 1,175 99 7 71 1,175 99 Social Protection and Labor 5 100 419 100 35 89 6,000 98 40 90 6,419 98 Social, Urban, Rural, and Resilience 5 80 851 99 91 70 7,731 79 96 71 8,582 81 Trade and Competitiveness 0 n.a. 0 n.a. 17 65 1,015 54 17 65 1,015 54 Transport and ICT 7 86 1,142 100 71 79 8,497 87 78 79 9,639 89 Water 10 80 1,518 94 62 66 4,551 67 72 68 6,069 74 Total 77 82 12,410 93 714 71 65,924 85 791 72 78,333 86 Source: IEG data. Note: The data excludes projects not mapped to a Region or Global Practice. ICT = information and communication technologies; MS+ = moderately satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. 197 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 2. South Asia Region: IEG Outcome Ratings of IFC Investment Projects, FY13–15 South Asia IFC overall Number of Net commitments Number of Net commitments projects (US$, millions) projects (US$, millions) IFC cluster Total Rated Total Rated Total Rated Total Rated number MS+ (%) amount MS+ (%) number MS+ (%) amount MS+ (%) Telecoms, Media, Technology, Venture Capital, 5 20 115 73 34 38 904 56 and Funds Financial Markets 4 75 42 83 70 59 1,705 62 Infrastructure and Natural 4 50 62 66 41 63 1,518 79 Resources Manufacturing, Agribusiness, 12 75 394 77 84 51 2,105 59 and Services Total 25 60 613 76 229 54 6,232 64 Source: IEG data. Note: MS+ = moderately successful or better; total number refers to the total number of projects rated by IEG. Table 3. South Asia Region: IEG Outcome Ratings of IFC Advisory Projects, FY13–15 South Asia IFC overall IFC business line Total Rated MS+ Total Rated MS+ number (%) number (%) Access to Finance 5 60 43 70 Cross-Cutting Advisory Solutions 0 n.a. 4 75 Cross-Cutting Advisory Solutions: Public-Private 3 100 14 43 Partnerships Environment, Social, and Governance 0 n.a. 1 100 Financial Markets 2 50 5 40 Financial Institutions Group 5 80 17 47 Infrastructure and National Resources 0 n.a. 2 100 Investment Climate 2 100 26 77 Manufacturing, Agribusiness, and Services 1 0 3 33 Public-Private Partnerships 5 40 17 47 Sustainable Business Advisory 4 50 38 66 Telecom, Media, Technology, and Venture Capital 1 0 1 0 Trade and Competitiveness 1 100 8 50 Total 29 62 179 61 Source: IEG data. Note: MS+ = moderately successful or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Please note that during FY13 and FY14, only four IFC advisory business lines (Access to Finance, Investment Climate, Public-Private Partnerships and Sustainable Business Advisory) existed. Following the World Bank Group realignment, new business line categories were introduced in FY15. 198 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 4. South Asia Region: MIGA Development Outcome Ratings by Sector, FY13–15 South Asia MIGA overall Sector Total Total Rated S+ (%) Rated S+ (%) number number Agribusiness 0 n.a. 1 0 Financial 0 n.a. 8 38 Infrastructure 0 n.a. 9 44 Manufacturing 0 n.a. 7 86 Mining 0 n.a. 0 n.a. Oil & Gas 0 n.a. 2 0 Services 0 n.a. 7 43 Tourism 0 n.a. 0 n.a. Others 0 n.a. 0 n.a. Total 0 n.a. 34 47 Source: IEG data. Note: S+ = satisfactory or better; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Definition of No Opinion Possible (NOP): Rating given to Development effectiveness indicator due to lack of appropriate and sufficient performance data at the obligor, project or beneficiary level. The success rates for the FY13-15 period were calculated including projects rated as No Opinion Possible (NOP) in the total number of projects rated. Table 5. South Asia Region: IEG Development Outcome Ratings by Country, FY13–15 World Bank projects IFC IEG XPSR ratings IFC IEG PCR ratings Total Rated Total Rated Total Rated Country number MS+ (%) number MS+ (%) number MS+ (%) IBRD India 22 91 18 56 13 69 IBRD total 22 91 18 56 13 69 Pakistan 12 75 4 75 5 60 Blend Sri Lanka 7 86 2 50 4 75 Blend total 19 79 6 67 9 67 Afghanistan 14 79 1 100 2 100 Bangladesh 8 50 1 100 2 100 IDA Bhutan 5 100 2 50 Maldives 2 50 1 100 2 0 Nepal 7 100 2 50 1 0 IDA total 36 78 5 80 13 54 TOTAL 77 82 29 62 35 63 Source: IEG data. Note: The data excludes projects not mapped to a Region. MS+ = moderately satisfactory or better (World Bank rating) or moderately successful or better (IFC rating); PCR = project completion report; XPSR = Expanded Project Supervision Report; total number refers to the total number of projects rated by IEG. 199 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 6. South Asia Region: IEG Ratings of Project M&E Quality, FY13–15 South Asia Other Regions Bank-wide Rated Rated Rated Total number substantial Total number substantial Total number substantial (%) (%) (%) IBRD 22 50 314 32 336 33 Blend 20 40 87 23 107 26 IDA 34 41 275 31 309 32 Other 0 n.a. 35 43 35 43 Total 76 43 711 31 787 32 Source: IEG data. Note: The data excludes projects not mapped to a Region. M&E = monitoring and evaluation; n.a. = not applicable; total number refers to the total number of projects rated by IEG. Table 7. South Asia Region: IEG CASCR/CLR Reviews Outcome and Performance Ratings, FY14–16 CASCR/CLR World Bank IFC MIGA FY of Outcome Country review performance performance performance review rating period rating rating rating 2014 Pakistan FY10–14 MU Fair 2015 Bhutan FY11–14 MS Good Fair Bangladesh FY11–15 MS Good 2016 Sri Lanka FY13–16 MS Fair Source: IEG data. Note: CASCR = Country Assistance Strategy Completion Report; CLR = completion and learning review; FY = fiscal year; MS = moderately satisfactory; MU = moderately unsatisfactory. 200 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Table 8. South Asia Region: IBRD and IDA New Lending Commitments by Global Practice, FY12–16 2012 2013 2014 2015 2016 % of % of % of % of % of $, $, $, $, $, Global Practice millions region millions region millions region millions region millions region total total total total total Agriculture 1,991 31 226 5 381 4 968 12 550 7 Education 1,250 19 595 13 1,498 14 1,115 14 752 9 Energy and Extractives 1,212 19 221 5 2,869 27 176 2 2,134 26 Environment and Natural Resources 0 0 100 2 0 0 3 0 45 1 Finance and Markets 30 0 130 3 57 1 900 11 150 2 Governance 0 0 119 3 117 1 50 1 45 1 Health, Nutrition, and Population 192 3 855 19 100 1 36 0 200 2 Macroeconomics and Fiscal Management 0 0 36 1 450 4 520 7 920 11 Poverty 0 0 0 0 0 0 0 0 0 0 Social Protection and Labor 150 2 613 14 84 1 300 4 138 2 Social, Urban, Rural, and Resilience 592 9 831 19 1,433 14 2,081 26 630 8 Trade and Competitiveness 0 0 0 0 200 2 22 0 230 3 Transport and ICT 505 8 749 17 2,142 20 950 12 735 9 Water 523 8 0 0 1,204 11 739 9 1,835 22 % of % of % of % of % of $, total $, total $, total $, total $, total millions IBRD millions IBRD millions IBRD millions IBRD millions IBRD and IDA and IDA and IDA and IDA and IDA South Asia Region 6,446 18 4,474 14 10,535 26 7,860 18 8,363 18 Other Regions 28,811 82 27,073 86 30,147 74 34,634 82 37,536 82 Total 35,256 31,547 40,681 42,495 45,899 Source: World Bank Business Intelligence as of September 26, 2016. Note: ICT = information and communication technologies. 201 APPENDIX I REGIONAL RESULTS AND PERFORMANCE Cross-Regional Findings 1. IEG’s evaluation of World Bank Group Engagement in Situations of Fragility, Conflict, and Violence focused on such situations, which are predominantly, though not exclusively, found in middle-income countries. The evaluation found the World Bank Group and governments may have differing views on how and when to tackle the underlying issues, which makes providing effective support particularly challenging (IEG 2016q). The World Bank Group’s comparative advantage in such circumstances is in supporting countries in tackling long-term development challenges, including early engagement, sustained presence, and continuous dialogue. The World Bank was adept at adjusting its strategies and analytical support to such situations, but its limited choice of instruments constrained its operational response. Furthermore, institutional and staff incentives to engage in situations of fragility, conflict, and violence are not yet commensurate with the spirit of its strategic approach. The community-driven development model often provided the World Bank Group with a useful point of entry in conflict-affected areas, establishing a presence and contextual learning to support efforts that are more ambitious once peace is restored. The World Bank Group can further enhance its impact in these situations by exploring opportunities beyond supporting livelihoods in conflict-affected communities, including supporting private sector development, using its expertise in public financial management more effectively, and improving its M&E frameworks. 2. The evaluation makes recommendations at the strategic and operational levels. At the strategic level, it recommends a review of the institutional setup and interaction of various World Bank Group units dealing with fragility, conflict, and violence matters to ensure cross-cutting thinking on strategy and policy. It also recommends developing a new, multidimensional, publicly disclosed system of markers and flags for monitoring and measuring fragility (including in countries that are not in fragile and conflict-affected situations) and the systematic integration of fragility assessments as part of the systematic country diagnostic and country partnership framework (CPF) process. The final, strategic-level recommendation is to scale up regional presence and outlook, such as through regional projects and provision of joint advisory and analytic services across relevant countries. At the operational level, the evaluation recommends development of new financial mechanisms or fast-response facilities as necessary (for example, cofinancing small pilot programs in areas affected by fragility, conflict, and violence, with potential for scaling-up). The evaluation also recommends expanding sector interventions beyond the community-driven development model, and improving coordination and information sharing within the World Bank Group. Another operational-level 202 APPENDIX I REGIONAL RESULTS AND PERFORMANCE recommendation is to develop institutional incentives for collaboration and strategic thematic guidance on partnering (particularly with the UN, but also with other partners) and to introduce systematic communication and staff exchange programs to increase the level of collaboration. 3. The evaluation classified the cases reviewed under four categories of stress factors that raise the risk of violence:  Subnational conflicts with a regional or ethnic identity market  Violence related to organized crime and drug trafficking  Systemic failures of governance and erosion of state  Overwhelming 4. Managing Environmental and Social Risks in Development Policy Financing assesses the application of the elements of the World Bank Operational Policy governing development policy financing (DPF)—OP8.60—related to implementation of the policy’s environmental and social risk management requirements to identify lessons and good practices. IEG found that World Bank policy is vague and lacks clear definition of key concepts. Guidance and training materials vary in quality, utility, and relevance. Furthermore, pressure to deliver operations quickly, and the lack of a formal role for environmental or social specialists provides disincentives for task leaders in setting priorities for managing environmental and social risks in development policy operations (DPOs). The review is positive about the inclusion of environmental and social themes and risks in the World Bank’s new tools and processes, such as the Systematic Operations Risk Rating Tool (SORT), the SCD, and the CPF. IEG also identifies several lessons, including the value of clear and updated guidance for task teams, early and consistent inclusion of environmental and social considerations, and strong messages from management of World Bank Group institutions about the importance of supported projects’ environmental and social aspects (IEG 2015i). 5. Working with a sample of World Bank and IFC projects from across the regions, IEG’s 2016 learning product World Bank Group Support for Housing Finance sought to generate knowledge and identify lessons regarding support for housing finance. World Bank Group support to housing finance represents a small portion of the entire portfolio FY05–15 (279 projects valued at $5.2 billion approved in 73 countries). Overall, IFC (investments and advisory services) accounted for the highest share of housing finance projects by number of projects (59 percent), and the World Bank (lending and nonlending) accounted for the highest share by commitment value (58 percent) because of a relatively larger average project size. 203 APPENDIX I REGIONAL RESULTS AND PERFORMANCE The volume of World Bank Group support to housing finance decreased post-crisis, particularly for IFC (IEG 2016s). 6. Key lessons identified in IEG’s review of housing finance include the following: • Commercial banks are logical partners in housing finance projects, but they need to be fully committed to mortgage lending. • Sustainability can be enhanced when there is a strong, well-timed, and sequenced link between lending and advisory services. • A needs assessment is fundamental to design interventions tailored to specific country needs. • When expanding affordable housing is a key objective of an intervention, it needs to be targeted, and the results adequately measured. • If there is no mortgage financing, the low-income segment should not be prioritized, but should be introduced gradually. 7. IEG’s 2016 learning product Lessons from Land Administration Projects: A Review of Project Performance Assessments synthesizes cross-cutting findings and lessons from IEG’s assessments of World Bank–supported projects dealing with often complex and politically fraught land administration issues. IEG compared World Bank Group experience across projects using different institutional structures to manage two important elements of a land administration system—registry, which records the rights to land, and the cadaster, which provides information on the location, boundaries, use, and values of land parcels. The comparison suggests that no single model is best for performing these functions. The review also emphasizes that making land tenure more secure is a process, not a single event, and the concept of land tenure and associated rights is highly context-specific, meaning there are no absolute, generally applicable standards for defining tenure security. Most of the projects reviewed did not explicitly target the poor or vulnerable groups, reflect social inclusion in their objectives, or sufficiently report on social impacts across projects. Experience shows that social impacts need to be monitored and not assumed, even when laws and procedures are nominally the same for all potential beneficiaries (IEG 2016j). 8. Transformational engagements are a crucial pillar of the World Bank Group’s current strategy. IEG’s 2016 learning product Supporting Transformational Change for Poverty Reduction and Shared Prosperity identified a sample of such engagements from 2000–14 and found that although it may be difficult to identify transformational engagements ex ante, some salient characteristics can be identified ex post (IEG 2016p). Most of the engagements supported fundamental change in a market, system, or behavior. Large-scale impacts were achieved through scaling up, 204 APPENDIX I REGIONAL RESULTS AND PERFORMANCE replication, or demonstration effects, and a significant degree of the intervention effects were sustained. IEG’s analysis suggests the World Bank Group can enhance the likelihood that interventions will catalyze faster development progress by taking a programmatic approach that addresses systemic and behavioral change based on a sound diagnosis of binding constraints. This does not necessarily mean more complexity in individual interventions, but rather a thorough understanding of the main constraints and issues. Adopting a programmatic approach can facilitate a plan of selective and sequenced interventions that can help address the constraints to a country or sector’s development. 9. The Quality of Results Frameworks in Development Policy Operations, a 2015 IEG learning product, presents the results of a review of the World Bank’s DPF portfolio based on a sample of operations from across the World Bank Regions. IEG found that World Bank projects mostly strike a good balance between breadth and specificity when defining lending objectives. The review finds the introduction of Strategic Country Diagnostics in planning may help strengthen DPF policy focus even further through the systematic identification of binding constraints. The review notes that building strong results frameworks depends heavily on identifying prior actions (actions that a country must meet to demonstrate its commitment to seeing through the policy reforms). In this respect, IEG found a wide variation in World Bank experience, from examples of excellence to instances in which prior actions were not credible or deep enough to trigger change. In many cases, IEG attributed weaknesses in prior actions to weak implementation of programmatic series instead of the original design and found examples of significant deterioration in the quality of prior actions because of not implementing triggers, to the extent that the World Bank likely would not have approved the final operations as stand-alone operations (IEG 2015l). 10. IEG’s 2015 evaluation World Bank Support to Early Childhood Development: An Independent Evaluation found that World Bank sector strategies in health, education, and social protection feature early childhood development (ECD). These sector- based structures tend to seek entry points for engagement with countries in ECD with the corresponding public sector authorities. IEG found that country strategies have limited discussion of the ECD interventions embedded in their programs, and suggests properly integrating ECD interventions into County Partnership Frameworks. According to the evaluation, a positive correlation exists between analytical work on ECD and subsequent successful World Bank operations (IEG 2015s). 205 Appendix J. IEG Note on Global Practices Equitable Growth, Finance, and Institutions Global Practice Cluster 1. The Equitable Growth, Finance, and Institutions (EFI) cluster consists of five global practices:  Finance and Markets: seeks to develop resilient, efficient, and transparent financial systems to help mobilize capital needed for investments in country development priorities.  Governance: develops innovative, integrated solutions to pernicious institutional problems using a problem-driven, diagnostic approach that combines knowledge of reform successes and failures with a keen understanding of institutional challenges and opportunities in developing countries.  Macroeconomics and Fiscal Management: provides integrative development strategies, policy-based lending, macro data, global perspectives, real-time policy analysis, country risk assessments, and innovative projection tools.  Poverty: seeks to deliver advice and integrated knowledge to identify key policies and multisectoral solutions that effectively reduce poverty and benefit the less well-off, and to help better understand the relationship between growth, poverty, and inequality.  Trade and Competitiveness: helps countries develop more dynamic and integrated economies by boosting trade, enhancing the investment climate, improving competiveness in sectors, and fostering innovation and entrepreneurship. 2. The Equitable Growth, Finance, and Institutions (EFI) cluster is the World Bank’s second largest Global Practice cluster by commitments. The cluster’s commitments in FY16 were $13.7 billion, which is 30 percent of total World Bank lending (table 1). In comparison, total FY16 commitments for the Sustainable Development cluster were almost $25 billion (54 percent of total World Bank commitments) and $7.6 billion for the Human Development cluster (16 percent of total World Bank lending). Performance Trends 206 APPENDIX J REGIONAL RESULTS AND PERFORMANCE 3. For the current review period, FY13–15, 69 percent of all projects in the Equitable Growth, Finance, and Institutions cluster were rated MS+, which is below the World Bank average of 72 percent (figure 3). 4. During the three-year period FY13–15, 62 percent of the investment projects in the EFI cluster were rated moderately satisfactory or better (MS), which compares to the 73 percent MS+ average rating for investment projects in the other two clusters in the same period. On a three-year rolling average basis since FY01, the percentage of EFI cluster investment projects rated MS is consistently below the average across the other two clusters (figure 1 panel b). 5. Development policy financing (DPF) performance within the cluster has traditionally been stronger when compared with the other two clusters. However, performance began to improve across the other clusters since FY09–11, and the average DPF performance for the other clusters is now marginally better than that in the EFI cluster—85 percent MS+ versus 82 percent MS+—during FY13–15 (figure 1 panel c). 6. Three-year rolling average ratings for IDA and IBRD projects in the cluster show IBRD projects consistently outperforming IDA projects, and the performance gap is widening (figure 2 panel a). During FY12–14, IBRD projects in the EFI cluster outperformed IDA projects 77 percent MS+ to 60 percent MS+. The latest three-year rolling average shows IBRD performing at 86 percent MS+ while IDA projects remain at a 61 percent satisfaction level. IBRD projects in the cluster also outperform the average ratings for IBRD projects across the other clusters (86 percent versus 71 percent MS+, respectively) during FY13–15 (figure 2 panel b). IDA projects perform well below the average ratings for IDA projects across the other clusters (61 percent versus 76 percent MS+, respectively) in the same period (figure 2 panel c). 7. Investment projects in three Global Practices were rated at or above the cluster average (69 percent MS) during FY13–15: Macroeconomics and Fiscal Management (82 percent), Finance and Markets (74 percent), and Poverty (71 percent). Macroeconomics and Fiscal Management performance during FY13–15 is above the corporate target for Global Practice performance of 75 percent MS+ by FY17. The Governance practice ranked the lowest of the five Global Practices with a rating of 56 percent MS+, and the gap between Governance and other Global Practices is statistically significant in FY13–15 (figure 3). 8. Disaggregated by World Bank Regions during FY13–15, EFI cluster projects were rated 85 percent MS+ for the Europe and Central Asia Region, and 81 percent MS+ for the Latin America and the Caribbean Region. These two high-achieving 207 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Regions outperformed the others: EFI projects in the Middle East and North Africa Region were rated 67 percent MS+, those in the South Asia Region were rated 64 percent MS+, and projects in both the Africa and East Asia Pacific Regions lagged at 59 percent and 58 percent MS+, respectively (figure 4). 9. IEG rated the EFI cluster’s monitoring and evaluation (M&E) quality for FY13– 15 as 36 percent substantial, which compares favorably with a 31 percent average rating across the other clusters. Within the cluster, the M&E rating for projects in the Macroeconomics and Fiscal Management and the Finance and Markets practices was, respectively, 46 percent and 44 percent substantial, and the M&E rating for projects in the Poverty practice was 43 percent. Other Global Practices in the cluster rated lower than the cluster average, with Trade and Competitiveness rated 28 percent substantial, and Governance rated 24 percent substantial (figure 5). Main Evaluation Findings and Lessons 10. This section summarizes key findings and lessons from IEG’s major evaluations, learning products, and project performance assessment reports (PPARs) completed during FY12–15 and relevant to the EFI cluster.1 Major Evaluations and Learning Products 11. IEG’s evaluation of the World Bank’s Group’s contributions to client countries’ capital market development spans all areas of capital market development as well as the use of capital market instruments in Bank Group support. It examines the relevance, effectiveness and sustainability of Bank Group interventions with particular reference to the period FY04-14. 12. The evaluation finds that the World Bank Group followed broadly correct strategic directions at critical points, especially, in emphasizing public goods aspects and venturing to develop new markets and asset classes. In terms of issuers and instrument, early emphasis on local currency government bond markets reflected global concerns in the wake of the Asian crisis. The Bank Group’s response was innovative, if only partially successful. Both the World Bank and IFC Treasuries also supported the development of local capital markets through their local currency bond issues. 1Findings and lessons related to IFC, MIGA, and public-private partnership operations relevant to the Equitable Growth, Finance, and Institutions (EFI) cluster were included in this section even though they may not be under the EFI cluster’s direct responsibility. 208 APPENDIX J REGIONAL RESULTS AND PERFORMANCE 13. IFC’s early support for emerging market asset classes, including contributions towards the building of investability indices, proved pioneering. Its later move away from support for developing stock markets towards private equity reflected its frontier role, although exits through public offerings were rare and returns mixed. In developing instruments such as asset-backed and mortgage-backed securities, the World Bank Group’s role was sometimes ahead of its time. On the investor side, IEG found little focus on asset management or capital market investment in most interventions in insurance and pensions, although this might have aided their sustainability. 14. The Bank Group’s convening power has been evident in contributions to global payments fora as well as in the development of Green Bond principles. Yet Bank Group use of capital markets instruments or project bonds for infrastructure financing in its own transactions was small; guarantees however were an important instrument. 15. Overall, coherence across areas of engagement was weak. Attempts to develop markets through Treasury bond issues could have had more sustained impact if linked to operational and advisory support. Investors could have been better linked to issuers and instruments, and limited use was made of underlying Financial Sector Assessment Program (FSAP) diagnostics. Program fragmentation partly reflected reliance on external or unusual financing sources. 16. IEG points out that ultimately the credibility and impact of this largely knowledge-based practice area rests on developing, maintaining, and disseminating information. If the World Bank Group is to develop the capacity to contribute as an innovator, there is clear scope for improvement in knowledge generation as well as knowledge management. IEG accordingly recommends that the World Bank Group:  Integrate capital market development within the World Bank Group across different areas; Enhance the use of the FSAP instrument to underpin capital markets interventions’ design.  Strengthen knowledge management and develop a frontier global knowledge program; and  Review funding sources and their impact upon program design. IEG’s learning product on housing finance particularly relates to the Finance and Markets Global Practice. The report notes that hundreds of millions of people live in inadequate conditions with little or no access to decent housing (IEG 2016s). 209 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Housing finance is crucial to the functioning of the housing market. Creating sustainable and effective housing finance requires the mobilization of long-term sources of funding for both rental and owner-occupied housing, which is often missing in emerging markets. In response, the World Bank Group’s approach evolved from using traditional stand-alone operations to the development of more strategic and comprehensive solutions, including supply-side interventions (such as reducing legal and regulatory constraints) and demand-side interventions (such as provision of guarantees and financing). The learning product aimed to provide lessons from experience about successful practices from World Bank Group housing finance projects. Analysis shows that the World Bank Group allocates resources primarily to countries with low mortgage penetration rates and with the most need. Despite its important role, the World Bank Group has limited resources to support housing finance, thus requiring a selective and strategic approach in the face of growing need. Furthermore, the dedicated staff and resources it deploys have both fallen below the pre-2008 financial crisis levels, creating challenges to achieve more with less. Summary findings from the housing finance learning product include the following:  Commercial banks willing to provide mortgage loans are logical partners in housing finance projects, but they need to be fully committed to mortgage lending.  Non-bank financial institutions have strategic weaknesses that endanger their long-term viability, and they might need to diversify funding sources, expand, or convert themselves into a bank.  Sustainability can be enhanced when there is a strong link with advisory services, but it has to be adequately timed.  Preconditions exist for the development of capital markets and primary market interventions, and if these preconditions are not in place, adequate sequencing, timely interventions, and committed sponsors can have compensating effects.  A needs assessment is fundamental to design interventions tailored to specific country requirements.  When expanding affordable housing is a key objective of an intervention, it needs to be targeted and the results adequately measured.  If there is no mortgage financing, the low-income segment should not be prioritized, but should be introduced gradually (IEG 2016s). The IEG learning product on The Role of Political Economy Analysis in Development Policy Operations is particularly relevant to the Macroeconomics and Fiscal Management practice. The report reviews the extent to which political economy 210 APPENDIX J REGIONAL RESULTS AND PERFORMANCE analysis is used to improve development policy operation (DPO) design and how effective it was. The main messages are:  Political economy analyses can be used to assess countries’ eligibility for budget support, provide important insights on the political feasibility of specific reforms supported by DPOs, inform specific design elements of important reform actions in a programmatic series (including choices of programmatic instruments versus stand-alone operations, or front-loading versus back-loading), and, as part of self-evaluation, to better analyze factors affecting program effectiveness and to contribute to knowledge and improved design of subsequent operations.  Political economy analysis customization specifically for DPO reform priorities is probably underprovided—most such analyses used in DPOs are geared toward countrywide governance and institutional issues. This may be associated with reliance on national development strategies signaling national ownership, variation in the institutional depth of reform areas, budget pressure, and other factors.  Setting better priorities for World Bank–supported reforms and assessing the risks of specific reform actions may strengthen DPOs’ results orientation regarding policy and institutional change, and may increase demand for political economy analyses (IEG 2016o). IEG has not recently undertaken a dedicated evaluation of projects mapped to the Trade and Competitiveness Global Practice in aggregate, but recent major evaluations provide specific lessons relevant to the practice. Critical challenges from the evaluation World Bank Group Support for Innovation and Entrepreneurship: An Independent Evaluation include the following:  Articulating a clear vision of how innovation will be used to solve major development problems and how this vision can be transformed into workable solutions.  Enhancing coordination, consultation, and links on innovation and entrepreneurship initiatives across networks, sectors, and regions, and across the World Bank Group institutions.  Developing practical solutions for people who earn less than $2 a day through sustained efforts to experiment with different mechanisms and implementation arrangements and to scale up successful models (IEG 2013c). From the IEG evaluation Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations, critical challenges in investment climate (regulatory reform) include the following: 211 APPENDIX J REGIONAL RESULTS AND PERFORMANCE a. Expanding the coverage of diagnostic tools and integrating them to provide full coverage of the business environment. b. Finding an approach to identify the social effects of regulatory reform on all affected groups, not just the business community. c. Ensuring that the World Bank Group takes advantages of relative strengths in World Bank and IFC business models and applies them collaboratively and synergistically (IEG 2015d). From Industrial Competitiveness and Jobs: An Evaluation of World Bank Group Industry- Specific Support to Promote Industrial Competitiveness and Its Implications for Jobs, critical challenges include the following: a. Better articulating the World Bank Group’s approach to industry-specific competitiveness work and ensuring a consistent treatment across the World Bank Group (a function clearly associated with the Trade and Competitiveness). b. Strengthening (in line with the Sustainable Development Goal 9.2) the World Bank Group’s industry level support to inclusive and sustainable industrialization. c. Integrating a jobs perspective into the World Bank Group’s industry-specific support to competitiveness (IEG 2016e). Project Performance Assessment Reports Macroeconomics and Fiscal Management IEG completed PPARs for projects in some, but not all, Global Practices in FY16. Three PPARs on budget support in Ghana and four PPARs on policy-based guarantees (PBGs) in the Balkans are relevant to the Macroeconomics and Fiscal Management practice. (The PPARs covering budget support in Ghana are also relevant to the Poverty practice and are presented under that heading.) The four PBGs evaluated in the Balkans were the Montenegro Financial Sector Policy-Based Guarantee, Serbia Private and Financial Sector Policy-Based Guarantee, the former Yugoslav Republic of Macedonia Policy-Based Guarantee, and the former Yugoslav Republic of Macedonia Public Expenditure Policy-Based Guarantee. The PPAR for the Montenegro Financial Sector Policy-Based Guarantee evaluates the guarantee of €60 million implemented in 2012–13, which supported a commercial loan of €100 million. The PBG had one objective: strengthening the banking system. Although the objective was highly relevant, the program design had weaknesses associated with the lack of operational focus on the program’s macroeconomic framework, and risks and shortcomings in assessing Montenegro’s 212 APPENDIX J REGIONAL RESULTS AND PERFORMANCE eligibility for a PBG. The achievement of the objective was modest, even though the banking system largely recovered from the crisis. The anticipated reform program remains incomplete, with gaps in crisis management, a failure to fully comply with the Basel Core Principles for Effective Banking Supervision, and slower than expected decline in nonperforming loans. This evaluation notes that many important outcomes achieved in Montenegro’s banking system were attributable to strong reform actions supported by the predecessor Financial Sector Development Policy Loan, underscoring the PBG’s relatively low value added. The PPAR for the Serbia Private and Financial Sector Policy-Based Guarantee reviews a guarantee approved on February 10, 2011. The operation’s objectives included facilitating Serbia’s access to international markets and assisting the country in achieving longer tenor and lower interest terms on its external commercial loans. They also included enhancing the business environment to support private sector investment, strengthening financial discipline with continued reform of the non–private enterprise sector (particularly focused on bankruptcy), and building a stable and more efficient financial sector through continued restructuring of state holdings in banking, crisis preparedness, support to insurance sector development, and promoting the development of capital markets. The guarantee covers the principal amount of a commercial bank borrowing $400 million with a six-year bullet maturity on a non-accelerable basis. The PBG’s relevance, design, and results largely met the high expectations of financial and policy reform objectives specified at the outset. The World Bank took a calculated risk with the new instrument to respond to the client’s request for a much larger volume of financing than was possible under the development policy loan (DPL). The government managed to double maturity and substantially lower the cost of borrowing, leading to Serbia independently and regularly accessing international markets on favorable terms and long-term maturities. Policy reform objectives also showed high achievement (15 of 18 outcome indicators were fully achieved and one was partially achieved). The evaluation established that most reforms were sustained, several reforms were deepened, and outcomes improved further as time passed. In one area (enhancing financial discipline), required actions were also completed and outcomes were achieved and sustained, but it was about two-and-a- half years before these reforms achieved renewed momentum in late 2014 and in 2015, in the context of new IMF and World Bank support programs. The PPAR for the FYR Macedonia Policy-Based Guarantee assesses a guarantee approved on November 10, 2011. The objectives of the operation were to improve the government’s access to international financial markets, fortify public finance sustainability and labor market functioning, strengthen social safety nets, and support financial sector resilience. The PBG’s relevance, design, and results met the 213 APPENDIX J REGIONAL RESULTS AND PERFORMANCE expectations of financial and policy reform objectives. As with the Serbia PBG, the World Bank took informed risks with the new instrument to respond to the client’s request for a much larger volume of financing than would have been available under a DPL. The reform agenda showed moderate achievements and sustainability. The objectives linked well to the PBG’s design and the desired outcome of the country eventually accessing markets independently. Policy reform objectives were appropriate based on a record of solid macroeconomic and fiscal management characterized by low fiscal deficits and moderate public debt levels, good policy dialogue, extensive consultation, and past analytical work. Prior actions linked to objectives, and were limited in numbers, relevant, and concrete. Achievement of financial objectives was substantial. The government managed to raise the required funds and extend maturity of external borrowing using the PBG, which lead to FYR Macedonia independently accessing international markets on favorable terms. Policy reform objectives showed mixed achievement. Critical actions and outcomes in the financial sector were substantially achieved, though achievement was modest in strengthening public finances and labor market functioning and in strengthening the social safety net. The emergence of health sector arrears and other payment arrears during implementation and supervision undermined the overall achievement. PBG-supported reforms proved important to budget and banking sector resilience, allowing the sector to absorb adverse economic and external shocks. The PPAR for the FYR Macedonia Public Expenditure Policy-Based Guarantee evaluates a guarantee of €155 million implemented in 2012–13. The PBG supported a commercial loan of €250 million. This was the country’s second PBG in the aftermath of the global crisis. The guarantee had three specific objectives: improving public expenditure efficiency by strengthening public financial management practices, improving service provision efficiency in the health sector, and strengthening social protection and inclusion. These objectives were substantially relevant to World Bank and government strategies and country context. However, IEG rated the program design as modest because it lacked operational focus on the program’s macroeconomic and fiscal risks, and the public financial management reform agenda’s scope was relatively narrow at a time of heightened risks. The achievement of objectives was mixed. IEG rated the efficacy of the first objective (improving public expenditure efficiency by strengthening public financial management practices) as modest. The program helped to achieve clearance of the arrears identified in the program and introduce changes in treasury practices to prevent accumulation of future arrears, but data gaps at the central level limited reporting, and actual arrears at the municipality level and in the health sector remain quite large. Furthermore, the capital expenditures efficiency deteriorated and is 214 APPENDIX J REGIONAL RESULTS AND PERFORMANCE concerning. The second objective (improving service provision efficiency in the health sector) was substantially achieved. The program helped reduce the unit price of pharmaceuticals and some medical equipment considerably, though the implication of these changes for budgetary savings in 2013–15 was less evident—an increase in the number and cost of prescriptions offset the positive impact of unit price reduction. Service provision to underserved areas improved through the introduction of a new framework that helped direct more resources to those areas. Achievement of the third objective (strengthening social protection and inclusion) was also substantial. Improving access of vulnerable groups to social services and generating fiscal savings strengthened the government’s active labor market programs. In social protection, the program helped improve targeting considerably. Fiscal resources freed up because of improved targeting enhanced the levels of financial assistance to beneficiaries. The four PPARs produced a range of important lessons, and the following are key:  PBGs can be a useful instrument for supporting countries facing large external financing needs. The instrument may be relevant for many countries seeking to strengthen access to international financial markets while advancing policy reforms. It fills the niche between commercial borrowing and traditional DPOs while providing borrowers with better-than-commercial terms and larger volumes than would be available under a DPO, allowing the World Bank to take informed risks to support client needs promptly.  PBGs implemented in a context of challenging macro-fiscal situation and large financing needs must incorporate a consistent macroeconomic framework with necessary macro and fiscal actions for risk mitigation.  Strong, relevant design and high achievement are much more likely when the operation’s design is informed by considerable knowledge work, intensive and long-term World Bank engagement and dialogue, and a relationship of trust between the World Bank and the borrower.  The borrower can be highly motivated to deliver on reforms, partly because of a trust relationship with the World Bank, the large volume of borrowing, and perhaps because of a third-party presence (a foreign commercial bank providing funding, for example), which puts additional pressure on the loan timing and closure.  A PBG can unlock multiple benefits for the borrower that may not be easily captured by alternatives such as a standard DPL—for example, borrowing a larger amount (because of the World Bank’s set-aside policy, which accounts for only one-quarter of the PBG against the country’s borrower limit), and opening the door to substantial subsequent international borrowing at favorable terms based on an established record. 215 APPENDIX J REGIONAL RESULTS AND PERFORMANCE  PBGs are associated with the risk of debt repayment (as with any large loans, especially with a guaranteed component). A strong policy reform program focused on key macro and structural issues with relevant prior actions are essential to guard against the possibility of reduced incentives for the borrower to exercise caution in borrowing and implementing reforms.  Comparing the aggregate interest rate of PBG-supported debt instruments with counterfactual market rates may not be enough for assessing the extent of reduction in the borrowing costs associated with the transaction. Estimating the change in sovereign risk perception is a challenging task, but it may give a more accurate assessment of the program’s impact. Governance IEG prepared six PPARs in FY16 that are particularly relevant from the Governance perspective. The assessments had two broad themes. In Rwanda and Tanzania, the theme was strengthening decentralization, local governance, and community empowerment, with special emphasis on decentralization process, form, and manifestation through empowering and building capacities at the local level. (The Rwanda project is under Social Protection and Labor, and the Tanzania project is under Social, Urban, Rural, and Resilience, and each is referenced under its particular heading. The cross-relevance of the projects’ governance-related components supports including them here.) The second theme (in Malawi, Zambia, and two PPARs in Vietnam) was strengthening public financial management, civil service reform, and oversight institutions, with special emphasis on the design, implementation, and utilization of financial management information systems (FMIS). A PPAR in Rwanda evaluated the Rwanda Decentralization and Community Development Project, which was developed based on government resolve that decentralization and democratization were necessary to reconcile the Rwandan people. Decentralization was a core element of the government’s good governance agenda under its first poverty reduction strategy (2002–05). It was also enshrined in the country’s 2003 constitution as a way to strengthen national unity and reconciliation and to bring national government closer to the people. The project was designed to contribute to Rwanda’s long-term goal for decentralization through empowering communities and improving local government accountability. The project’s development objective was to boost the emergence of a dynamic local economy through empowering communities to lead their own development process under effective local government. A PPAR in Tanzania evaluated the Local Government Support Project. The project’s development objectives were to strengthen fiscal decentralization and improve local 216 APPENDIX J REGIONAL RESULTS AND PERFORMANCE government resources accountability and intergovernmental transfer systems management, and to increase access to infrastructure and services in the unplanned areas of Dar es Salaam and improve revenue performance for sustainable operation and maintenance. The project was the first World Bank operation in Tanzania to support the Local Government Reform Program and, in particular, the intergovernmental transfer system. The government’s 1998 policy paper on local government reform, which was implemented in two phases (1998–2008 and 2009– 14) with mixed results, underpinned the decentralization effort, articulating a vision of a system of autonomous local governments. The Rwanda and Tanzania PPARs focused on strengthening decentralization, local governance, and community empowerment, and provided the following lessons:  Well-designed participatory processes to promote demand-side governance and empower communities, along with intensive institutional and capacity building at the central and local levels, can foster decentralization in a post- conflict environment (Rwanda) with a history of highly centralized structures and dominant central government. The project in Rwanda provided the building blocks for fostering community participation and local accountability, and enhanced social cohesion through financing subprojects that communities identified through a participatory planning process that feeds into the districts’ annual action plans.  Decentralization and capacity building is a long-term process that needs sustained government and World Bank engagement (Rwanda and Tanzania) and buy-in from other stakeholders.  Flexibility to adapt to changing circumstances is crucial to the operation’s continued relevance and success in a rapidly evolving environment (Rwanda). The project’s ability to respond quickly after a major territorial reform and rapidly evolving decentralization priorities was essential to its success in achieving development objectives.  Rapid scaling up poses challenges to capacity, particularly when capacity is uneven and resources are limited. In Tanzania, the decision to expand gradually at the outset was warranted, but the subsequent rapid expansion to all local government agencies created challenges that seriously strained the system’s capacity and credibility.  The decentralization agenda cuts across sectors. Therefore, interventions within and across sectors and supporting broader government public sector reforms need to be well coordinated to guarantee real synergies and impact. The PPAR for the Malawi Financial Management, Transparency, and Accountability Project assessed World Bank support in public financial management and civil service 217 APPENDIX J REGIONAL RESULTS AND PERFORMANCE reform to facilitate sound economic management. It also assessed the effective and accountable use of public resources to address core challenges (for example, 75 percent of the population was living on less than $1.25 per day). The project involved investment in a financial management information system (FMIS), strengthening public financial management processes and accountability institutions, and developing civil service capacity. Considerable progress in basic implementation was achieved across the various areas, but the systems did not lead to the expected benefits because compliance was not sufficiently enforced (for example, spending units were reported to maintain an off-FMIS registry). Failure of the underlying institutional and accountability environment also led to a major corruption episode, despite the systems in place. The PPAR for the Zambia Public Sector Management Program Support Project notes that the project operated in a country that had been peaceful and politically stable for most of its post-independence history. However, Zambia had a national poverty rate of about 60 percent and extreme poverty of about 42 percent in 2010. Zambia is generously endowed with natural resources, but it relied most heavily on copper, which exposed it to fluctuations in global demand. The macroeconomic situation had worsened since 2011, which led to a high current account deficit and a decline in mining-related investments. Expenditure adjustments were commensurate with reduction in revenue, and the fiscal balance deteriorated from a 1.8 percent deficit in 2010 to 7.7 percent in 2015. Against this background, the project aimed to contribute to making public service delivery institutions more effective and efficient through strengthening public financial management and reforming the civil service. The achievement of project objectives was mixed, reflecting achievement of significant process outputs (integrated FMIS and payroll management and establishment control), but also failure to translate these into tangible outcomes because of political considerations and because various systems were bypassed or not used properly. IEG conducted a PPAR for the Vietnam Public Financial Management Reform Project and the Multi-Donor Trust Fund for Public Financial Management Modernization. Vietnam, a one-party socialist state dominated by the Communist Party of Vietnam, has moved toward a pragmatic, growth-oriented approach to economic policy since the 1990s. The country has experienced remarkable economic and social development during the past three decades, even though economic growth has been somewhat slower and the pace and the depth of reforms more modest since the international financial crisis. Vietnam was one of the poorest countries in the world in 1986 and is now a middle-income country, with some development indicators comparable to those of OECD members. The Public Financial Management Reform Project’s development objectives were “to strengthen the borrower’s capacity to (i) plan, (ii) execute and (iii) report on its budget and to 218 APPENDIX J REGIONAL RESULTS AND PERFORMANCE improve the (iv) transparency and (v) accountability of the budgetary systems and processes.” The Public Financial Management Modernization Project’s objectives were “to support Vietnam to strengthen its capacities for: (a) increased (i) coordination and (ii) transparency in public finance management; (b) improved revenue mobilization; and (c) (i) effective and (ii) efficient public expenditure management.” The project achieved its various objectives principally on the successful implementation of a treasury and budget management information system, development of the debt database, and piloting the medium-term expenditure framework in four ministries and three regions. The approach is being mainstreamed as a result. The three PPARs that focused on strengthening public financial management, civil service reform, and oversight institutions with a special emphasis on the design, implementation, and use of FMIS produced several lessons, including the following:  A functioning FMIS alone is not sufficient to achieve good public financial management. To be effective, the ministry of finance must clearly commit to routing all transactions through the system so that all expenditures are subject to the FMIS automated, ex ante, and internal controls (Malawi, Vietnam).  An FMIS should include all government expenditures to act as a fiscal and budget management tool (debt payments, fiscal transfers, the capital budget, extrabudgetary funds, and internally generated funds). Donor funds should also be channeled through the FMIS—this can be done and still retain their own accounting mechanisms and fiduciary safeguards (Malawi, Zambia).  Systems deployment strategies should make sure that a large part of the budget is covered from the outset. A major portion of the budgetary resources should be subjected to ex ante budget and cash control so that meaningful fiscal control and cash management are possible early in the deployment (Vietnam).  Implementing all FMIS modules simultaneously is difficult and rarely feasible. Instead, a sequenced rollout approach that sets priorities for core treasury reforms can be beneficial and ensure that client capacity is not overburdened (Zambia).  Sequencing public financial management reforms is important to avoid unnecessary complexity and ensure smooth and orderly introduction of desired reforms. Advanced budgeting techniques, such as program- and activity-based budgeting, should be introduced only after a fully functioning FMIS is in place (Zambia, Vietnam).  Changing an FMIS platform carries substantial risks. It is costly and time consuming, the change provides no guarantee of eventual fit and functionality, human capacity will be lost and needs to be rebuilt, and the reform focus 219 APPENDIX J REGIONAL RESULTS AND PERFORMANCE during the transition period will be on technical issues instead of the underlying public financial management environment (Malawi).  Good analytical work is important for program design, but program design must also be strategic, practical, and prioritized, and it must consider the borrower’s implementation and absorptive capacity (Zambia).  Consolidating fragmented bank accounts takes significant political will. However, it might be given low priority in times of economic growth and fiscal expansion. Idle balances should be of particular concern to countries prone to external shocks because not having a treasury single account means these funds cannot be used to cushion against unanticipated financing gaps (Zambia).  Financing government employee retrenchment can be deeply unpopular and carries significant reputational risk for the World Bank. The intervention is unlikely to be effective without accompanying complementary reforms, and progress should be closely monitored to avoid reentry of civil servants (Zambia).  Merit-based recruitment of senior civil servants is a crucial aspect of performance management. Performance contracts are of limited value if senior civil servants are politically appointed and can be dismissed arbitrarily (Zambia).  Donor coordination modalities should be carefully considered before project inception and designed to reduce transaction costs (Zambia).  Explicit recognition of the iterative nature of reform is important. For example, project design based on a complex institutional reform agenda and its implementation would benefit from provisions for frequent, in-depth supervision to inform periodically on critical intervention areas that may need adjustment and priorities (Zambia). Public financial management reforms need decades to complete—in this context, continued, effective provision of financial support and, more important, knowledge by development partners will be needed (Vietnam).  A flexible approach to M&E can be useful. For example, an evolving project- level M&E might be a good way to respond to the iterative, demand-driven nature of activities that do not allow clear, measurable objectives to be set from the start. Approaches such social media may be informative and cost effective. Stand-alone studies and assessments provide important complements to results-based M&E (Vietnam), and M&E with periodic, complementary assessments strengthen project implementation. Finance and Markets 220 APPENDIX J REGIONAL RESULTS AND PERFORMANCE IEG produced three PPARs in FY16 that are relevant to the Finance and Markets Global Practice. The first two looked at PBGs in Serbia and in Montenegro. The main lessons from the two PPARs conducted for the PBGs include the following:  PBGs can be a useful instrument for supporting countries facing large external financing needs.  PBGs could be relevant for many countries seeking to strengthen access to international financial markets while advancing policy reforms, filling the niche between commercial borrowing and traditional DPOs while providing borrowers with better-than-commercial terms and larger volumes than would otherwise be available under a DPO.  The World Bank can productively take informed risks with new and innovative instruments to support client needs promptly that would not have been possible with standard World Bank support instruments.  PBGs implemented in a context of challenging macro-fiscal conditions and large financing needs benefit from incorporating a consistent macroeconomic framework with necessary macro and fiscal actions for risk mitigation.  The choice of a PBG as an instrument should be based on robust macro and fiscal projections indicating that the financing framework is sustainable and macro risks are mitigated.  Comparing the aggregate interest rate of PBG-supported debt instruments with counterfactual market rates may not be enough for assessing the extent of reduction in the borrowing costs associated with the transaction.  Estimating the change in sovereign risk perceptions, although challenging, may give a more accurate assessment of the program’s impact.  Strong and relevant design and high achievement are much more likely when considerable knowledge work, intensive and long-term World Bank engagement and dialogue, and a relationship of trust between the World Bank and the borrower informs the operation’s design. The PPAR for the Arab Republic of Egypt Mortgage Finance Project provides the following lessons:  Housing finance is not the exclusive domain of formal financial institutions. In designing projects, a good understanding of the various formal and informal market players that affect and would be affected by the intervention is important.  Developers are usually well positioned to assess and manage home buyer and borrower credit risk, but their financing activities raise some concerns.  If functioning bond markets are not in place, the mortgage liquidity facility would not be able to perform its most critical role as central bond issuer. 221 APPENDIX J REGIONAL RESULTS AND PERFORMANCE  Even if bond markets are functional, high and persistent inflation may make the mortgage liquidity facility impotent.  To compete with banks, the mortgage liquidity facility needs large-scale operations and frequent visits to the bond markets to reduce the cost of operations. If the mortgage markets are too small, the mortgage liquidity facility will not be competitive.  If mortgages are predominantly fixed rate, the case for adopting a mortgage liquidity facility is stronger. Poverty IEG prepared one PPAR under the Poverty Global Practice in FY16 that is also relevant to the Macroeconomics and Fiscal Management practice. The assessment evaluates three consecutive DPOs in Ghana during 2009–12. The World Bank committed $300 million in 2009 through a two-tranche economic governance and poverty reduction credit (EGPRC). A programmatic series of two poverty reduction support credits (PRSCs) followed: PRSC-7 provided a credit of $215 million in 2011, and PRSG-8 provided a grant for $100 million in 2012. The objective of the Ghana EGPRC, disbursed in two tranches for a total of $300 million, was “to support the authorities’ three-pronged effort to restore budgetary discipline and tackle long-standing public sector and energy issues, while protecting the poor.” The objectives of PRSCs 7 and 8 were similar to the EGPRC’s and included restoring budget discipline and financial stability, tackling long-standing public sector and energy issues, protecting the poor, and preparing the economy for the oil era (including managing expected oil revenues). The EGPRC contributed to reducing the fiscal deficit between 2008 and 2011 from 8.4 percent to 3.2 percent, and poverty decreased substantially across the country, coinciding with a faster-than-expected implementation of the Livelihood Empowerment against Poverty (LEAP) Program expansion it supported. IEG rated this part of the reform as substantial. However, another part of the program achieved only modest results. Utility companies’ reforms were never fully operationalized, and the government continued to set administrative fuel retail prices that were too low. Furthermore, only a few civil service reforms were undertaken, and wages increased. PRSCs 7 and 8 achieved substantial efficacy in social protection and oil revenue management, but only modest achievement in fiscal discipline and public sector reforms. Ghana decided on a fiscal expansion before the 2012 election that erased stabilization gains made during the previous EGPRC operation. Monetary accommodation of the fiscal deficit produced high inflation and persistent pressure on the exchange rate. Macroeconomic stabilization 222 APPENDIX J REGIONAL RESULTS AND PERFORMANCE was achieved only in 2015. Modest results were achieved related to long-standing public sector and energy issues. However, the LEAP Program continued to produce positive results in poverty reduction. Ghana also established institutions that could allow transparent oil revenue management and account for intergenerational considerations by creating the Ghana Heritage Fund for accumulating oil revenues for the next generations. The EGPRC and two PRSC operations in Ghana generated the following lessons:  Prudent macro-fiscal management is important for countries that were supported under the Heavily Indebted Poor Countries Initiative (HIPC), and the World Bank can in ensuring this through policy lending that helps clients preserve borrowing space and safeguards sustainability of pro-growth and pro-poor expenditure in the long term. Furthermore, adequate macroeconomic management is a critical precondition of budget support—the World Bank is no substitute for the IMF in a challenging context that requires a comprehensive macroeconomic program, and Ghana is a good example of World Bank–IMF cooperation in this regard.  DPOs deployed in an extremely weak public financial management environment are unlikely to be successful. Ghana experienced a deterioration in resource allocation during 2008–12 because of debt-financed expenditure hikes that led to interest payments of about 7 percent of gross domestic product per year by 2016, taking precious resources away from pro-poor spending, thus undermining Ghana’s overall growth and poverty reduction agenda and the effectiveness of the World Bank’s budget support. This emphasizes that the quality of public resource use in both public financial management and allocations is an important precondition of budget support success.  Donor coordination on budget support can be difficult in a challenging environment. Despite a move toward performance criteria based on agreed actions, various donors may interpret emerging shocks differently, and this can affect donor coordination. Dealing with macro and fiscal mismanagement when the macro framework is not formally a part of the performance assessment can be especially contentious.  Fiscal pressure can crowd out discretionary expenditures, and service delivery sectors are especially vulnerable. The World Bank can use DPF operations to take a crucial role in protecting pro-poor expenditures in such an environment. The World Bank has a crucial role in providing advisory services that are complementary to budget support. In Ghana, a more proactive engagement in estimating the fiscal impact of the salary reform introduction could have helped in designing a phased approach consistent with the long-term budgetary envelope. 223 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Figure 1. a. Percentage of Projects rated MS+ on Outcome in the Equitable Growth, Finance, and Institutions Cluster: IPF versus DPF IPF DPF 100 81 82 84 81 81 82 80 80 80 76 76 Percentage 75 73 80 60 70 69 70 64 65 66 65 67 64 66 65 60 62 40 b. Percentage of IPF Projects rated MS+ on Outcome: Equitable Growth, Finance, and Institutions versus Other Clusters EFI Cluster Other Clusters 100 77 81 81 79 Percentage 74 76 76 73 80 71 70 69 69 70 60 70 69 70 64 65 66 65 67 64 66 65 60 62 40 c. Percentage of DPF Projects rated MS+ on Outcome: Equitable Growth, Finance, and Institutions versus Other Clusters EFI Cluster Other Clusters 100 88 86 85 84 85 85 82 80 80 81 81 76 Percentage 75 80 80 81 82 82 76 77 76 60 72 71 73 66 67 62 40 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with a lending instrument other than IPF or DPF. DPF = development project financing; EFI = equitable growth, finance, and institutions; IPF = investment project financing; MS+ = moderately satisfactory or better. 224 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Figure 2. a. Percentage of Projects rated MS+ on Outcome in the Equitable Growth, Finance, and Institutions Cluster: IBRD versus IDA IBRD IDA 100 91 91 82 82 84 86 78 78 78 80 82 77 Percentage 74 80 60 72 75 73 69 70 70 69 70 66 63 63 60 61 40 b. Percentage of IBRD Projects rated MS+ on Outcome: Equitable Growth, Finance, and Institution versus Other Clusters EFI Cluster 100 91 91 Other Clusters 83 86 83 82 84 86 78 78 79 82 77 Percentage 80 82 80 77 78 78 76 60 74 71 69 70 68 68 67 40 c. Percentage of IDA Projects rated MS+ on Outcome: Equitable Growth, Finance, and Institution versus Other Clusters EFI Cluster 100 Other Clusters 77 76 76 76 Percentage 72 73 75 74 73 75 80 70 70 71 72 75 73 60 69 70 70 69 69 66 63 63 60 61 40 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with an agreement type other than IBRD or IDA. DPF = development project financing; EFI = equitable growth, finance, and institutions; IPF = investment project financing; MS+ = moderately satisfactory or better. 225 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Figure 3. Project Outcomes Rated MS+ (%) in the Equitable Growth, Finance, and Institution Cluster by Global Practice (Closing FY13–15) Rated MS+ (%) by Number of Projects Rated MS+ (%) by Net Commitments Finance & Markets 74% 96% Governance 56% 76% Macro Economics & Fiscal Management 82% 97% Poverty and Equity 71% 99% Trade & Competitiveness 65% 54% Equitable Growth, Finance and Institutions Cluster 69% 91% Other Clusters 73% 85% Bank-wide 72% 87% Source: IEG data. Note: The data excludes projects not mapped to a cluster. EFI = equitable growth, finance, and institutions; MS+ = moderately satisfactory or better. Figure 4. Project Outcome Rated MS+ (%) by Region: Equitable Growth, Finance, and Institution Cluster versus Other Clusters, FY13–15–15 EFI Cluster Other Clusters Africa 59% 41% 70% 30% East Asia and Pacific 58% 42% 76% 24% Europe and Central Asia 85% 15% 77% 23% Latin America and the Caribbean 81% 19% 71% 29% Middle East and North Africa 67% 33% 60% 40% South Asia 64% 36% 86% 14% All Regions 69% 31% 73% 27% Source: IEG data. Note: The data excludes projects not mapped to a cluster. EFI = equitable growth, finance, and institutions; MS+ = moderately satisfactory or better. 226 APPENDIX J REGIONAL RESULTS AND PERFORMANCE Figure 5. M&E Quality Rated Substantial+ (%) for the Equitable Growth, Finance, and Institution Cluster by Global Practice (Closing FY13–15) Macro Economics & Fiscal Management 46% N=50 Finance & Markets 44% N=34 Poverty and Equity 43% N=7 Trade & Competitiveness 28% N=18 Governance 24% N=54 Equitable Growth, Finance and Institutions Cluster 36% N=163 Other Clusters 31% N=625 Bank-wide 32% N=788 Source: IEG data. Note: The data excludes projects not mapped to a cluster. EFI = equitable growth, finance, and institutions; M&E = monitoring and evaluation; N = total number of projects rated by IEG on M&E quality. 227 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Table 1. IBRD and IDA Lending Commitments in the Equitable Growth, Finance, and Institutions Cluster, FY12–16 2012 2013 2014 2015 2016 $, % of $, % of $, % of $, % of $, % of millions cluster millions cluster millions cluster millions cluster millions cluster total total total total total Finance and Markets 693 6 1,037 11 1,427 13 3,322 30 1,802 13 Governance 591 5 1,882 21 1,459 13 532 5 2,121 16 Macroeconomics and Fiscal Management 8,778 79 5,446 60 6,213 56 6,538 58 8,842 65 Poverty 530 5 10 0 628 6 9 0 236 2 Trade and Competitiveness 560 5 647 7 1,312 12 855 8 669 5 $, % of $, % of $, % of $, % of $, % of millions total millions total millions total millions total millions total IBRD IBRD IBRD IBRD IBRD and IDA and IDA and IDA and IDA and IDA Equitable Growth, Finance, and Institutions 11,152 32 9,022 29 11,038 27 11,255 26 13,670 30 cluster Other clusters 24,104 68 22,525 71 29,643 73 31,240 74 32,229 70 Source: World Bank Business Intelligence. 228 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Human Development Global Practice Cluster The Human Development cluster consists of three Global Practices:  Education: supporting countries that request financing or technical assistance to reach Sustainable Development Goal 4, which calls for access to quality education and lifelong learning opportunities for all by 2030.  Health, Nutrition, and Population: promoting universal health coverage, and providing financing, advanced analysis, and policy advice to help countries expand access to quality, affordable health care. It also sets priorities for protecting people from falling into poverty or becoming poorer because of illness, and promotes investments in all sectors that form the foundation of healthy societies.  Social Protection and Labor: supporting well-functioning social safety nets proven to reduce poverty and inequality, promoting access to health and education among poor children, empowering women, and supporting sustainable social insurance programs that help cushion the impact of crises on households. It supports promoting effective policies for productive employment that help people access labor markets and accumulate skills in normal times and during recovery from economic crisis. The Human Development cluster is the World Bank’s smallest Global Practice cluster by commitments. The cluster’s commitments in FY16 were almost $8 billion, which is 16 percent of total World Bank commitments (table 1). In comparison, total FY16 commitments for the Sustainable Development cluster were almost $25 billion (54 percent of total World Bank lending), and FY16 commitments for the Equitable Growth, Finance, and Institutions cluster were $13.7 billion (30 percent of total World Bank lending). Performance Trends For the current review period, FY13-15, 77 percent of projects in the Human Development cluster were rated MS+, which is above the World Bank average of 72 percent (figure 3). The percentage of Human Development cluster investment project financing projects rated moderately satisfactory or better (MS+), on a three- year rolling average basis since FY04–06, was somewhat lower than that of the other two clusters before it made progress and surpassed performance in those clusters in the last three analysis periods. During FY13–15, 77 percent of the investment projects in the cluster were rated MS+—which compares to the 69 percent MS+ rating for the other two clusters 229 APPENDIX J IEG NOTE ON GLOBAL PRACTICES in the same period (figure 1 panel b). Development policy financing (DPF) within the cluster has been slightly stronger than the average across the other two clusters until the performance gap began widening since FY10–12. The ratings for DPF in the Human Development cluster in the current period (FY13-15) is 88 percent MS+ compared with an average 82 percent MS+ across the other two clusters (figure 1 panel c). Three-year rolling average ratings for IDA and IBRD projects in the cluster are broadly similar over time, though a slight gap opened in FY13–15 when IBRD projects performed better than IDA projects—78 percent of IBRD projects were MS+ versus 74 percent of IDA projects (figure 2 panel a). IBRD projects in the cluster outperform the average ratings for IBRD projects across the other clusters (78 percent versus 74 percent MS+, respectively) between FY13–15 (figure 2 panel b). IDA projects perform at just above the average for IDA projects across the other two clusters (74 percent and 72 percent MS+, respectively) during FY13–15 (figure 2 panel c). Ratings for 90 percent of the 40 projects rated in the Social Protection and Labor Global Practice during FY13-15 were MS+ in the review period, which is well above the corporate target of 75 percent MS+ by FY17. However, 74 percent of 80 projects under Education and 73 percent of 67 projects under Health, Nutrition, and Population were rated MS+, which is slightly below the FY17 corporate target (figure 3). Disaggregated by World Bank Regions, 89 percent of 19 Human Development projects for the South Asia Region were rated MS+, as were 80 percent of 25 projects rated in the East Asia and Pacific Region, 79 percent of 66 projects rated in the Africa Region, and 78 percent of 36 projects rated in the Latin America and the Caribbean Region. The Human Development cluster rating for the Europe and Central Asia Region was 70 percent MS+. Performance for projects in the Human Development cluster lagged for the Middle East and North Africa Region at 61 percent MS+ (figure 4). The cluster’s overall monitoring and evaluation (M&E) rating for FY13–15 is 37 percent substantial or better, which is above the average for other clusters (31 percent substantial) and the overall World Bank average (32 percent substantial). Coincidental with a high development outcome rating, projects in Social Protection and Labor also performed particularly well regarding M&E (50 percent substantial). The M&E rating for Health, Nutrition, and Population (37 percent substantial) was also better than the portfolio average, and the M&E rating for Education projects was 31 percent substantial (figure 5). 230 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Main Evaluation Findings and Lessons 1. This section summarizes key findings and lessons from IEG’s major evaluations, learning products, and project performance assessment reports (PPARs) completed during FY13–16 and relevant to the Human Development cluster. Major Evaluations and Learning Products Education 2. IEG did not disclose any major evaluations or learning products relating to the Education Global Practice in FY16, but its evaluation of World Bank Group support for higher education FY04–15 will be disclosed in the latter half of FY17. The evaluation analyzes the World Bank Group’s role in promoting equitable access, employable skilled graduates, and research and innovation, and intends to be forward-looking because it seeks to understand how well it aligns with the World Bank’s twin goals in its current context. Health, Nutrition, and Population 3. IEG is in the process of evaluating World Bank Group support for health services. The evaluation (scheduled for disclosure in late FY17) will collect evidence, develop lessons, and propose recommendations that could enhance World Bank Group support to country clients as they move toward universal health coverage. The following are the evaluation’s overarching questions: What has the World Bank Group’s role and contribution been in supporting health services? What should the World Bank Group’s role and contribution be in supporting health services, considering its comparative advantages? The evaluation will cover all IBRD and IDA projects, IFC investments and advisory services, World Bank advisory services and analytics, and partnership programs and multidonor trust funds approved during FY05–16. 4. IEG’s learning product Public-Private Partnerships in Health: World Bank Group Engagement in Health PPP: An IEG Synthesis Report (IEG 2016l) is a desk-based portfolio review of existing evidence associated with the engagement of all World Bank Group institutions in health PPPs. The report identifies lessons from successful and failed efforts to structure health PPPs approved during FY04–15. The study identifies five lessons: • IFC and the World Bank jointly should analyze the entire range of options with government clients—from the public and mixed options to potential PPPs models—considering the health sector context, country capacity, and its track record in PPPs. 231 APPENDIX J IEG NOTE ON GLOBAL PRACTICES • World Bank Group support to health sector reforms and policy work could be better integrated with PPP structuring and financing. • Distributional and affordability aspects should be considered at the design stage and systematically tracked during PPP implementation. • The World Bank–IMF jointly developed PPP fiscal risk assessment model could be applied to structuring PPPs with potentially substantial fiscal implications. • Systematic preparation of post-completion reports for IFC operations supporting PPP would allow for assessing health PPP results considering their long contractual duration. 5. IEG evaluations and reports completed in recent years that were cited in the Human Development Global Practice Update section of Results and Performance of the World Bank Group 2015: An Independent Evaluation (IEG 2016n) include:  World Bank Support to Early Childhood Development: An Independent Evaluation, which examined the World Bank’s design and implementation of operations (FY00–14) that support interventions for young children and their families. The evaluation recommends that the World Bank adopt a well-coordinated and strategic framework for early childhood development (ECD), use analytic work on ECD in preparing systematic country diagnostics to determine need, increase knowledge to address key ECD operational challenges, and improve the M&E of ECD intervention (IEG 2015s).  IEG’s learning product, Delivering the Millennium Development Goals to Reduce Maternal and Child Mortality: A Systematic Review of Impact Evaluation Evidence, found that appropriately designed interventions are more likely to yield significant results in countries with a larger burden, such as lower skilled birth attendance rates or higher mortality. Households with lower socioeconomic status realized larger benefits from these interventions, but utilization among the poor remains a challenge (IEG 2016c).  An evaluation of World Bank Group support for health financing reform during FY03–12, which provides the following key findings and recommendations: technical capacity and government commitment greatly facilitate reform; the poverty and equity effects of health financing need more attention; health financing requires a particular skill set—IEG advises that Global Practices expand staff technical capacity and focus on health financing as a comparative advantage; collaboration across public sector and health teams and between the World Bank and IFC was limited; and the quality of M&E is weak in World Bank and IFC operations and needs to be strengthened to better analyze the impact of World Bank and IFC operations on final outcomes (IEG 2014d). 232 APPENDIX J IEG NOTE ON GLOBAL PRACTICES  IEG’s learning product, Later Impacts of Early Childhood Interventions: A Systematic Review on the impacts of interventions during early childhood on later outcomes found that early childhood interventions can, but do not always, lead to benefits later in life in cognition, language, education, and the labor market. Evaluated interventions have not shown consistent long-term advantages for physical development, although these outcomes are less salient to adult welfare. Nutrition interventions may need to be in place throughout and beyond the first 1,000 days to take advantage of the window of opportunity from conception to the child’s second birthday and achieve sustained effects beyond early childhood. Sizable knowledge gaps persist, but careful planning and design can close the gaps (IEG 2015f). Social Protection and Labor 6. IEG did not produce any major evaluations or learning products relating to social protection and labor in FY16. The most recent major evaluation IEG conducted in this area was the systematic review of social safety sets and gender, detailed in the 2015 update. The study found that men and women respond to and benefit differently from social safety nets. It also found that not everybody in the household benefits equally from them because men, women, boys, and girls have different roles, responsibilities, and constraints, and they typically respond differently to incentives. The evaluation highlights the need to identify expected and desired gender results at preparation and address them in the project, and the need to collect more gender-disaggregated data in project M&E. Project Performance Assessment Reports Education 7. A PPAR for the Education for All Project (2004–09) in Nepal, a primary education project that the World Bank and four other donors jointly financed through a sectorwide approach (SWAp), distilled four lessons:  Emphasizing the quality of learning inputs (such as teachers and instructional and learning materials) is important to improving the quality of education, particularly learning outcomes.  In a low-capacity environment, a strategy to build local-level capacity is critical in a decentralized service provision model.  Central, district, and local-level roles and responsibilities should be clearly defined when education provision is decentralized.  Quality assurance in M&E data is essential when funding links to school-level data. 233 APPENDIX J IEG NOTE ON GLOBAL PRACTICES 8. IEG conducted a PPAR for the Millennium Science Initiative Project in Uganda (approved on May 25, 2006, became effective on March 2, 2007, and closed on June 30, 2013). The PPAR’s main lessons are as follows:  Although research funds are common in OECD countries, it is also possible to establish and implement a world-class science research fund in a low-income country.  An appropriate cost-sharing agreement is needed at entry to facilitate sustainability.  Promoting a knowledge-based economy requires an integral approach involving several ministries and the private sector.  A research fund is a viable mechanism to increase research and create programs for science and technology graduates, but its impact may be enhanced by extending capacity building and technical assistance to grantees.  Requiring data collection for grants may also yield better impact or at least a clearer understanding of what was achieved.  The low numbers of Ugandan women scientists, merits attention from a gender perspective. Health, Nutrition, and Population 9. IEG conducted PPARs for two projects in Niger. The Institutional Strengthening and Health Sector Support Project was supported by an IDA credit of special drawing rights (SDR) 24.2 million (approved on January 5, 2006), and the Multisector Demographic Project was supported by an IDA grant of SDR 6.7 million (approved on June 19, 2007). The PPARs reviewed project documents, aide- mémoires, supervision reports, data, and studies. Planned fieldwork was not conducted because of Niger’s security situation. The PPARs’ key lessons are as follows:  The SWAps’ capacity-strengthening potential (including program-wide support) is not likely to be achieved without clear capacity-strengthening objectives and viable institutional arrangements, intermediate objectives, a plausible results chain, relevant performance indicators, and proactive M&E.  A SWAp’s success (or failure) depends on four critical factors: the quality and relevance of strategies and annual work plans; country capacity and systems for financial management, procurement, and strategic management; the quality and functionality of partnerships with national and international actors and stakeholders; and the predictability and flow of funds and the absorptive capacities of implementing agencies. 234 APPENDIX J IEG NOTE ON GLOBAL PRACTICES  Improved quality and relevance of strategies and plans for health and population might have contributed to the support of fewer, more relevant activities and a higher impact. Social Protection and Labor 10. IEG conducted a PPAR for the Third Malawi Social Action Fund Project, Adaptable Program Loan II, which is a community development fund project approved on June 20, 2008, and completed on June 30, 2014. The main lessons drawn from the project are as follows:  Community-driven development can facilitate decentralization, but the right balance must be struck between community priority setting and local development planning.  Assessing institutional capacity at the design stage and systematically during implementation is crucial, especially when the project includes a component on capacity building at the local level.  Beneficiary feedback throughout the assessment provides important evidence of the operation’s impact from the beneficiaries’ perspective.  Subproject economic and financial costs should consider the affordability of subprojects in poor communities.  Linking income generation interventions with asset building opportunities can help build economic resilience. 11. IEG also completed a PPAR for the Second Tanzania Social Action Fund, a community development fund project implemented during FY05–12. The main lessons drawn from the project are as follows:  Mainstreaming the social fund subproject process at the local government level can strengthen decentralized planning outcomes.  Community participation in the local planning process does not ensure that community priorities are realized.  When poverty alleviation is the underlying goal, as in the Tanzania Social Action Fund, a combination of interventions can be effective.  Setting realistic guidelines can improve subproject outcomes. 235 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 1. a. Percentage of Projects rated MS+ on Outcome in the Human Development Cluster: IPF versus DPF IPF DPF 100 92 90 93 88 86 88 82 83 82 Percentage 78 78 81 75 80 74 75 75 75 77 60 72 68 72 67 66 66 66 65 40 b. Percentage of IPF Projects rated MS+ on Outcome: Human Development versus Other Clusters HD Cluster 100 Other Clusters Percentage 79 82 81 75 75 78 77 74 72 71 72 80 70 68 73 74 75 75 60 72 68 69 67 66 66 66 65 67 40 c. Percentage of DPF Projects rated MS+ on Outcome: Human Development versus Other Clusters HD Cluster Other Clusters 93 100 92 90 88 86 88 81 82 83 81 82 78 77 Percentage 80 80 83 82 80 76 78 78 77 76 75 74 73 74 60 40 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with a lending instrument other than IPF or DPF. DPF = development project financing; HD = human development; IPF = investment project financing; MS+= moderately satisfactory or better. 236 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 2. a. Percentage of Projects rated MS+ on Outcome in the Human Development Cluster: IBRD versus IDA IBRD 100 IDA 79 78 82 81 78 78 Percentage 80 70 73 69 68 67 67 69 74 72 71 71 72 74 60 70 69 67 65 63 64 65 40 b. Percentage of IBRD Projects rated MS+ on Outcome: Human Development versus Other Clusters HD Cluster 100 87 Other Clusters 82 86 79 78 82 81 78 78 78 Percentage 75 73 72 80 78 81 78 77 77 74 60 69 67 69 70 65 63 65 40 c. Percentage of IDA Projects rated MS+ on Outcome: Human Development versus Other Clusters HD Cluster 100 Other Clusters 77 78 76 Percentage 74 72 75 76 72 73 74 80 71 68 69 60 71 71 70 71 71 69 70 70 72 68 64 67 67 40 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with an agreement type other than IBRD or IDA. HD = human development; MS+ = moderately satisfactory or better. 237 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 3. Project Outcomes Rated MS+ (%) in the Human Development Cluster by Global Practice, FY13–15 Rated MS+ (%) by Number of Projects Rated MS+ (%) by Net Commitments Education 74% 85% Health, Nutrition & Population 73% 83% Social Protection & Labor 90% 98% Human Development Cluster 77% 89% Other Clusters 71% 86% Bank-wide 72% 87% Source: IEG data. Note: The data excludes projects not mapped to a cluster. MS+ = moderately satisfactory. Figure 4. Project Outcomes Rated MS+ (%) by Region: Human Development Cluster versus Other Clusters, FY13–15 Human Development Cluster Other Clusters Africa 79% 21% 64% 36% East Asia and Pacific 80% 20% 72% 28% Europe and Central Asia 70% 30% 81% 19% Latin America and the Caribbean 78% 22% 72% 28% Middle East and North Africa 61% 39% 60% 40% South Asia 89% 11% 79% 21% All Regions 77% 23% 71% 29% Source: IEG data. Note: The data excludes projects not mapped to a cluster. MS+ = moderately satisfactory or better. 238 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 5. M&E Quality Rated Substantial+ (%) for the Human Development Cluster by Global Practice (Closing FY13–15) Social Protection & Labor 50% N=40 Health, Nutrition & Population 37% N=65 Education 31% N=80 Human Development Cluster 37% N=185 Other Clusters 31% N=603 Bank-wide 32% N=788 Source: IEG data. Note: The data excludes projects not mapped to a cluster. M&E = monitoring and evaluation; N = total number of projects rated by IEG on M&E quality. 239 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Table 1. IBRD and IDA Lending Commitments in the Human Development Cluster, FY12–16 2012 2013 2014 2015 2016 $, millions % of $, % of $, % of $, % of $, % of cluster millions cluster millions cluster millions cluster millions cluster total total total total total Education 2,076 48 1,678 27 3,217 58 3,024 33 1,994 26 Health, Nutrition, and 916 21 1,796 29 1,239 22 2,978 32 1,984 26 Population Social Protection and Labor 1,352 31 2,807 45 1,071 19 3,257 35 3,572 47 $, millions % of total $, % of total $, % of total $, % of total $, % of total IBRD and millions IBRD and millions IBRD and millions IBRD and millions IBRD and IDA IDA IDA IDA IDA Human Development cluster 4,344 12 6,280 20 5,527 14 9,259 22 7,550 16 Other clusters 30,913 88 25,268 80 35,155 86 33,236 78 38,349 84 Source: World Bank Business Intelligence. 240 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Sustainable Development Global Practice Cluster 1. The Sustainable Development cluster consists of six Global Practices and two cross-cutting solutions areas. The six Global Practices are as follows:  Agriculture: aims to link farmers to markets to increase food availability and stimulate general economic growth using a value chain approach, including on-farm inputs, land, water, financial services, and post-harvest agro- processing.  Energy and Extractives: sets priorities for improving the energy investment climate by promoting sector reform and governance, strengthening utilities, enhancing investment frameworks, encouraging private participation, and rationalizing subsidies.  Environment and Natural Resources: seeks to promote a green, clean, and resilient world in which natural resources are managed to support livelihoods and strong economies, and to share prosperity by transferring wealth from downstream beneficiaries of ecosystem services to upstream communities that carry the opportunity costs of protecting nature.  Social, Urban, Rural, and Resilience: gives priority to ensuring that marginalized and vulnerable populations have a voice in defining the growth of cities, human settlements, and rural areas, which have local and global implications for sustainability and climate change.  Transport and Information and Communication Technologies: seeks to promote virtual and physical connectivity, to improve the movement of people, goods, and information, thus enabling access to economic opportunities, to vital services like healthcare and education, and to knowledge and ideas.  Water: aims to ensure that water is a reliable foundation for poverty reduction and shared prosperity by delivering public water goods coupled with private initiatives to add value to water services throughout the water cycle. The two cross-cutting solutions areas are:  Climate Change: aims to help country clients accelerate efforts to tackle climate change and deliver on their national climate plans.  Public-Private Partnerships (PPPs): provides strategic, leadership, and coordination support for the provision of basic services. PPPs introduce private sector technology, innovation, and funding to public services. 241 APPENDIX J IEG NOTE ON GLOBAL PRACTICES 2. The Sustainable Development cluster is the World Bank’s largest Global Practice cluster by commitments. The cluster’s commitments in FY16 were almost $25 billion, which is 54 percent of total World Bank commitments (table 1). In comparison, total FY16 commitments for the Equitable Growth, Finance, and Institutions cluster were $13.7 billion (30 percent of total World Bank lending), and $7.6 billion for the Human Development cluster (16 percent of the total World Bank lending). Performance Trends 3. For the current review period, FY13-15, 71 percent of all projects in the Sustainable Development cluster were rated MS+, slightly below the World Bank average of 72 percent (figure 3). 4. Taken on a three-year rolling average basis since FY01, the percentage of investment projects rated MS for the cluster was somewhat higher than that of the other two clusters before narrowing toward convergence in ratings since FY08–10 (figure 1 panel b). However, the development policy financing (DPF) performance within the cluster has traditionally been weaker when compared with the other two clusters until it began moving toward convergence since FY07–09. The rating for Sustainable Development DPFs in FY13–15 is 83 percent MS+ compared with an average of 83 percent MS+ across the other two clusters (figure 1 panel c). 5. Three-year rolling average ratings for IDA and IBRD projects in the cluster were broadly similar between FY07–09 and FY10–12. Since then, a gap in performance began opening in FY13–15 when IDA projects performed better than IBRD projects—77 percent of IDA projects were MS+ versus 69 percent of IBRD projects (figure 2 panel a). IBRD projects in the cluster also underperform average ratings for IBRD projects across the other clusters (69 percent versus 82 percent MS+, respectively) in FY13–15 (figure 2 panel b), and IDA projects outperform average ratings for IDA projects across the other clusters (77 percent and 68 percent MS+, respectively) in the same period (figure 2 panel c). 6. Projects in two Global Practices in the cluster were rated above the cluster average (71 percent MS) during FY13–15 and above the corporate target (75 percent MS+ by FY17). In Transport and Information and Communication Technologies (ICT), 79 percent of projects were rated MS+, as were 79 percent of projects in Agriculture, which performed particularly well between FY10–12 and FY13–15. The Social, Urban, Rural, and Resilience practice scored at the average (71 percent), and the other Global Practices performed below the cluster average: 68 percent for Water, 242 APPENDIX J IEG NOTE ON GLOBAL PRACTICES 66 percent for Energy and Extractives, and 62 percent for Environment and Natural Resources (figure 3). 7. Disaggregated by World Bank Regions during FY13–15, 84 percent of 44 Sustainable Development projects rated for the South Asia Region were MS+, as were 79 percent of 63 projects rated in Europe and Central Asia, and 76 percent of 94 projects rated in the East Asia Pacific Region. The other three Regions performed below the cluster average of 71 percent MS+: 68 percent of 85 projects rated in Latin America and the Caribbean, 66 percent of 122 projects rated in Africa, and 59 percent of 34 projects rated the Middle East and North Africa (figure 4). 8. IEG rated the Sustainable Development cluster’s monitoring and evaluation (M&E) quality for FY13–15 as 29 percent substantial compared with a 37 percent average rating across the other clusters. Within the cluster, the M&E rating for Energy and Extractives projects was a strong 48 percent substantial, which is well above the next highest rated Global Practice (29 percent for Agriculture). All other Global Practices rated lower than the cluster average (figure 5). Main Evaluation Findings and Lessons 9. This section summarizes key findings and lessons from IEG’s major evaluations, learning products, and project performance assessment reports (PPARs) completed during FY12–16 and relevant to the Sustainable Development cluster. Major Evaluations and Learning Products 10. After IEG’s major electricity access evaluation (IEG 2015q), IEG produced the learning product Financial Viability of the Electricity Sector in Developing Countries: Recent Trends and Effectiveness of World Bank Interventions. The learning product compiles a comprehensive inventory of World Bank electricity sector investments and development policy operations (DPOs) approved during FY00–15 that contain components and covenants for improving sector financial performance and viability. Findings suggest that leveraging investment operations and DPOs with technical assistance and analytical work will yield better and more sustainable results. Key lessons and conclusions include the following:  Addressing the political economy of sector financial viability is key because measures such as tariff adjustments toward cost recovery levels, which are crucial to setting the electricity sector on the path to long-term financial viability, can be politically sensitive. As such, aligning the DPO program’s timeline with that of a government reform program, matching the scale of 243 APPENDIX J IEG NOTE ON GLOBAL PRACTICES World Bank support to the scope of reforms and political risk, and balancing the window of opportunity with achievable targets should be considered.  Simple designs and realistic time frames are preferable—an overambitious agenda and too many loan conditionalities in DPOs can affect performance.  Retaining focus on the underlying structural causes of sector financial viability and using programmatic series of single-tranche policy loans can be effective in supporting a well-specified, medium-term financial recovery program (IEG 2016c). 11. The IEG learning product Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from World Bank Group Experience highlights good practice country program experiences in supporting commercially viable and rapidly scalable off-grid access and services provision models and institutional frameworks. These include supply and service delivery chains for solar home systems and microgrids, and the technologically and commercially fast-evolving pico-solar products and their growing transformative record. These experiences took different routes in mainstreaming off-grid electrification in line with their own needs and context. In Bangladesh, the scaled-up solar home systems program in grid-proximate areas took hold as an opportunistic market response to the then-stalled grid extension program. Off-grid electrification in Sri Lanka (isolated village hydro minigrid networks, initially) grew from a strategy anchored by an island-wide spatial electrification plan, and implemented as staged preelectrification until the planned grid extensions arrived at those locations. Off-grid electrification in Rwanda and Kenya is proceeding as planned. Morocco is an example of the planned preelectrification cycle transitioning to mostly grid-based electrification. Myanmar, drawing on previous good practice experiences, also began a conscious strategy of ex ante planned preelectrification, designing an off-grid program that promotes solar products, solar home systems, and isolated minigrid networks coordinated with grid expansion (IEG 2016m). 12. The IEG learning product Lessons from Environmental Policy Lending provided operationally relevant lessons from World Bank DPOs to inform the design of environmental and other sectoral development policy lending. The learning product provides several key insights. First, environmental development policy lending is most effective when it uses the instrument’s strengths—that is, when policy issues are the main barrier to improving environmental outcomes instead of capacity or other issues. Development policy lending offers advantages for achieving sectorwide or multisectoral goals, especially for policy issues that need attention from high-level decision makers. It can be most effective when the World Bank has prior knowledge of the country and sector, and strong institutional relationships that can be developed through use of other instruments. Another insight is that a 244 APPENDIX J IEG NOTE ON GLOBAL PRACTICES few key design and implementation considerations tend to determine how effective environmental policy lending can be. Policy lending is most effective when a clear political theory of change exists for how the operation will influence policy outcomes. Examples include supporting policy reforms that would not happen without the World Bank operation, or in other cases, influencing prioritization, timing, or technical quality. Furthermore, the results framework design is at the core of DPO design. The design of results frameworks requires intensive dialogue and debate between World Bank teams and governments involving tradeoffs and tensions between ambition and realism, additionality and country ownership, and depth and breadth. The strongest policy actions are relevant, critical, additional, and measurable. Programmatic series offer several advantages, including the ability to induce or support long-term government commitment to reforms, which is particularly important to ensure sustained implementation of policy reforms. However, M&E systems for environmental DPOs were often weak. Objectives were often imprecise or unclear, and indicators did not provide a direct or adequate reflection of the objectives or their associated subobjectives. Results frameworks sometimes measure processes instead of results or impact. The report offers advice on selecting objectives and indicators, noting pitfalls to avoid. Finally, analytical work and technical assistance are important to the success of environmental policy lending operations. Analytical work has a key role as a diagnostic and in providing the evidence base for persuading decision makers. However, sufficient analytic and advisory work is not always present because of tightening budgets and declining trust fund availability, timing and reliance on previous analytical work instead of new work commissioned specifically for the operation, and the unwillingness of many governments to borrow for technical assistance (IEG 2016i). 13. IEG’s learning product on housing financing refers to recent UN-Habitat estimates to assert that without significant action, the number of people living in slums worldwide will rise to 900 million by 2020, and that level of access to adequate housing will exacerbate problems associated with rapid urbanization and population growth (IEG 2016s). Mortgage markets in many developing countries remain weak and underfunded, with mortgage-to-gross domestic (GDP) product ratios below 10 percent. World Bank Group investment in housing finance is small compared with estimates of the value of the housing gap at ($9 trillion to $11 trillion). Given disparities in scale between its resources and the challenge, the World Bank Group must act strategically and needs to engage in catalytic interventions to make a real impact in the sector. The learning product highlights the importance of establishing compensating factors, such as adequate sequencing, timely interventions, and committed sponsors, particularly when preconditions for developing capital market and primary market interventions in housing are not in 245 APPENDIX J IEG NOTE ON GLOBAL PRACTICES place. A well-functioning housing finance sector typically evolves in three stages. Countries often start by strengthening the enabling environment, then progress to initiating the primary market before reaching a stage in which they can sustain mortgage funding through tapping the capital markets. Getting the sequence right can be crucial. For example, in Colombia, IFC and the World Bank helped the government develop a new housing law that contained the regulatory framework for asset securitization. IFC then supported the creation of a securitization company. As much as timing and sequencing matter—particularly on the policy side— conditions in the field might interfere. In such cases, committed sponsors who support passing key reforms can be crucial. Gaining the commitment of key players is also vital. For example, in Morocco, the government and World Bank succeeded in sustaining a low-income housing program mainly because there was strong, sustained commitment from key actors, particularly in government. 14. IEG’s 2016 learning product on land administration projects highlights the complexity and diversity of issues associated with land administration. A key lesson is that because of this complexity, stand-alone operations are better for addressing land administration challenges than a component of a multisector operation (IEG 2016j). The report also shows that there is no universal best practice for the most effective institutional model for carrying out cadaster and registry functions. The best model is one that matches a given historical and institutional development context. Land reforms require medium to long-term support, which programmatic instruments can facilitate, but they also require continuous monitoring of political commitment. Projects perform best when structured to make incremental improvements to the legal and policy framework. Interventions are more successful when tailored to the existing implementation capacity and incorporating measures to enhance the capacity as experience is gained, particularly regarding the introduction of technological advances. Improvements to land tenure alone will not always be sufficient to generate broader development outcomes. Reforms beyond the land sector may also be required, and these can be challenging. Parallel projects supporting reforms in other sectors are more manageable than a single operation that tries to tackle both land and non-land reforms. Social inclusion needs to be an explicit part of project design. Targeted strategies are required to reach vulnerable groups and ensure that all segments of the population will benefit from a particular land administration intervention, even when the laws and procedures are nominally the same for all potential beneficiaries. 246 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Project Performance Assessment Reports Agriculture 15. The Mozambique Market-Led Smallholder Development in the Zambezi Valley Project, approved in 2006 and valued at $28 million, was designed to help poor rural farmers in the Zambezi Valley increase their productivity and incomes and connect to wider opportunities within the rural economy. The project was located in some of Mozambique’s poorest districts in the poorest provinces. IEG’s PPAR, based on 365 beneficiary interviews and 68 subproject validations, found that the use of country systems (including financial management, procurement, and monitoring systems) was premature and prone to mismanagement. This led to overreliance on the country’s new decentralization framework, and the project lacked robust and clear participation criteria. This, coupled with the need to distribute capital-intensive infrastructure that required skills and access to labor and land, resulted in elite capture of productive assets. Overall, agricultural productivity increased, and farmers achieved additional crop diversity through the provision of enhanced technical assistance and the distribution of improved inputs. But the project did not tackle the underlying poverty drivers related to the composition and nature of the agricultural input and marketing chains, which were unstructured, asymmetric, and highly unfavorable for rural, small-scale farmers. The PPAR found no evidence that the project supported sustained, increased marketing opportunities for the smallholders. 16. The Mozambique Poverty Reduction Support Credits (PRSCs) 3, 4, and 5, valued at $220 million and implemented between 2007 and 2009, aimed at achieving better macroeconomic management and improved governance, and at removing constraints to development by enhancing the business environment, improving infrastructure, and promoting agricultural growth. The program’s objectives aligned well with the World Bank’s assistance and partnership strategies and were highly relevant to the challenges facing Mozambique’s economy. However, the program relied too much on public financial management reforms to achieve the broader macroeconomic objective, excluding macroeconomic stability indicators. The program’s design could have paid more attention at the design stage to the urban- rural distribution of pro-poor expenditure from the national budget and the quality of these expenditures. Macroeconomic performance improved during the implementation of PRSCs 3–5, but other macroeconomic indicators deteriorated after the series. For example, public debt increased to about 40 percent of GDP between 2010 and 2013, and then to 57 and 74 percent of GDP in 2014 and 2015, respectively. Allocation of budget resources to priority sectors was close to the 65 percent target during and after the period of PRSCs 3–5, but there was no 247 APPENDIX J IEG NOTE ON GLOBAL PRACTICES information on the distribution of these funds to the provinces or districts where the poor are located. Decentralization reforms to enhance public investments and service delivery at the provincial and district levels were modestly achieved, but substantial gains were made in the real sector through improving the business environment, removing constraints to growth, and promoting agricultural growth. 17. Gravely reminiscent of the 1984 famine, about 20 million people in Ethiopia faced destitution and starvation in 2003 after the sixth poor rainy season in three years. In the aftermath of this crisis, the World Bank helped to develop and fund a program that would increase the resilience of Ethiopia’s 12 million livestock- dependent pastoralist and agro-pastoralist communities. The Pastoral Community Development Program implemented during 2003–08 (Phase I) and 2008–13 (Phase II) provided basic social services, productive activities, an early warning system, and mechanisms for mobilizing group savings and lending. Overall, the assessment found that project support for pastoral development should take place within the context of a sound sector strategy that can offer well-researched, analytical advice about climatic and environmental conditions, and the implications of shifting social, economic, and demographic pressures and trends. Most of the project’s assistance focused on the provision of social infrastructure and some limited economic infrastructure, and the two projects assessed made a significant contribution to meeting critical social and economic infrastructure needs. But a sustainable livelihood-centered approach would have required a much broader range of coordinated investments, considering the combination of investments package needed to elevate livelihoods significantly. Given the constraints and challenges of pastoralist communities, a sustainable livelihood approach would focus on the livestock value chain, rangeland management, access to water and grazing land, and development of supplementary income sources in the non-farm economy. 18. In Brazil, IEG assessed the Ceará Rural Poverty Reduction Project (2001–09) and the Santa Catarina Natural Resource Management and Rural Poverty Reduction Project (2002–09) comparatively. The two projects are comparable examples of community-driven development approaches to rural poverty reduction under different conditions. Both projects sought to improve the rural poor’s incomes and living conditions, though the Santa Catarina project had an integrated environmental focus. But poverty conditions in the two states differed greatly—at approval, Ceará had the second largest rural poverty rate in Brazil, and Santa Catarina had the lowest rate. The Ceará project substantially improved the poor’s access to water and electricity and, to some extent, productive capacity (mainly tractors). The project made positive and significant impacts for employment and income obtained through enhanced farm production, but gains in income, though positive, were not statistically significant compared with a control group. Because 248 APPENDIX J IEG NOTE ON GLOBAL PRACTICES IEG assessed profitability for only four of the 264 productive subprojects (masonry, cashew nut marketing, hammock marketing, and irrigated agriculture), data were not available to validate the income effects of a statistically representative sample of productive investments, many of which were rural, non-farm activities. The Santa Catarina project substantially helped integrate environmental and social sustainability into the state’s development and poverty reduction strategies. However, it only modestly helped enhance local governance and community participation in decision making. The objective of reversing land degradation and achieving better protection of natural resources was modestly achieved based on data obtained on water quality and soil protection, and improvements to income- generating opportunities and living conditions for the rural poor were substantial. Energy 19. IEG produced a PPAR for the Turkey Environmental Sustainability and Energy Sector Development Policy Loan (DPL). Progress had stalled in the mid- 2000s on some aspects of Turkey’s market-oriented energy sector liberalization, and Turkey faced the prospect of electricity shortages. The World Bank aimed to alleviate these shortages with programmatic policy lending to support an increase in the pace of reforms and encourage investment, approving the Programmatic Electricity Sector DPL in 2009. During preparation, the government increased its prioritization of environmental issues after the opening of the environmental chapter of the EU acquisition and accession to the Kyoto Protocol, so the World Bank expanded its energy program to include environmental issues, approving the Second and Third Environmental Sustainability and Energy Sector DPLs.1 20. The program’s power sector aspects were strong, with a coherent design of mutually reinforcing prior actions covering electricity sector market development, pricing reform, renewable energy development, distribution company privatization, energy efficiency, and others (though little was done to address issues in the gas sector). A long history of engagement, substantial analytic work, and the World Bank’s ability to act as an honest broker to government and private investors supported the overall effort. The reforms were largely successful and supported substantial private sector investment in generation and distribution capacity, which prevented significant shortages. However, the program’s environmental aspects were mixed. Environmental management’s main weaknesses were in capacity and enforcement, not policy, so increasing some standards had little impact. Some 1The program’s first operation was approved on June 11, 2009, and closed on December 31, 2009. The second operation was approved on June 15, 2010, and closed on December 31, 2010. The third operation was approved on March 27, 2012, and closed on June 30, 2013. 249 APPENDIX J IEG NOTE ON GLOBAL PRACTICES environmental policy reforms were relatively minor, but the World Bank DPL added modest value for others because government—motivated by EU harmonization—was already pursuing the reforms. It was more difficult to use policy lending to try to open an engagement on environmental issues without strong existing relationships or parallel technical assistance. Social, Urban, Rural, and Resilience 21. The Bangladesh Social Investment Program, implemented during 2003–12 and valued at $60 million, was designed to improve access to local infrastructure and basic services through the provision of effective and efficient financing mechanisms using a community-driven development approach. The project operated in two of the country’s poorest districts with limited access to infrastructure and basic services. However, a lack of poverty targeting at the community level and adoption and adherence to clear rules of engagement (including the use of social accountability tools) undermined the relevance of project design. However, the relevance of design was enhanced at mid-term when measures were taken to increase community participation—including women and youth—and to direct services to the poor. The original results framework did not align with the project objective. The objective of improving access to local infrastructure and basic services was substantially met, although the distributional impacts are unclear. Like other community-driven development programs that were operating when natural disasters hit, the project had potential for flexibility and use as a tool for disaster crisis response. With its extensive outreach, the program directed two additional rounds of financing to remote areas affected by flooding and a cyclone. However, the model was too rigid and the high level of additional finance available for the response outstripped capacity and required coverage in areas where the program, did not have a presence before the crisis. Transport 22. The PPAR for the Morocco Urban Transport Sector DPL—one of only two World Bank–financed transport DPLs—highlighted the importance of conducting a thorough analysis of the macroeconomic context and the necessary transport sector reforms to define the priority interventions and set the DPL objective. The DPL’s design benefited significantly from several years of high-value World Bank analytical and advisory work. The PPAR found that the DPL’s prior actions were critically needed to accelerate policy and institutional strengthening measures, raise the profile of key issues, coalesce stakeholders, set timelines, and monitor results. However, the additionality of the prior actions and the attribution of outcomes to the World Bank’s intervention are not fully clear. The government had already identified and initiated many of the prior actions before the DPL’s approval and 250 APPENDIX J IEG NOTE ON GLOBAL PRACTICES implementation, which leads to the counterfactual question of whether the reforms would have proceeded without the DPL. Furthermore, the choice to use the DPL instrument alone, without accompanying and much-needed investment project financing or technical assistance, detracted from the full achievement of several physical infrastructure outcomes. Consequently, sector governance and environmental sustainability improved, but increased supply and performance of urban transport services and infrastructure were only modestly achieved. 23. The PPAR highlighted the following useful lessons:  Broad and early participation from implementing agencies and local governments and strong government reform momentum already need to be in place at entry for DPLs to achieve results.  Flexible and adaptive responses by the government are required to continue pursuing the DPL’s objectives when specific actions do not achieve expected results.  The World Bank’s convening authority and intellectual leadership need to support the government’s work on consensus building and interagency coordination, especially at the initial stages of DPLs.  The World Bank has an important role in ensuring that all investment options are analyzed to select the least-cost options.  Sector DPLs could be more effective if accompanied by parallel investment lending and technical assistance, or if they are designed as a programmatic series that would expand the implementation time frame to several years. 24. IEG prepared a PPAR for the Senegal Urban Mobility Improvement Project to understand the factors behind the unsatisfactory outcomes of the World Bank’s first project in Senegal, which had the objective of improving the safety, efficiency, and quality (operational and environmental) of urban mobility in Greater Dakar. Given the growing crisis in urban mobility in Dakar, the project’s objective was substantially relevant to the World Bank’s country strategies and the government’s own transport sector policies. However, the project design, though modest in some respects, was complex and overambitious. The design tackled the road and railway subsectors and assigned project implementation to an agency created just before appraisal. The agency did not have the adequate mandate and resources to fulfill its designated regulatory and organizing authority functions. The project’s innovative minibus leasing pilot program (which replaced old minibuses with more efficient and environmentally clean vehicles) had substantial outcomes. Road safety outputs were achieved, but the outcomes were modest given the lack of road maintenance, rapid growth in vehicular traffic, and continued use by overloaded trucks, all of which negated many of the project’s benefits. Railway upgrading outcomes were not 251 APPENDIX J IEG NOTE ON GLOBAL PRACTICES achieved—suburban railway operations were worse when the project closed—and environmental outcomes were only partially achieved. 25. The project yielded the following useful lessons:  Innovative leasing mechanisms can be effective in replacing aging public transport fleets, but their success depends on operator inputs at the design stage, technical assistance to professionalize operators and drivers, and restructuring the network of informal transport operators.  Establishing an effective lead agency for urban transport planning and management requires strong and sustained support from the government and stakeholders.  Land use and transport planning need to be coordinated at the metropolitan scale, and spatial analysis needs to be mainstreamed into urban transport project design.  Adequate road maintenance is a key factor in achieving sustainable outcomes.  Rigorous M&E is crucial for setting priorities for policy and regulatory actions. 26. IEG assessed urban transport projects in Lagos, Nigeria, and Mumbai, India, as an input into the ongoing evaluation of the World Bank Group’s support for urban transport development. Lagos is the largest city in Sub-Saharan Africa. The Lagos metropolitan area suffers from unreliable public transportation and chronic traffic congestion. Consequently, the World Bank supported the Lagos Urban Transport Project to improve transportation services for public transportation users, especially the poor, through five subobjectives: improvement in the metropolitan transport sector management, road network enhancement, more efficient public transportation services, promotion of water transportation, and preparation for future phases of the urban transport program. 27. The IEG assessment of urban transport projects drew the following lessons:  Establishing a strong, single authority (the Lagos Metropolitan Transport Authority) enabled a coordinated approach to planning and regulation.  The World Bank continued to support the initiative with continuous engagement through a follow-on project, which was crucial because urban transport reforms require long periods of intervention and support.  The initiatives were underpinned by the creation of a Transport Fund, which contributed to greater sustainability. 252 APPENDIX J IEG NOTE ON GLOBAL PRACTICES  World Bank supervision was sufficiently flexible to adjust the project when an opportunity arose to support the design of a bus rapid transit system (this program was implemented successfully).  A crucial factor in bringing about change was encouraging local stakeholders to observe similar successful systems in other developing countries. 28. Mumbai is one of the world’s top 10 centers for commerce measured by financial flows. The Mumbai Metropolitan Region has a population of more than 20 million, and the rail backbone of the transportation system carries more than 7 million passengers daily. The objectives of the Mumbai Urban Transport Project were to facilitate urban economic growth and improve the quality of life by fostering the development of an efficient and sustainable transportation system, including effective institutions to meet users’ needs. To improve public transportation, it was necessary to relocate more than 100,000 people, many of whom lived in squatter settlements by the rail tracks. This was the World Bank’s largest urban resettlement project. 29. IEG’s assessment of the Mumbai Urban Transport Project highlighted the following lessons:  Project development objectives needed to be more specific and measurable.  Large-scale resettlement is a complex operation that needs considerable planning, adequate time, good negotiation skills, and strong capacity to implement successfully.  Not adequately synchronizing resettlement activities with civil works can lead to significant time and cost overruns.  Transition from an administrative compensation approach to a win-win negotiated approach can resolve even the most seemingly intractable disputes between affected parties.  Introducing entitlements and market-based solutions were major contributors to resettlement success. Water 30. IEG assessed two water projects in Ghana. The objectives of the Second Urban Environmental Sanitation Project were to improve urban living conditions in Accra, Kumasi, Sekondi-Takoradi, Tamale, and Tema sustainably pertaining to environmental health, sanitation, drainage, vehicular access, and solid waste management, with special emphasis on the poor. Unsanitary landfills were closed or improved, and staff was trained to operate new equipment. Environmental conditions improved through closure of unsafe or unsanitary dumps and the opening of new landfills. Scavengers from the old waste dumps were organized into 253 APPENDIX J IEG NOTE ON GLOBAL PRACTICES associations and trained to maintain basic safety standards, such as using protective gear. The project’s investments in solid waste management, sanitation, and training the environmental health units’ public health and environmental health staff are likely to contribute to environmental health outcomes for reduced diarrheal diseases and vector-borne diseases. However, the extent of the improvement in environmental health is unknown because the project did not include an indicator to track this outcome. 31. The objective of the Small Towns Water Supply and Sanitation Project was to increase access to a sustainable water supply and sanitation services in small towns in six regions in Ghana. The project increased access to sustainable water supply facilities, which represented about 85 percent of the investments and about 90 percent of beneficiaries. Access to sanitation increased, but not sustainably because the water supply sector has stronger institutional support than the sanitation sector. 32. IEG’s PPAR for the project drew the following lessons:  To use school toilets sustainably, an integrated hygiene education program needs to be offered on a continuous basis. Sustained provision of hygiene education (availability of information, and soap and water near toilets) ensures that incoming classes continue to learn and use safe hygiene practices.  Stakeholder analysis and citizen engagement during project and facility design is important for assessing the willingness to pay for the services.  Changing the rules of the game for short-term political gains during implementation disrupts community involvement and sends the wrong signal to communities about government intentions. In the Small Towns Water and Sanitation Project, the decision to exempt communities from the 5 percent copayment requirement alienated communities that paid it, and may increase resistance to paying other obligations if they anticipate more changes in government policy. 33. IEG prepared a cluster PPAR in Peru for the Lima Water Rehabilitation and Management Project and the National Rural Water Supply and Sanitation Project. The objective of the Lima Water Rehabilitation and Management Project was to improve the efficiency of water and sanitation delivery in the Lima-Callao metropolitan area. The project sought to promote water conservation, support the privatization of the Lima water utility, SEDAPAL, rehabilitate damaged water supply and sewerage systems, expand services to the urban poor in low-income neighborhoods, and support reforms in the water and sanitation sectors’ legal and institutional framework. The PPAR found that the momentum of privatizing 254 APPENDIX J IEG NOTE ON GLOBAL PRACTICES SEDAPAL was diminished by the existence of a back-up plan to strengthen its capacity if the privatization did not go through. The objective of the National Rural Water Supply and Sanitation Project was to increase the sustainable use of water supply and sanitation facilities in rural areas and small towns while emphasizing improvement in hygiene practices and training in operations and maintenance. Despite restructuring at project appraisal, there was no clear policy and institutional context to benchmark the country’s priorities and strategies for rural water and sanitation services. Toward the end of the project, the government rolled back the principle of cost recovery in providing infrastructure, which was out of line with a basic premise of the project. 34. Lessons from IEG’s assessment include the following:  A strong focus on project outcomes and adequate accountability mechanisms are key factors in achieving significant improvements in service provision. Project design needs to address the gaps and weaknesses in sector institutions and governance identified during project preparation, because not doing so increases the risk to achieving project outcomes and prompt implementation.  Efforts to replicate successful experiences from other contexts must carefully consider the receptivity of the implementing institutions and beneficiaries. The modalities of cost sharing and community participation need to be adapted to the local context, and supplementary resources made available as needed.  Conserving water resources through demand and supply management can yield quick benefits to liberate water resources for new clients.  Community participation in planning and operations underpins sustainability, but adequate training support, financial incentives, and contractual arrangements are necessary for continued and effective participation. Behavioral change for improved water and sanitation-related hygiene requires ongoing engagement with the community and collaboration with health and education sectors. 35. Finally, with specific reference to Climate Change, one of the two cross- cutting solution areas of relevance under this cluster, IEG evaluated the Indonesia Climate Change DPL. Indonesia is the third largest emitter of greenhouse gases in the developing world after China and India, but until recently, the World Bank had avoided involvement in climate change policy lending to Indonesia established by other donors because it did not see the opportunity for ambitious reforms. However, the World Bank joined the ongoing program of support by approving the Indonesia Climate Change DPL in 2010, after the government showed a willingness to make reforms on energy pricing and forest governance to support its pledge to cut 255 APPENDIX J IEG NOTE ON GLOBAL PRACTICES emissions substantially. The operation was planned to be the first in a series of four operations covering policy actions on climate change mitigation, adaptation, and disaster preparedness, along with cross-sectoral and institutional issues. However, the series did not proceed past the first operation because of several factors, including the loss of critical reform champions, availability of budget support from other financing sources, availability of other climate finance through the UN’s Reducing Emissions from Deforestation and Forest Degradation Program, and noncompletion of some policy actions. Although performance in some policy subareas was generally positive (such as those related to renewable energy, water resource management, and natural disaster risk management), it was less true in other subareas, especially those concerned with peatland conservation, the UN program, and forest governance. Political economy barriers were substantial. For example, local governments did not implement central government decisions when those decisions challenged local economic and political interests (peatland use, palm oil concessions, and curbing fires and deforestation), and the public showed strong resistance to cutting energy subsidies. Although the DPL was somewhat unsuccessful in achieving its objectives, it had a positive and strategically important role as part of an evolving long-term policy dialogue with the World Bank on climate change issues. 256 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 1. a. Percentage of Projects rated MS+ on Outcome in the Sustainable Development cluster: IPF versus DPF IPF 100 DPF 83 85 85 83 82 82 80 80 78 Percentage 80 74 71 71 69 76 78 77 77 74 60 68 69 71 50 65 63 58 40 52 b. Percentage of IPF Projects rated MS+ on Outcome: Sustainable Development versus other clusters 100 85 85 SD Cluster 83 80 76 78 Other Clusters 74 Percentage 80 74 71 71 71 69 69 73 71 71 72 71 71 60 66 66 66 67 67 67 65 40 c. Percentage of DPF Projects rated MS+ on Outcome: Sustainable Development versus other clusters SD Cluster 100 Other Clusters 82 82 82 84 83 81 80 81 80 80 80 Percentage 77 76 80 81 83 77 78 77 77 76 60 68 65 63 58 40 50 52 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with a lending instrument other than IPF or DPF. DPF = development project financing; IPF = investment project financing; MS+= moderately satisfactory or better; SD = sustainable development. 257 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 2. a. Percentage of Projects rated MS+ on Outcome in the Sustainable Development cluster: IBRD versus IDA IBRD IDA 100 89 89 84 80 78 78 77 Percentage 76 74 72 73 76 80 71 78 82 81 81 78 60 72 74 73 71 70 67 66 69 40 b. Percentage of IBRD Projects rated MS+ on Outcome: Sustainable Development versus other Clusters SD Cluster Other Clusters 100 89 89 84 80 82 79 78 80 80 77 78 Percentage 76 75 80 76 78 79 80 60 76 75 73 73 71 70 67 69 66 40 c. Percentage of IDA Projects rated MS+ on Outcome: Sustainable Development versus other clusters SD Cluster Other Clusters 100 78 82 81 81 78 77 Percentage 72 74 74 72 73 76 80 71 60 72 69 70 71 73 70 71 69 67 65 65 67 68 40 Source: IEG data. Note: The data excludes projects not mapped to a cluster or with an agreement type other than IBRD or IDA. MS+ = moderately satisfactory or better; SD = sustainable development. 258 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 3. Project Outcome Ratings for the Sustainable Development Cluster by Global Practice (Closing FY13–15) Rated MS+ (%) by Number of Projects Rated MS+ (%) by Net Commitments Agriculture 79% 90% Energy & Extractives 66% 80% Environment & Natural Resources 62% 81% Social, Urban, Rural and Resilience 71% 81% Transport & ICT 79% 89% Water 68% 73% Sustainable Development Cluster 71% 83% Other Clusters 73% 90% Bank-wide 72% 87% Source: IEG data. Note: The data excludes projects not mapped to a cluster. ICT = information and communication technologies; MS+ = moderately satisfactory or better. Figure 4. Project Outcomes Rated MS+ (%) by Region: Sustainable Development Cluster versus Other Clusters, FY13–15 Sustainable Development Cluster Other Clusters Africa 66% 34% 70% 30% East Asia and Pacific 76% 24% 69% 31% Europe and Central Asia 79% 21% 79% 21% Latin America and the Caribbean 68% 32% 79% 21% Middle East and North Africa 59% 41% 63% 38% South Asia 84% 16% 79% 21% All Regions 71% 29% 73% 27% Source: IEG data. Note: The data excludes projects not mapped to a cluster. MS+ = moderately satisfactory or better. 259 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Figure 5. M&E Quality Rated Substantial+ (%) for the Sustainable Development Cluster by Global Practice (Closing FY13–15) Energy & Extractives 47% N=72 Agriculture 29% N=73 Social, Urban, Rural and Resilience 27% N=96 Water 24% N=71 Environment & Natural Resources 22% N=50 Transport & ICT 22% N=78 Sustainable Development Cluster 29% N=440 Other Clusters 37% N=348 Bank-wide 32% N=788 Source: IEG data. Note: The data excludes projects not mapped to a cluster. ICT = information and communication technologies; M&E = monitoring and evaluation; N = total number of projects rated by IEG on M&E quality. 260 APPENDIX J IEG NOTE ON GLOBAL PRACTICES Table 1. IBRD and IDA Lending Commitments in the Sustainable Development Cluster, FY12–16 2012 2013 2014 2015 2016 $, % of $, % of $, % of $, % of $, % of millions cluster millions cluster millions cluster millions cluster millions cluster total total total total total Agriculture 2,897 15 1,989 12 1,872 8 3,551 16 1,311 5 Energy and Extractives 4,958 25 3,122 19 6,414 27 4,320 20 6,722 27 Environment and Natural Resources 1,550 8 420 3 566 2 567 3 2,397 10 Social, Urban, Rural, and Resilience 4,142 21 4,383 27 4,437 18 5,163 23 3,613 15 Transport and ICT 3,870 20 4,784 29 6,722 28 5,035 23 5,678 23 Water 2,344 12 1,547 10 4,105 17 3,346 15 4,958 20 $, % of $, % of $, % of $, % of $, % of millions total millions total millions total millions total millions total IBRD IBRD IBRD IBRD IBRD and and and and and IDA IDA IDA IDA IDA Sustainable Development cluster 19,761 56 16,246 51 24,116 59 21,981 52 24,680 54 Other clusters 15,495 44 15,302 49 16,565 41 20,514 48 21,220 46 Source: World Bank Business Intelligence. 261 Bibliography ADB (Asian Development Bank). 2011. Framework for Results-Based Public Sector Management and Country Cases. Asia-Pacific Community of Practice on Managing for Development Results. Manila: ADB. ADB, IMF (International Monetary Fund), and World Bank. 2010. The Kyrgyz Republic Joint Economic Assessment. Manila: ADB. Andrews, Matt, Lant Pritchett, and Michael Woolcock. 2016. “Scaling PDIA Solutions through Broad Agency, and Your Role.” Harvard University Center for International Development (CID) Working Paper No. 315, Harvard University, Cambridge, MA. Argyris, Chris, and Donald A. Schön. 1974. Theory in Practice: Increasing Professional Effectiveness. San Francisco: Jossey-Bass. ———. 1978. Organizational Learning: A Theory of Action Perspective. Reading, MA: Addison-Wesley. Asian Development Bank, International Monetary Fund, and the World Bank. 2010. The Kyrgyz Republic Joint Economic Assessment: Reconciliation, Recovery, and Reconstruction . Washington, DC: World Bank. Blanc, Marc, Talib Esmail, Caroline Mascarell, and Rukshan Rodriguez. 2016. “ Predicting Project Outcomes: A Simple Methodology for Predictions Based on Project Ratings.” Policy Research Working Paper No. WPS7800, World Bank, Washington, DC. Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2011. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Policy Research Working Paper 5646, World Bank, Washington, DC. Drucker, Peter F. 1954. The Practice of Management. New York: Harper & Row. ———. 1964. Managing for Results: Economic Tasks and Risk-Taking Decisions. New York: Harper & Row. ———. 1989. “What Business Can Learn from Nonprofits.” Harvard Business Review 67 (4): 88–93. Fritsche, György Bèla, Robert Soeters, Bruno Meessen, Cedric Ndizeye, Caryn Bredenkamp, and Godelieve van Heteren. 2014. Performance-Based Financing Toolkit. Washington, DC: World Bank. Gaarder, Marie Moland, and Ulrich Bartsch. 2015. “Creating a Market for Outcomes: Shopping for Solutions.” Journal of Development Effectiveness 7 (3): 304–16. GAO (U.S. General Accounting Office). 1998. “Managing for Results: Measuring Program Results that are Under Limited Federal Control.” Report to the U.S. Senate Committee on Labor and Human Resources, Publication No. GAO/GGD-99-16, GAO, Washington, DC. ———. 2002. “Managing for Results: Challenges Agencies Face in Producing Credible Performance Information.” Report to Congressional Requesters, Publication No. GAO/GGD-00-52, GAO, Washington, DC. ———. 2015. “Managing for Results: Practices for Effective Agency Strategic Reviews.” Report to Congressional Committees, Publication No. GAO-15-602, GAO, Washington DC. Garvin, David A. 2000. Learning in Action—A Guide to Putting the Learning Organization to Work. Cambridge, MA: Harvard Business School Press. 262 BIBLIOGRAPHY Moll, Peter, Patricia Geli, and Pablo Saavedra. 2015. Correlates of Success in World Bank Development Policy Lending. Policy Research Working Paper 7181, World Bank Group, Washington, DC. Gigler, Björn-Sören, and Savita Bailur, eds. 2014. Closing the Feedback Loop: Can Technology Bridge the Accountability Gap? Washington DC: World Bank. Hatry, Harry P., Elaine Morley, Shelli B. Rossman, and Joseph S. Wholey. 2003. How Federal Programs Use Outcome Information: Opportunities for Federal Managers. Washington, DC: IBM Center for the Business of Government. IAD (World Bank Internal Audit Department). 2015. “Final Report on an Advisory Review of the Information Quality Supporting the Bank’s Portfolio Monitoring.” World Bank, Washington, DC. IDA (International Development Association). 2006. “IDA14 Results Measurement System: Mid-Term Review Report.” Board Report No. 37957, World Bank, Washington, DC. Independent Evaluation Group (IEG). 2008. Biennial Report on Operations Evaluation in IFC 2008: Enhancing Monitoring and Evaluation for Better Results. Washington, DC: World Bank. ———. 2011a. Assessing IFC’s Poverty Focus and Results. Washington, DC: World Bank. ———. 2011b. Social Safety Nets: An Evaluation of World Bank Support, 2000 –2010. Washington, DC: World Bank. ———. 2012a. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness . Washington, DC: World Bank. ———. 2012b. World Bank Group Impact Evaluations: Relevance and Effectiveness. Washington, DC: World Bank. ———. 2013a. Biennial Report on Operations Evaluation: Assessing the Monitoring and Evaluation Systems of IFC and MIGA. Washington, DC: World Bank. ———. 2013b. Results and Performance of the World Bank Group 2012. Washington, DC: World Bank. ———. 2013c. World Bank Group Support for Innovation and Entrepreneurship: An Independent Evaluation . Washington, DC: World Bank. ———. 2014a. The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises, 2006–12. Washington, DC: World Bank. ———. 2014b. Learning and Results in World Bank Operations: How the World Bank Learns. Washington, DC: World Bank. ———. 2014c. Results and Performance of the World Bank Group 2013. Washington, DC: World Bank. ———. 2014d. World Bank Group Support to Health Financing. Washington, DC: World Bank. ———. 2015a. Africa Microfinance Program Review. Washington, DC: World Bank. ———. 2015b. Financial Inclusion: A Foothold on the Ladder toward Prosperity? An IEG Evaluation of World Bank Group Support for Financial Inclusion for Low-Income Households and Microenterprises. Washington, DC: World Bank. ———. 2015c. How Does Knowledge on Public Expenditures Integrate with the Design of Development Policy Operations? Washington, DC: World Bank. ———. 2015d. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations. Washington, DC: World Bank. ———. 2015e. Investments in Renewable Energy Generation. Washington, DC: World Bank. 263 BIBLIOGRAPHY ———. 2015f. Later Impacts of Early Childhood Interventions: A Systematic Review . Washington, DC: World Bank. ———. 2015g. Learning and Results in World Bank Operations: Toward a New Learning Strategy . Washington, DC: World Bank. ———. 2015h. Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT). Washington, DC: World Bank. ———. 2015i. Managing Environmental and Social Risks in Development Policy Financing. Washington, DC: World Bank. ———. 2015j. The Poverty Focus of Country Programs: Lessons from World Bank Experience . Washington, DC: World Bank. ———. 2015k. Quality of Macro-Fiscal Frameworks in Development Policy Operations. Washington, DC: World Bank. ———. 2015l. The Quality of Results Frameworks in Development Policy Operations. Washington, DC: World Bank. ———. 2015m. Results and Performance of the World Bank Group 2014. Washington, DC: World Bank. ———. 2015n. Results Frameworks in Country Strategies: Lessons from Evaluations. Washington, DC: World Bank. ———. 2015o. Selectivity in Country Strategies: The Evidence. Washington, DC: World Bank. ———. 2015p. World Bank Group Engagement in Resource-Rich Developing Countries: The Cases of the Plurinational State of Bolivia, Kazakhstan, Mongolia, and Zambia. Clustered Country Program Evaluation Synthesis Report. Washington, DC: World Bank. ———. 2015q. World Bank Group Support to Electricity Access, FY2000 –2014: An Independent Evaluation. Washington, DC: World Bank. ———. 2015r. World Bank Group Support to Electricity Access, FY2000-FY2014: An Independent Evaluation, Volume II: Together for Energy: How Partnership Programs Support Energy Access . Washington, DC: World Bank. ———. 2015s. World Bank Support to Early Childhood Development: An Independent Evaluation. Washington, DC: World Bank. ———. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Washington, DC: World Bank. ———. 2016b. Delivering the Millennium Development Goals to Reduce Maternal and Child Mortality: A Systematic Review of Impact Evaluation Evidence. Washington, DC: World Bank. ———. 2016c. Financial Viability of the Electricity Sector in Developing Countries: Recent Trends and Effectiveness of World Bank Interventions. Washington, DC: World Bank. ———. 2016d. Findings from Evaluations of Policy-Based Guarantees. Washington, DC: World Bank. ———. 2016e. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Washington, DC: World Bank. ———. 2016f. Knowledge-Based Country Programs: An Evaluation of World Bank Group Experience. Washington, DC: World Bank. ———. 2016g. Learning from IDA Experience: Lessons from IEG Evaluations. Washington, DC: World Bank. 264 BIBLIOGRAPHY ———. 2016h. Learning in IFC. Washington, DC: World Bank. ———. 2016i. Lessons from Environmental Policy Lending. Washington, DC: World Bank. ———. 2016j. Lessons from Land Administration Projects: A Review of Project Performance Assessments .” Washington, DC: World Bank. ———. 2016k. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument. Washington, DC: World Bank. ———. 2016l. Public-Private Partnerships in Health: World Bank Group Engagement in Health PPP: An IEG Synthesis Report. Washington, DC: World Bank. ———. 2016m. Reliable and Affordable off-Grid Electricity Services for the Poor: Lessons from World Bank Group Experience. Washington, DC: World Bank. ———. 2016n. Results and Performance of the World Bank Group 2015: An Independent Evaluation. Washington, DC: World Bank. ———. 2016o. The Role of Political Economy Analysis in Development Policy Operations. Washington, DC: World Bank. ———. 2016p. Supporting Transformational Change for Poverty Reduction and Shared Prosperity: Lessons from World Bank Group Experience. Washington, DC: World Bank. ———. 2016q. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence: An Independent Evaluation. Washington, DC: World Bank. ———. 2016r. World Bank Group Engagement in Small States: The Cases of the OECS, PICs, Cabo Verde, Djibouti, Mauritius, and the Seychelles. Washington, DC: World Bank. ———. 2016s. World Bank Group Support for Housing Finance. Washington, DC: World Bank. ———. 2016t. The World Bank Group’s Support to Capital Market Development. Washington, DC: World Bank. ———. 2016u. The World Bank’s Risk Framework for Operations: Update on the First Year of Implementation. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/450751468184738008/The-World-Bank-s- risk-framework-for-operations-update-on-the-first-year-of-implementation IFC (International Finance Corporation). 2016. IFC Annual Report 2016: Experience Matters. Washington, DC: World Bank. IMF (International Monetary Fund). 2016. Regional Economic Outlook: Western Hemisphere: Managing Transitions and Risks. Washington DC: IMF. Kang, Young Cheoul. 2005. “Strategic Management in the Public Sector.” Public Performance and Management Review 29 (1): 85–92. Kiechel, Walter. 2012. “The Management Century.” Harvard Business Review 90 (11): 62–75, 148. Kumar, Anjali, Sushma Narain, and Swizen Rubbani. 2015. “ World Bank Lending for Financial Inclusion: Lessons from Reviews of Select Projects.” IEG Working Paper Series No. 2015/1, World Bank, Washington, DC. Lee, Mordecai. 2004. E-Reporting: Strengthening Democratic Accountability . Washington, DC: IBM Center for the Business of Government. MacKenzie, Alexander. 2008. Results Based Management at Country Level: Systemic Issues that Prevent Good UNDAF Results and the Use of UNDAF Results Information. New York: UNDAF (United Nations Development Assistance Framework). 265 BIBLIOGRAPHY Macroconsult-Cuanto. 2014. Elaboración de la Evaluación de Impacto y la Ampliación de la Línea de base del Programa de Transporte Rural Decentralizado (PTRD)—Informe Final. Lima, Peru: Macroconsult. Meier, Werner. 2003. “Results-Based Management: Toward a Common Understanding Among Development Cooperation Agencies.” Discussion Paper, Canadian International Development Agency, Ottawa, Ontario, Canada. MOPAN (Multilateral Organisation Performance Assessment Network). 2012. Assessment of Organisational Effectiveness and Development Results. AfDB (African Development Bank). Morra Imas, Linda G., and Ray C. Rist. 2009. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC: World Bank. Nutt, Paul C. 2006. “Comparing Public and Private Sector Decision-Making Practices.” Journal of Public Administration Research and Theory 16 (2): 289–318. OECD (Organisation for Economic Co-operation and Development). 2009. Managing for Development Results. Paris: OECD. OECD-DAC (Development Assistance Committee). 2002. Glossary of Key Terms in Evaluation and Results-Based Management. Washington, DC: OECD-DAC. ———. 2008. Emerging Good Practice in Managing for Development Results Source Book . Joint Venture on Managing for Development Results. Washington, DC: OECD-DAC. Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?” Policy Research Working Paper No. 7726, World Bank, Washington, DC. Robinson, Marc, and Jim Brumby. 2005. “Does Performance Budgeting Work? An Analytical Review of the Empirical Literature.” IMF (International Monetary Fund) Working Paper No. WP/05/210, IMF, Washington, DC. Schön, Donald A. 1973. Beyond the Stable State. Public and Private Learning in a Changing Society. Harmondsworth: Penguin. Senge, Peter M. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Doubleday Business. UNDG (United Nations Development Group). 2011. Results-Based Management Handbook: Harmonizing RBM Concepts and Approaches for Improved Development Results at Country Level. New York: UNDG. UNDP (United Nations Development Programme). 2007. Evaluation of Results-Based Management at UNDP. New York: UNDP. U.S. Department of State. 2015. Evaluation of the Office of U.S. Foreign Assistance Resources Implementation of the Managing for Results Framework. Washington, DC: U.S. Department of State, Office of U.S. Foreign Assistance Resources. Wholey, Joseph S. 2001. “Managing for Results: Roles for Evaluators in a New Management Era.” The American Journal of Evaluation 22 (3): 343–47. World Bank. 1998. 1998 Annual Report on Operations Evaluation. Washington, DC: World Bank. ———. 2002. “Better Measuring, Monitoring, and Managing for Development Results.” Board Report No. 30645, World Bank, Washington, DC. ———. 2006. “Accelerating the Results Agenda: Progress and Next Steps.” Report to the Committee on Development Effectiveness, World Bank, Washington, DC. 266 BIBLIOGRAPHY ———. 2011. The 2011 Multilateral Development Banks’ Common Performance Assessment System. Washington, DC: World Bank. ———. 2013. The World Bank Group Strategy 2013. Washington, DC: World Bank. ———. 2014. World Bank Group/World Bank Corporate Scorecard April 2014 . Washington, DC: World Bank. ———. 2015. World Bank Group/World Bank Corporate Scorecards October 2015. Washington, DC: World Bank. ———. 2016a. “Forward Look: A Vision for the World Bank Group in 2030.” Report to Governors at Annual Meetings 2016, World Bank, Washington, DC. ———. 2016b. Global Economic Prospects: Divergences and Risks. Washington, DC: World Bank. ———. 2016c. “New Framework for Corporate Scorecards, FY18.” Seminar at technical briefing, “FY18 World Bank Group and World Bank Corporate Scorecards,” Washington, DC, September 19, 2016. ———. 2016d. Poverty and Shared Prosperity 2016: Taking on Inequality . Washington, DC: World Bank. ———. 2016e. “Regional Update 2016—Africa: Sustaining Growth and Fighting Poverty amid Rising Global Risks." Presentation at an informal Board meeting, Washington, DC, February 17, 2016. ———. 2016f. “Regional Update 2016—East Asia and Pacific: Asia on the Rise." Presentation at an informal Board meeting, Washington, DC, February 17, 2016. ———. 2016g. “Regional Update 2016—Europe and Central Asia: Adjusting to the New Normal and Managing Risks." Presentation at an informal Board meeting, Washington, DC, February 17, 2016. ———. 2016h. “Regional Update 2016—Latin America and the Caribbean: Reviving Growth." Presentation at an informal Board meeting, Washington, DC, February 18, 2016. ———. 2016i. “Regional Update 2016—Middle East and North Africa: Economic and Social Inclusion for Peace and Stability." Presentation at an informal Board meeting, Washington, DC, February 16, 2016. ———. 2016j. “Regional Update 2016—South Asia Region: Moving Toward the Next Level of Development." Presentation at an informal Board meeting, Washington, DC, February 16, 2016. ———. 2016k. World Bank Group/World Bank Corporate Scorecards April 2016. Washington, DC: World Bank. ———. 2016l. World Bank Group/World Bank Corporate Scorecards October 2016. Washington, DC: World Bank. ———. 2017. World Development Report. Washington, DC: World Bank. 267