STRENGTHENING WORLD BANK GROUP OUTCOME ORIENTATION- A Roadmap STRENGTHENING WORLD BANK GROUP OUTCOME ORIENTATION- A Roadmap June 2021 Operations Policy and Country Services The World Bank @ June 2021. International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank with external contributions. Note that The World Bank does not necessarily own each component of the content included in the work. The World Bank therefore does not warrant that the use of the content contained in the work will not infringe on the rights of third parties. The risk of claims resulting from such infringement rests solely with you. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 Unported license (CC BY 3.0) http://creativecommons.org/licenses/by/3.0. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: The World Bank Group. 2021. Strengthening World Bank Group Outcome Orientation: A Roadmap. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0 Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Rights and Permissions Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; e-mail: pubrights@worldbank.org. Cover Photo: Laos: Nutritious meals are bringing more children to school. Before they eat, students like these in the Had Ane Primary School of Oudomxay Province are taught the importance of proper hand washing, drinking safe water, and using clean utensils. Oudomxay province, Lao PDR. Photo by Bart Verweij / World Bank. Contents Acknowledgements ................................................................................................................................. ii Abbreviations ......................................................................................................................................... iii Glossary .................................................................................................................................................. iv Executive Summary .................................................................................................................................... v I. Background ..................................................................................................................................... 1 II. What the WBG means by outcome-orientation ........................................................................... 3 III. How the WBG aims for high-level outcomes ................................................................................ 8 IV. Moving towards better articulation of high-level outcomes ...................................................... 11 V. Adjusting the country engagement model to account for high-level outcomes ....................... 15 VI. Applying the new country engagement guidance to FCV ......................................................... 20 VII. Reinforcing incentives for high-level outcomes .......................................................................... 22 VIII. Conclusion and next steps ............................................................................................................ 25 Annex A: Outcome Orientation Roadmap ......................................................................................... 27 Annex B: Outcome Orientation During COVID19 Response........................................................... 30 Annex C: Summary Findings from Country Outcome Pilots ........................................................... 32 Selected References ................................................................................................................................ 45 i Acknowledgements The task was led by Karima Saleh, with team members including, Brett Libresco, Kamal Siblini, Debra Ladner, Philip Jespersen, Peter Isabirye (OPSRS). Inputs were received from Christopher Finch (OPSRR), Michael Geiger, Kamer Ozdemir, Mariela Alanes (OPSPO); Sabine Durier, Luciana Harrington, Junko Oikawa (IFC); Hiroyuki Hatashima, Vlajko Senic (MIGA). Graphics support was provided by Marianna Camino, and formatting support was provided by Mimoza Velo. The four country outcome pilots were led by Karima Saleh and Peter Isabirye (OPSRS) in collaboration with the operations teams: Tazeen Fasih and Koen Gevens (HSAED); Ralph Van Doorn and Anthony Obeyesekere (EEAM2); Laurencia Karimi Njagi and Zubair K. M. Sadeque (IAEE1); Livia M. Benavides (HLCDR). Other team members for the country pilots included: Patrick Thaddayos Balla, Alessandra Marini, Hugo Brousset Chaman, Nelson Gutierrez, Claudia Rokx, Brett Libresco, Naresha Duraiswamy, Tsegaye Anduanbessa Assayew, Sara Okada, and Tamara Arnold Urzua. The task was conducted under the overall guidance of Manuela Ferro (OPCS) and Johannes Zutt (OPSRL), and under the supervision of Lisandro Martin, Manager (OPSRS). The team is grateful for contributions from Stephane Guimbert and Christine Richaud (OPSPO). Comments were also received from Mona Haddad (IFC) and Merli Baroudi (MIGA). The paper benefitted from consultations with the Operations Board and the Operations Council. The Independent Evaluation Group (IEG) was also consulted. The paper was prepared for presentation to the Committee for Development Effectiveness (CODE) in June 2021.1 1 Since July 1st, 2021, updated Country Engagement guidance for the World Bank Group (WBG) has been issued that addresses several of the issues identified in this paper. These introduced changes are not reflected in this paper; observations on the WBG Country Engagement refer to the applicable guidance prior to July 1 st, 2021. ii Abbreviations ADB Asian Development Bank AfDB African Development Bank AIMM Anticipated Impact Measurement and Monitoring ASA Advisory Services and Analytics CLR Completion and Learning Review CODE Committee for Development Effectiveness CPF Country Partnership Framework CPPR Country Portfolio Performance Review CPSD Country Private Sector Diagnostic DPL Development Policy Lending FCV Fragility, Conflict, and Violence GRID Green, Resilient, and Inclusive Development IBRD International Bank for Reconstruction and Development IDA International Development Association IFC International Finance Corporation IEG Independent Evaluation Group IMF International Monetary Fund IMPACT Impact Measurement and Project Assessment Comparison Tool IPF Investment Project Financing ISR Implementation Status Results Report JET Jobs and Economic Transformation MDB Multilateral Development Bank M&E Monitoring and Evaluation MFD Maximizing Finance for Development MfDR Managing for Development Results MIGA Multilateral Investment Guarantees Agency MOPAN Multilateral Organization Performance Assessment Network OECD Organization for Economic Cooperation and Development PLR Performance and Learning Review PPP Public-private partnership RMES Results Measurement and Evidence Stream RMS Results Measurement System SCD Systematic Country Diagnostic SDG Sustainable Development Goal SORT Systematic Operations Risk-rating Tool WBG World Bank Group iii Glossary Adaptive management: Adaptive management is defined in ADS 201.6 as “an intentional approach to making decisions and adjustments in response to new information and changes in context.” Adaptive management is not about changing goals during implementation, it is about changing the path being used to achieve the goals in response to changes.2 Contribution analysis: a methodology used to identify the contribution a development intervention has made to a change or set of changes in a complex environment. It aims to produce a credible, evidence-based narrative of contribution that a reasonable person would be likely to agree with, rather than to produce conclusive proof. Extrinsic incentives: external rewards and punishment (carrots and sticks) to motivate desired behaviors, for example tying salary increases to performance ratings or providing recognition and visibility. High level outcomes: a sustained improvement in the well-being of the poorest and most vulnerable people—for example, their health, security, mobility, opportunity, livelihood, or standard of living. Indirect pathways: institutional development, capacity building, knowledge transfer, demonstration effects, and market creation. Intrinsic motivations: enjoyment, meaning, or sense of accomplishment one derives from an activity. Line of sight: a clear path connecting an activity with its ultimate desired outcome. Outcome orientation: a continuous focus, in all public and private development activities, on helping the poorest and most vulnerable people to realize their full potential. Sentinel indicator: proxy indicator used not to measure a result, but rather as a bellwether for indicating that greater changes are occurring within a complex system. Strategic adaptations: involve higher level changes, for example adding or removing a development objective in response to new knowledge or emerging constraints, priorities, or opportunities. Tactical adaptations: focus on course corrections to navigate implementation challenges and ensure the achievement of existing objectives. 2 https://usaidlearninglab.org/lab-notes/what-adaptive-management-0 iv Executive Summary The COVID-19 global crisis provides further impetus for the World Bank Group (WBG) to ramp-up its efforts to strengthen outcome orientation. The WBG has responded rapidly to the crisis, mobilizing expertise and financing, coordinating support from multiple agencies, and serving as a trusted partner to clients. However, there is a need to help countries maintain focus on high-level outcomes that are achieved over the long run, even as they respond to the emergency of the moment. Such focus requires sustained investments in institutions, systems, and capabilities to deliver services to citizens. Nowhere is this more pronounced than in countries affected by fragility, conflict, and violence (FCV), where rapidly changing contexts can impede long-term planning. The WBG views outcome orientation as a continuous focus, in all its public and private development activities, on helping poor and vulnerable people improve their well-being to realize their full potential. Building on IEG’s classification for levels of results, high-level outcomes are sustained improvements in the well-being of the poorest and most vulnerable people—for example, their health, security, mobility, opportunity, livelihood, or standard of living. Achieving high-level outcomes often requires contributions from the public and private sectors. These are primarily drawn from a client country’s own development strategy and are aligned with the sustainable development goals (SDGs), for example increased private sector investment and jobs, reduced incidence of stunting, or increased incomes for farmers. The track record shows that coordination across the Bank, International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) maximizes the impact of WBG support. As the Independent Evaluation Group (IEG) and other observers have recognized, the WBG starts from a position of strength, as it already deploys a robust approach to aiming for outcomes. In a recent evaluation, IEG found that “the model of how the Bank Group aims for outcomes in its client countries is sound, and, for the most part, teams practice this well.”3 Yet the WBG has an opportunity to articulate a more wide-ranging narrative of the results it helps clients achieve, many of which are not easily quantifiable. Some critically important activities that contribute to improved well-being, for example Bank efforts to strengthen institutions and upstream IFC and MIGA support for policy and regulatory reforms to create markets and attract private investment and foreign capital, are among the most difficult to measure quantitatively and capture in reporting systems. In some cases, these activities are better suited to qualitative narratives. The biggest opportunity for the WBG is to strengthen its outcome orientation in country engagements where its results are predominantly delivered. Maximizing Finance for Development (MFD) is the WBG’s approach to systematically leverage all sources of finance, expertise, and solutions to support developing countries' sustainable growth. It is a holistic approach that assesses development challenges and leverages the private sector in sustainable ways that bolster or substitute for scarce public resources. Both the public and private sectors have distinct and important coordinated roles to play in helping countries achieve desired outcomes. This requires building a strong enabling environment with the right policy and governance structures in place, incentivizes for financing, space for skills, and opportunities for private sector engagement and market creation. If done correctly, growth in 3 Independent Evaluation Group. 2020a. The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation. World Bank, Washington, DC. October 16, 2020. v private sector engagement can help market creation and foreign direct investment, which could facilitate a country’s economic growth, create jobs, and meet development outcomes. The WBG uses the MFD principles as the conceptual basis to determine the role of the public and private sectors on a country-by-country basis. The emphasis on the MFD approach is critical to the achievement of country outcomes, involving both the public and private sector; it allows for a clear determination of the roles and its associated impact. WBG efforts to enhance outcome orientation will continue for as long as the WBG exists, as new goals, methods, and technologies evolve. The WBG has been improving its outcome orientation since it first explicitly introduced its results agenda in 2002, and it is currently engaged in about twenty activities that are seeking further improvements (see Annex A). These activities build on existing practices as well as the August 2020 paper, Strengthening Outcome Orientation: An Update, aiming to make operational the four pathways for outcome orientation presented to the Committee for Development Effectiveness (CODE) in FY20.4 They are a snapshot of what the WBG is doing in this area at present. Identifying what the WBG should do to help countries to achieve high-level outcomes is a key component of its outcome orientation, and this currently involves articulating both the COVID-19 response and the proposed policy commitments to be supported through the advanced 20th financing cycle of the International Development Association (IDA20). The WBG adapts its programs to help countries achieve high-level outcomes that are anchored in a country’s long-term development goals. These high-level outcomes are positioned around SDGs5, WBG twin goals, corporate priorities, and other country development priorities. As countries grow and develop in a changing world, and as they learn from experience, their development objectives will necessarily evolve. The WBG is a partner in this process. As a knowledge institution, it produces analytical work in large quantities at all levels, builds upon lesson learnt from operations, consolidates global know-how and best practices, and embeds the findings of that work in the country-level operations and investments that it finances. Generating and using this theoretical and practical knowledge is at the center of WBG operations, and it is constantly under review and revision. At present, the key medium-term focus is on addressing the impacts of the COVID-19 pandemic, implementing the special themes identified in the IDA18-19 replenishments (namely, climate change, gender, FCV, governance and institutions, and jobs and economic transformation), implementing the parallel commitments embedded in the International Bank for Reconstruction and Development (IBRD) and IFC capital-increase package, MIGA’s Strategy and Business Outlook FY21-23, and also extending and potentially redirecting IDA’s thematic directions during the IDA20 replenishment negotiations, which have been advanced into FY21-22. The roadmap in Annex A sets out some of the key workstreams in these areas. How the WBG aims to help countries to achieve their development goals is also a critical part of its outcome orientation, and a key aspect of this endeavor at present involves defining and implementing a refined approach to outcome orientation at the country-level. This effort grows out of the work done in the first work stream, exploratory country pilots that were 4 World Bank. 2020b. Strengthening Outcome Orientation: An Update. Document for discussion at a meeting of the Committee on Development Effectiveness on August 31, 2020. August 21, 2020. 5 Many but not all SDGs are high-level outcomes. For example, some SDGs, such as SDG 3.2 on under-5 mortality, are high-level outcomes. But others, such as SDG 6.5 on implementing integrated water resource management, are important steps toward achieving high-level outcomes, but not high-level outcomes themselves. vi undertaken following the July 2019 discussion with CODE.6 These so-called “outcome pilots” tested a retrospective methodology, using contribution analysis7 to track the WBG’s role in helping clients achieve selected high-level outcomes that aligned with at least one SDG indicator. The pilots show that the WBG uses scalable solutions, direct and indirect investments, risk calibration, combinations of lending and non-lending, and evidence collection to maintain a focus on intermediate outcomes that help countries, over time, to move towards a sustainable achievement of higher-level outcomes, increasingly using their own capacities and resources. As expected, while intermediate outcomes are relatively well addressed in country engagement documents, high-level outcomes are significantly less so. The work on the outcome pilots was undertaken as IEG completed its evaluation on The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation.8 The evaluation suggested that the WBG revise its country engagement instruments to incorporate more explicit attention to monitoring outcomes through a selection of country indicators (with no assumption of attribution or specific targets), key corporate indicators, and project-level indicators; evaluating the contribution of WBG activities to country outcomes as well as country portfolio results and stakeholder satisfaction; and learning, by stock-takings and mid-term reviews, to identify necessary adaptations as well as knowledge gaps and ways to fill them. This IEG work constituted a second work stream pointing to a need to review the fitness of country engagement guidance and to revise it to address its perceived shortcomings. A third work stream, intersecting the outcome pilots and the IEG evaluation, concerns measure 5 in the FCV strategy. This measure aims to “[e]nhance the WBG evaluation framework that assesses country programs and operations in FCV settings, including encouraging more realism in objective-setting and project design and implementation, as well as adaptability to dynamic circumstances and situations of low institutional capacity and high levels of uncertainty and risk.” This action points to enhanced guidance across the full cycle of country engagement products, a process to be integrated with similar efforts for other country groupings. The common thread across these three initiatives is that the WBG should adjust its country engagement guidance, including its results frameworks, to articulate more explicitly and purposefully a line of sight to selected high-level outcomes as well as a more structured approach to mid-course learning and adaptation during the implementation of supported activities. The purpose of these changes is to ensure that the WBG country programs more explicitly aim and manage for outcomes that matter, and that the results system more carefully captures the WBG’s contributions along the pathway to country outcomes, in line with IEG’s recommendations in the Outcome Orientation evaluation.9 To this end, the WBG is identifying appropriate changes in its country engagement guidance to articulate more explicitly and purposefully a long-term line of sight to selected high-level outcomes anchored in the country’s development priorities and to trace the WBGs’ contribution towards these outcomes. Following consultations with stakeholders, Management considers it more viable to adjust existing systems 6 World Bank. 2019a. Development Results in the World Bank Group. Executive Directors’ Seminar. Outcome Orientation in the World Bank Group. July 29, 2019. 7 Contribution analysis is a methodology used to identify the contribution a development intervention has made to a change or set of changes in a complex environment. It aims to produce a credible, evidence-based narrative of contribution that a reasonable person would be likely to agree with, rather than to produce conclusive proof. 8 Independent Evaluation Group, 2020a. 9 Independent Evaluation Group, 2020a. vii rather than creating new products and processes ex novo. With a strengthened focus on outcomes and tighter connection to higher-level outcomes, the Country Partnership Framework (CPF) results framework remains the cornerstone of the WBG’s results architecture at the country level. Key adjustments can shape the role of the results framework as an integral part of outcome orientation. Ensuring that the approach is implementable at scale with adequate rigor, Management will roll out the new guidance, with adjusted templates, training, and hands-on support in FY22. The following dimensions summarize the forthcoming adjustments, which, while relevant for all clients, are of particular significance in FCV countries. This will involve ensuring: (a) Focusing on a few high-level outcomes by improving the line of sight to high-level outcomes over time in the CPF narrative, and the time horizon extending beyond the CPF cycle, and maintaining selectivity of outcomes; (b) Connecting the results framework to high-level outcomes by adjusting the CPF results framework to match the new CPF line-of-sight narrative; (c) Adapting for high- level outcomes by client countries regularly reflecting on the country’s progress toward achieving selected high-level outcomes and adapting course as appropriate, (d) Assessing progress on the pathway towards high-level outcomes by, at the end of the CPF cycle, the Completion and Learning Review (CLR) contextualizing and putting in perspective the results achieved on the CPF objectives and progress observed on a subset of selected high-level outcomes; (e) Learning for high-level outcomes by leveraging better the knowledge gained from operational work to inform pathways, identifying key knowledge gaps, including to achieve the selected high level-outcomes, in a more structured way, and then selectively filling those gaps to inform government policy and programs. Finally, better articulating WBG contributions to high-level outcomes can reinforce positive behaviors, facilitating a more homogeneous approach to results across countries. This agenda is no longer one of creating a culture of results in operational teams, as it was two decades ago. Steady positive trends in outcome ratings for operations and CPFs show that teams aim for clear outcomes and strive for them. This is now an agenda of ensuring that existing country engagement guidance keeps up with the outcome orientation culture that predominates in operational teams. viii I. Background 1. The COVID-19 global crisis provides further impetus for the World Bank Group (WBG) to ramp-up its efforts to strengthen outcome orientation. The WBG has responded rapidly to the crisis, mobilizing expertise and financing, coordinating support from multiple agencies, and serving as a trusted partner to clients. Across the board, the WBG response also helped countries take the longer view, maintaining focus on high-level outcomes that are achieved over the long run, as they respond to the emergency of the moment. Such focus requires sustained investments in institutions, systems, capabilities, and market actors to deliver services to citizens. Nowhere is this more pronounced than in countries affected by conflict and violence (FCV), where rapidly changing contexts can obscure long-term planning. 2. As IEG and other observers have recognized, the WBG starts from a position of strength, as it already deploys a robust approach to aiming for outcomes . In a recent evaluation, IEG found that “the model of how the Bank Group aims for outcomes in its client countries is sound, and, for the most part, teams practice this well.”10 In addition, a set of outcome pilots that assessed the WBG’s contribution to high-level outcomes over time confirmed that view, finding that WBG teams typically set out to influence high-level country outcomes and do a good job of identifying appropriate intermediate outcomes, and then adapting their approach and adjusting their intermediate objectives considering higher-level objectives to respond to new evidence and new circumstances. Moreover, the WBG’s business model rests on working upstream to help country clients develop policies, systems, and capabilities and create markets so that its support is amplified well beyond the programs it finances. Teams understand well that a small improvement in the efficiency of a government program or the functioning of a market can leverage substantially the impact of investments, and so they aim to help strengthen markets and institutions to amplify their impact over time. 3. Yet the WBG has an opportunity to articulate a more wide-ranging narrative of the results it helps clients achieve, many of which are not easily quantifiable. Some of the outcomes that lead to improved well-being, through strengthened institutions and market creation, are among the most difficult to measure and capture in WBG reporting systems. To address this issue, the WBG has, with the support and guidance of the WBG’s Committee on Development Effectiveness (CODE), intensified its efforts to better capture its contributions towards high-level outcomes. Progress has been reported through three CODE engagements (July 29, 2019; April 22, 2020; and August 31, 2020).11 These laid out the customized approaches to outcome orientation in the Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantees Agency (MIGA) and also described how the Bank manages risk and contributes over time to high-level outcomes. At the request of CODE, the outcome roadmap presented in Annex A of this paper describes key milestones of the approach presented to CODE in FY20 in the 10 Independent Evaluation Group. 2020a. 11 World Bank. 2019a. Development Results in the World Bank Group. Executive Directors’ Seminar. Outcome Orientation in the World Bank Group. July 29, 2019; World Bank. 2020a. Focusing on Outcomes and Managing Risks: Helping Countries achieve the SDGs. Document for discussion at a meeting of the Committee on Development Effectiveness on April 22, 2020; World Bank. 2020b. Strengthening Outcome Orientation: An Update. Document for discussion at a meeting of the Committee on Development Effectiveness on August 31, 2020. August 21, 2020. 1 document Strengthening Outcome Orientation: An Update.12 IEG has significantly informed the WBG’s thinking on the matter, including through recent evaluations, direct consultations, and presentations to the Board and CODE. 4. Identifying what the WBG should do to help countries maintain a focus on high-level outcomes, and how it should do it, currently involves articulating both the COVID-19 response and, for the poorest countries, the proposed policy commitments to be supported through the advanced IDA 20 financing. As countries grow and develop in a changing world, and as they learn from experience, their development objectives will necessarily evolve. The WBG is a partner in this process. The WBG generates knowledge through multiple vehicles—in the preparation and implementation of operations, through staff experience, and in the large quantities of advisory, analytical, and diagnostic work at all levels covering both the public and private sectors—and embeds the findings of that work in the country-level operations and investments that it finances. As described in the WBG Strategic Framework for Knowledge13, generating and using this theoretical and practical knowledge is at the center of WBG operations, and it is constantly under review and revision. At present, the key medium-term focus is on addressing the impacts of the COVID-19 pandemic, implementing the special themes identified in the IDA18-19 replenishments (namely, climate change, gender, FCV, governance and institutions, and jobs and economic transformation), implementing the parallel commitments embedded in the IBRD and IFC capital-increase package, MIGA’s Strategy and Business Outlook FY21-23, and extending and potentially redirecting IDA’s thematic directions during the IDA20 replenishment negotiations, which have been advanced into fiscal year 2022 (FY22). (Annex B). 5. Over the last several months the WBG has benchmarked its practices with those of other development partners to inform three ongoing work streams on outcome orientation: • The first stream involves undertaking at the country level exploratory outcome pilots, in accordance with a July 2019 discussion with CODE. As discussed in section IV, these outcome pilots tested a retrospective methodology to track the WBG’s role in helping clients achieve selected high-level outcomes that aligned with at least one Sustainable Development Goals (SDG) indicator. • The work on the outcome pilots was undertaken as IEG completed its evaluation on The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation.14 The evaluation found that the WBG’s “country-level results system does not capture [its] contribution to country outcomes well because it relies on results frameworks premised on metrics, attribution, and time-boundedness”, and it recommended that the WBG reform the country engagement model “to ensure that it accurately captures the contribution to country outcomes.” This IEG work constituted a second work stream pointing to a need to review the fitness of country engagement guidance and to revise it to address its perceived shortcomings (see section III). 12 World Bank. 2020b. 13 World Bank. 2020e. Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework. World Bank. Washington, DC. 14 Independent Evaluation Group. 2020a. The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation. World Bank, Washington, DC. October 16, 2020. 2 • A third workstream, intersecting the outcome pilots and the IEG evaluation, concerns measure 5 in the fragility, conflict, and violence (FCV) strategy.15 This measure aims to “[e]nhance the WBG evaluation framework that assesses country programs and operations in FCV settings, including encouraging more realism in objective-setting and project design and implementation, as well as adaptability to dynamic circumstances and situations of low institutional capacity and high levels of uncertainty and risk.” While the description of the related activity in the FCV strategy points to enhanced guidance for Completion and Learning Reviews (CLRs), any changes in CLR guidance would need to integrate with broader changes in the country engagement guidance (see section VI). 6. Considering all three work streams together allowed Management to identify opportunities for further action informed by practices in other organizations. The common thread across these three initiatives is that the WBG should adjust its country engagement guidance to articulate more explicitly and purposefully a line of sight to selected high-level outcomes, and to maintain a focus on these high-level outcomes through mid-course learning and adaptation during implementation and over time. Several other workstreams—including the IDA replenishments, the IBRD-IFC capital increase package, MIGA’s Strategy and Business Outlook FY21-23, and the new strategic approaches in FCV settings, gender, climate, knowledge, and also green, resilient and inclusive development (GRID)—have been developed in tandem with the outcome orientation agenda and already encapsulate many of its principles. 7. This paper is divided into six sections. First it describes what the WBG means by outcome orientation, highlighting IEG’s classification of results and connecting it to the WBG’s program. Second, it details how the Bank aims for high-level outcomes, building on the findings of the recent set of outcome pilot studies. Third, it lays out how the Bank is moving towards greater articulation of high-level outcomes, introducing a framework for reporting on outcomes based on benchmarking and discussing a roadmap to spearhead the Bank’s efforts towards greater outcome orientation. Fourth, it details planned adjustments to the WBG’s country engagement approach to better articulate high-level outcomes -- a central activity of this roadmap. Fifth, it looks at the special case of countries affected by fragility, conflict, and violence. Finally, it examines how Management efforts are reinforcing incentives for high-level outcomes. II. What the WBG means by outcome-orientation 8. The WBG views outcome orientation as a continuous focus, in all its public and private development activities, on helping poor and vulnerable people improve their well-being to realize their full potential. Building on IEG’s classification for levels of results, high-level outcomes are sustained improvements in the well-being of the poorest and most vulnerable people—for example, their health, security, mobility, opportunity, livelihood, or standard of living (see figure 1). Achieving high-level outcomes often requires contributions from the public and private sectors. These are primarily drawn from a client country’s own development strategy and are aligned with the SDGs, for example increased private sector investment and jobs, reduced incidence of stunting, 15 World Bank. 2020d. Strategy for Fragility, Conflict, and Violence 2020–2025. World Bank. Washington, DC. February 27, 2020. 3 or increased incomes for farmers. The track record shows that coordination across the Bank, IFC, and MIGA maximizes the impact of WBG support. 9. High-level outcomes are transformative changes required to achieve the country’s long-term vision for development. They differ by country, depending on key development challenges and priorities. High-level outcomes are typically articulated in national development plans and align with the WBG’s twin goals and the SDGs. Achieving high-level outcomes generally requires progress across multiple sectors. As an example, reducing under five mortality rates can be achieved through combined efforts across a range of sectors, including health, water and sanitation, education, agriculture, and sometimes even transport. Some SDGs, such as SDG 3.2 on under-5 mortality, are high-level outcomes. But others, such as SDG 6.5 on implementing integrated water resource management, are important steps toward achieving high-level outcomes, but do not rise to that level themselves. Progress towards high-level outcomes, including many SDGs, is typically tracked through country systems, including national statistics and M&E systems. Figure 1: An outcome-oriented approach aims to improve people’s well-being Source: Adapted from Independent Evaluation Group’s classification presented in Independent Evaluation Group, 2020b. Note: In IEG’s classification, the highest level of outcomes includes improvements in delivery and governance in addition to well-being. We adopt a more limited definition, focused on sustained improvements in well-being. Improvements in delivery and governance are pathways to the desired change. 10. The importance of focusing on outcomes is not new but the emphasis is. The Aid Effectiveness Agenda has established a broad consensus on the need to orient how aid is managed to maximize results. Managing for development results is a principle articulated in key international commitments, including the Paris Declaration and the Accra Agenda for Action. The focus on outcomes is also a defining feature of the 2030 Agenda for Sustainable Development, which sets clear targets and indicators for each of the SDGs. The WBG has played a role in shaping these discussions and has aligned its architecture and practices towards managing for development results. Defining outcome orientation as helping poor and vulnerable people improve their well- 4 being to realize their full potential clarifies the ultimate direction of WBG interventions. Earlier efforts towards results orientation had a more flexible definition of the what, encouraging the focus at any of the four levels, prioritizing early wins and attribution. 11. Outcome orientation is not only about what but also about how development interventions lead to improved well-being for poor and vulnerable people to realize their full potential. An outcome-oriented approach aims to address barriers towards well-being, generate evidence on what works and does not work for helping the poor and vulnerable realize their full potential, and why, and use this evidence to help country teams adapt their engagements to enhance their contribution to achieving those high-level outcomes over time. Most behaviors that support the achievement of high-level outcomes are summarized in the Organization for Economic Cooperation and Development (OECD) Principles of Managing for Sustainable Development Results, recently revised through broad consultations.16 Successive IEG evaluations touch on one or more of these principles, including in their definition of outcome orientation. These principles aim to provide a framework and broad set of values to guide the refinement of results-based management systems, considering the evolving context and increasing complexity of development. The Bank is an early adopter of these principles through their successive revisions. Already in the early 2000s, the OECD acknowledged the Bank’s strategic approach to planning, implementing, and monitoring development efforts at the country level through a framework “based on principles of results-orientation, country ownership and participation, partnership, and a shared long-term holistic vision of development goals.”17 A set of recently conducted outcome pilots discussed in the next section illustrate how some of these principles materialize in country engagements. 12. Maximizing Finance for Development (MFD) is the WBG’s approach to systematically leverage all sources of finance, expertise, and solutions to support developing countries' sustainable growth. It is a holistic approach that assesses development challenges and leverages the private sector in sustainable ways that bolster scarce public resources. Both the public and private sectors have distinct and important coordinated roles to play in helping countries achieve desired outcomes. This requires building a strong enabling environment with the right policy and governance structures in place, incentivizes for financing, space for skills, and opportunities for private sector engagement and market creations. If done correctly, growth in private sector engagement can help market creation and foreign direct investment, which could facilitate country economic growth, create jobs, and meet development outcomes. The WBG uses the MFD principles as the conceptual basis to determine the role of the public and private sectors on a country-by-country basis. The MFD principles are: (i) support private investments where they can help achieve development goals in a way that provides value for money; (ii) create an enabling environment where private solutions are not feasible due to regulatory, policy, or capacity gaps; (iii) reduce risks with blended financing with the priority to apply public resources to reduce risks; and (iv) finance public solutions where private solutions are not viable. The emphasis on the MFD 16 The OECD’s 2019 Principles on Managing for Sustainable Development Results are 1. support the SDGs and desired change, 2. Adapt to context, 3. Enhance country ownership, mutual accountability, and transparency, 4. Maximize the use of results information for learning and decision-making, 5. Foster a culture of results and learning, 6. Develop a results system that is manageable and reliable. 17 Binnendijk, A. 2000. Results Based Management in Development Co-operation Agencies: A Review of Experience. OECD DAC Working Party on Aid Evaluation. 5 approach is critical to the achievement of country outcomes, involving both the public and private sector; it allows for a clear determination of the roles and its associated impact. 13. IFC is particularly well equipped to support the MFD approach through the IFC 3.0 Creating Markets Strategy. Since 2016, IFC has ramped up its efforts to help create markets at scale. To do this, IFC has launched the Country Private Sector Diagnostics (CPSDs) to identify the constraints and opportunities for private sector solutions. CPSDs are designed and sequenced to inform IFC Country Strategies, allowing IFC to strategically target interventions and policy reforms coordinated across the WBG to open key sectors to private investment. Next, IFC is also accelerating execution of IFC 3.0 by scaling Upstream efforts through creating and expanding new Upstream Departments to develop a long-term pipeline of projects across regions and industries, particularly in IDA and FCS markets, prioritizing mobilization from new sources. COVID-19 has underscored the need for continued investment in IFC 3.0 as private enterprises across emerging markets facing severe challenges and a clouded near-term outlook. 14. MIGA plays a critical role in maximizing finance for development by mobilizing cross-border investment into Emerging Market and Developing Economies (EMDEs) through its risk mitigation and guarantee solutions. MIGA’s FY21-23 Strategy and Business Outlook aims to scale up its political risk insurance guarantees in IDA and FCS countries, strengthen its support of climate finance, raise its gender ambition and increase the development impact of its guarantees to client countries through continuous innovation. This strategy will leverage deeper and more sustained WBG collaboration and partnerships with other MDBs. MIGA’s new product – The Trade Finance Guarantee (TFG) – aims to boost its partnership with the IFC by covering the IFC against the non-payment risk of banks in selected IDA/FCS and EMDEs in IFC’s Global Trade Finance Program (GTFP), thereby supporting economic resilience, and continued access to critical goods and services in the IDA/FCS countries most severely affected by the economic fallout resulting from the COVID-19 pandemic. 15. Among development agencies, the WBG has been at the forefront in building systems to advance the results agenda. The WBG has made significant progress in developing robust project-level M&E systems, one of the foundations for delivering and tracking progress towards outcomes and instilling a culture of managing for results across country teams. The Bank’s long- established practice of having all self-evaluations of its operations validated by IEG drives a focus on achieving results. Most recently in 2018 the Bank introduced the use of theories of change in operations to articulate logical pathways to change and to make it easier to track progress. The Bank’s leadership and architecture to support results orientation have been recognized within the international development community. For example, the OECD highlighted in 2007 the Bank’s “structured and well-established approach to results-based management”, which includes results practices at the project, country, and global levels.18 Assessments of the Multilateral Organization Performance Assessment Network (MOPAN) have also recognized that “[t]he Bank has tools and processes in place for measuring and managing results, signaling a strong corporate commitment to results-based management.” (MOPAN 2016).19 Moreover, in 2020, the Bank was elected chair of the Multilateral Development Bank’s Working Group on Managing for Development Results (MfDR), a useful platform to advance knowledge sharing about outcome orientation. 18 OECD. 2017. Results in Development Cooperation, Provider Case Studies: World Bank Group . 19 MOPAN. 2016. The World Bank Institutional Assessment Report. 6 16. IFC strengthened its results agenda in 2017 with the introduction of the Anticipated Impact Measurement and Monitoring (AIMM) system, which unified an existing results measurement system with an ex ante project rating framework. The AIMM system—which assesses both project outcomes (e.g., power generated, small and medium enterprises (SMEs) financed, students enrolled, etc.) and systemic changes in market development ("market creation" under the IFC 3.0 strategy)—has a strong outcome orientation and is an end-to-end impact management framework that is at the forefront of such systems adopted by the most advanced impact investors, including those that have signed on to the Operating Principles for Impact Management. Included in IFC’s impact management framework is also a robust evaluation approach that includes both project-level self-evaluations (many validated by IEG) and thematic impact evaluations intended to strengthen IFC's accountability and learning feedback loops. 17. MIGA launched its own ex ante development impact framework, IMPACT ("Impact Measurement and Project Assessment Comparison Tool") on a pilot basis in FY19 and with enhancements in FY21. MIGA aims to support developmentally impactful projects, as well as to create positive effects by addressing market and non-market frictions (e.g., barriers to entry and regulatory gaps) in host countries which may be holding back additional private sector cross- border investment. This is reflected in the structure of the IMPACT assessment, which separately assesses the expected impact across two dimensions: direct Project Outcomes and Foreign Investment Effects. The latter is similar to IFC’s Market Creation, as the framework focuses on outcome orientation to assess “beyond-the-project” outcomes in the host countries that receive foreign direct investment. The framework is evidence-based and relies on substantial country- and project due diligence. MIGA’s objectives in deploying IMPACT include to: (i) inform project prioritization; (ii) compare expected development impacts across the portfolio; (iii) standardize communications to relevant stakeholders; and (iv) serve as an input for MIGA's project assessment along with other criteria. The ex ante IMPACT assessment complements the ex post self- evaluation and IEG validation of every executed MIGA project. MIGA also solidifies learning and accountability with a regular and frequent corporate-wide learning engagement with IEG to discuss and disseminate findings and lessons from its ex post project evaluation program. 18. Building on this progress, improving outcome orientation is about embellishing a results architecture that has strong foundations. Like other development partners, the WBG’s results systems are primarily geared towards capturing outcomes that are more attributable to WBG support and emerge within the typical period of a project or a country engagement cycle. These features are fundamental for accountability purposes and for ensuring that WBG operations and strategies deliver their intended results. While this results system has served stakeholders relatively well, Management acknowledges that the rigorous focus on attribution has curtailed parallel attempts to articulate the WBG’s contributions to high-level outcomes over extended time horizons. As a result, some of the WBG’s recurrent contributions to global goals over time are not regularly captured, especially via institutional strengthening and market creation. Nonetheless, donors and shareholders alike still demand results information that can be traced to WBG interventions so any adjustments must maintain this function. 7 III. How the WBG aims for high-level outcomes 19. The WBG’s country engagement model drives its efforts to help poor and vulnerable people improve their well-being to realize their full potential. This model has evolved over time to be more transparent, evidence-informed, and results-oriented. Country Partnership Frameworks (CPF) are the WBG’s main strategic documents for articulating support to country outcomes. In consultation with key stakeholders, including private sector clients, and drawing on IFC Country Strategies, the WBG works with the government to draw from evidence—primarily through the Systematic Country Diagnostic (SCD) and the Country Private Sector Diagnostic (CPSD)— to determine the CPF objectives. The instrument for assessing WBG effectiveness in country programs, the CLR, focuses on capturing results that are achieved within the typical CPF cycle of 4-6 years and directly attributable to the WBG’s program. This architecture facilitates evaluability of country strategies and corresponding accountability to shareholders. 20. Four country pilots assessed how the WBG aimed for high-level outcomes at the country level. These pilots used contribution analysis20 to track the WBG’s contribution to a selected high-level outcome per country, linked to at least one SDG indicator. The four pilots were (i) Peru, child stunting (contributing to SDG Target 2.2), (ii) Kenya, access to electricity (contributing to SDG Target 7.1), (iii) Pakistan, primary school enrolment (contributing to SDG Target 4.1), and (iv) Indonesia, fiscal management (contributing to SDG Target 16.6)21 (Annex C). These pilots were purposefully selected for learning, and their findings cannot be generalized across the WBG experience. The pilots looked beyond the typical CPF timeframe, adopting a roughly twenty-year horizon, to track the Bank’s contribution towards those high-level outcomes over multiple CPF cycles. Once WBG support was understood and documented, the pilots compared the emerging narratives to those captured in successive CPFs and CLRs (or earlier equivalent products) when available and relevant. 21. Similarities emerged in these pilots to explain the WBG’s contributions to high-level country outcomes over multiple CPF cycles. Figure 2 illustrates how these similarities— scalable solutions, direct and indirect investments, risk calibration, a combination of lending and non-lending support, and evidence gathering—combine to paint a nuanced understanding of the way WBG programs deliver intermediate outcomes that help countries over time to move towards achieving higher-level outcomes. While the paragraphs below present a stylized model of the way the WBG aims for outcomes, conversations with stakeholders confirm that the model is grounded in reality. As expected, while intermediate outcomes are relatively well addressed in country engagement documents in all pilots, high-level outcomes are significantly less so. 20 Contribution analysis is a methodology used to identify the contribution a development intervention has made to a change or set of changes in a complex environment. It aims to produce a credible, evidence-based narrative of contribution that a reasonable person would be likely to agree with, rather than to produce conclusive proof. 21 The Indonesia pilot is still ongoing and close to completion. 8 Figure 2: Pathways to High-level outcomes Source: Review of WBG’s approach for achieving high-level outcomes. 22. First, the WBG typically tests development solutions before they are brought to scale with the contribution of other development partners. WBG teams maintained, continuity in dialogue, formulating a pathway to solutions, and aiming for high-level outcomes when engaging with clients. In all pilots, the Bank played a convening role to build a coalition to scale-up early results and successful outputs. It also leveraged funds from both domestic and international sources (government budget, development partners, and the private sector). In Kenya, for example, the Bank’s early support for the exploration and drilling of geothermal wells attracted several development partners (for example, African Development Bank (AfDB), European Investment Bank (EIB), Agence Francaise de Developpement (AFD), Japan International Cooperation Agency (JICA), and Kreditanstalt fur Wiederaufbau (KfW)) to make significant investments in geothermal energy. The Government was also able to grant its first set of licenses to private sector developers. Like in the other pilot countries, successive Kenya CPFs described interventions in the sector and identified quantifiable results, but the narrative of how the Bank’s support to electricity access was scaled-up by partnerships was captured only in a limited manner in CLRs. 23. Second, WBG interventions weave investments in infrastructure and service delivery (direct pathways) with support to reform policies and strengthen institutions (indirect pathways). Investment lending typically finances a given service for a population segment, and often relies on temporary project implementation modalities, injecting exogenous capacity, to deliver the intended service. Investments in policy reform and institution-building have a wider scope, as they aim to create an enabling environment by shifting specific government policies, systems, and capabilities to improve the effectiveness of government finance flowing through these systems. Investments in institutions take more time to mature as they put a premium on building the sustainability of reforms and leveraging much larger funds flowing annually through government systems. The Bank helped the Government of Punjab to put in place institutional monitoring systems to track implementation of various programs in the education sector. These critical contributions to strengthen the client’s ability to deliver effective services were not adequately highlighted in CPFs and CLRs as they were not easily quantifiable and would normally be considered implicit to the way in which the WBG aims for outcomes. 24. Third, the WBG continuously recalibrates interventions in response to risks to the desired high-level outcome, as inherent in the MFD approach. WBG teams periodically assess how risks could hinder the country’s journey towards high-level outcomes through these indirect 9 and direct pathways and seek risk mitigation measures. The risks associated with these two pathways are quite different.22 As shown in figure 2, investments in policy reform and institutional strengthening reduce the inherent capacity risks in each sector, so that the sector gradually moves from a higher capacity risk to lower capacity risk as institutions are strengthened to deliver effective services to achieve the relevant high-level outcome. This can lay the groundwork for private sector solutions with an improved policy and regulatory environment, Understanding and helping counterparts identify and consider these differing risks was crucial to support reforms across all pilot cases. While the Bank supported initial investments, exploratory studies and drilling of wells in the Kenya case, IFC and MIGA provided financing, guarantees and advisory services to the private sector to leverage additional financing, and de-risk investments to mobilize additional resources to meet the countries electrification needs. 25. Fourth, the Clients’ quest for high-level outcomes is supported by the WBG through a combination of lending, guarantees, and non-lending services. In Peru, for example, the Bank sustained a program of technical assistance to help the Ministry of Economics and Finance (MEF) in establishing results-based budgeting (RBB) for the social sectors. In subsequent years, successive Bank Public Expenditure Reviews (PERs) assessed the distributional impact at subnational levels to further refine RBB. An IEG evaluation showed that “the Bank’s technical assistance in this process helped the MEF adopt RBB and a focus on children’s chronic malnutrition more quickly.” While stakeholders had a clear understanding of the scope of the WBG contribution to reducing malnutrition, CLRs did not fully articulate this narrative, which was reconstructed retrospectively through the outcome pilot. Moving forward, there is an opportunity to anticipate high-level outcome pathways that combine lending and non-lending interventions. If done with appropriate flexibility and litheness, it can help country teams to enhance selectivity, course-correction, and evaluability. 26. Finally, the WBG builds learning opportunities that generate evidence to adjust programs towards high-level outcomes. The WBG teams supported countries with evidence- building exercises to answer specific program implementation questions. From the long-term perspective beyond any given CPF cycle, these exercises appear as a learning plan that organically evolved with WBG support. Among other things, household income expenditure reviews, poverty assessments, and impact evaluations helped test innovative interventions and inform program midcourse corrections. In Peru, for example, the Bank conducted a randomized control trial and a process evaluation to help improve beneficiary coverage and the quality of the health and nutrition services. These evaluations helped address shortcomings in the Juntos conditional cash transfer (CCT) program and were thus instrumental in addressing chronic malnutrition. And in Pakistan, the Bank’s technical dialogue and analytical work on private school subsidies in developing countries supported the reform of the Punjab Education Fund (PEF) which promoted public-private partnership (PPP) to rapidly scale up education provision. During the second phase, strengthening and expanding the PPP model was a key component all through the result-based financing operations starting in 2009. After 17 months of implementing this PPP model, a Bank funded impact evaluation found 59 percent enrollment gains as a result of PPP (Barrera & Raju, 2015). In general, this learning agenda was not anticipated in CPFs, in part because it emerged 22 Whereas direct investments, especially in infrastructure, often face significant direct social and environmental risks and sometimes fiduciary risks, institutional strengthening results typically face significant political economy and stakeholder risks, because they improve governance and linkages of resources with results, and therefore often have longer time horizons. 10 organically to address implementation challenges. The contribution of these products to high-level outcomes was also not systematically assessed as it is primarily about how rather than what results are achieved. 27. The four country pilots support IEG’s conclusion in its evaluation, Outcome Orientation at the Country Level, that the WBG is strong in aiming for high-level outcomes at the country level, yet opportunities exist to better articulate this contribution. In these pilots, the WBG continuously aimed for outcomes that matter, built knowledge and evidence around what worked, and incorporated lessons from the past into new support. These dynamics do not imply that opportunities for improvement did not exist. In fact, successive independent evaluations and self-assessments reflected on that. Nevertheless, the overall drive towards outcomes is clear. Yet the individual programs’ articulation of their linkages to the high-level outcomes analyzed in the pilots were mostly only implied in CPF documents themselves. The WBG is not alone in this regard. Country-level results reporting has been an area of frequent reform among Multilateral Development Banks (MDBs) and bilateral donors, as organizations struggle to balance competing imperatives. An OECD Development Assistance Committee synthesis of donor results-based management practices found that the country level is “the most challenging aspect of the results approach.”23 The MDBs are currently collectively focusing on this matter through the MfDR working group, which the Bank chairs. IV. Moving towards better articulation of high-level outcomes 28. A review of approaches in peer organizations, recent studies,24, 25 and the WBG’s own experience shows that articulating contribution to high-level outcomes is laborious. Development organizations tend to use some combination of the following four mechanisms to articulate support to and report on progress towards outcomes at the operational, country, and global levels: • Mechanism 1: Apex reports. These reports focus on high-level results and aim for simplicity and priority. They typically use indicators with a bird’s eye view on an organization’s goals and targets in different contexts to keep the high-level focus on what matters. These indicators are simple and used to identify the priority high-level outcomes to which an organization ultimately aspires. • Mechanism 2: Directional reports. These reports focus on breadth and direction. A synthesis of indicators and evidence can together provide a sense of how an organization is performing and what it is aiming towards. These indicators should provide a broad view of whether an organization is moving in the right direction towards the right outcomes. • Mechanism 3: Topical reports. These reports focus on depth and evidence. Deep dives into specific topics can uncover actionable lessons to improve development outcomes or 23 Zwart, R. 2017. 24 Bengo, I., Arena, M., Azzone, G., Calderini, M. 2016. “Indicators and Metrics for Social Business: A Review of Current Approaches”. Journal of Social Entrepreneurship, 1 (2), 1-24. 25 Zwart, R. 2017. 11 organizational performance in future support. These analyses provide depth and inform decisions with evidence. • Mechanism 4: Narrative reports. These reports focus on richness and context. Evidence- based nuanced narratives about contributions to meaningful outcomes can offer an understanding of how an organization best influences people’s well-being in a complex context. These analyses generate a rich understanding of contributions, putting support in a broader context of government programs and development partner support. 29. The WBG uses all four mechanisms, as shown in figure 3, both to articulate outcomes and to manage toward those outcomes. On the one hand, these tools are used to provide clients, the Board, and the public with detailed reporting on outcomes the WBG has helped to influence with varying degrees of attribution. On the other, Management and country teams use many of these tools as feedback loops to adjust implementation, to draw lessons or consider alternative approaches, or to calibrate WBG support over varying timeframes. Figure 3. Examples of how the WBG uses four mechanisms articulate and report outcomes Source: Review of WBG systems for articulating and reporting outcomes. 30. While opportunities for further improvements exist, these four mechanisms are hardwired into WBG systems at the operational and country levels. The WBG uses an apex approach when it establishes project objectives at the operational level to set the ultimate results an operation aspires to achieve, in line with the twin goals of reducing extreme poverty and promoting shared prosperity. Project development objectives (PDO) shape the focus of WBG operations and are the primary benchmark when considering alternative courses of action. Implementation Status and Results Reports (ISRs) for operations update the full range of indicators in results frameworks to assess the direction towards high-level outcomes, and flag emerging or materialized risks. Advisory and Analytical Services (ASA) are central to WBG support and generate knowledge to inform strategies to address topical issues that are impeding progress towards high-level outcomes. CLRs narrate the story of country programs’ effectiveness in meeting their objectives in the specific circumstances of each country yet, as seen from the pilots, 12 falling short of accounting for contributions to high-level outcomes. Each of these mechanisms is prominent in templates, garners attention at review meetings, and is made available to the public. 31. At the global level, the first two mechanisms are part of regular reporting, while the third and fourth are more ad-hoc. The SDGs and poverty and prosperity goals set the priorities. Scorecards such as the WBG’s Corporate Scorecards and the IDA’s Results Measurement System (RMS) use a range of indicators that collectively describe the direction of the organization. These are both reported regularly to the Board and the public. Country, regional, and global ASA, flagship reports, and IEG thematic evaluations generate evidence and lessons to shape the WBG’s development approaches. These are chosen to be relevant as new challenges emerge, priorities shift, and client demands evolve. 32. Development organizations have found the fourth mechanism—providing a contextual narrative of their contributions to high-level outcomes—the most difficult, particularly at the country level. Multiple actors and factors operating in dynamic environments influence change, making it difficult to disentangle the contribution of any one institution. MDBs and bilateral donors frequently revise frameworks, processes, and guidelines for country-level results, as development agencies struggle to strike the right balance between ambition and feasibility in setting country level outcomes.26 In consultations, most development agencies reported that they see value in capturing contributions to high-level outcomes over a long timeframe, but they do not have systems and processes in place that facilitate this type of retrospective exercise. Efforts are ongoing in a few MDBs to increase the focus on contributions to high-level outcomes, but it is too early to draw lessons from their experiences as many adjustments are not yet implemented.27 In short, no development organization has solved this problem, and the WBG, too, has ample opportunity for improvement. 33. Building on existing practices as well as the August 2020 Strengthening Outcome Orientation paper,28 the WBG has prepared an outcome roadmap, identifying 19 activities over FY21-23 to strengthen the Bank’s ability to articulate support to high-level outcomes (see Annex A). These activities reinforce the WBG’s current usage of all four mechanisms described above, presenting a holistic approach to outcome-orientation. This roadmap brings together several workstreams where outcome orientation is central, facilitating synergies and corporate tracking. The roadmap makes operational the four pathways for outcome orientation presented to CODE in FY20. These activities are not meant to be exhaustive but incremental. They build-up on progress already made across these areas. 34. The first seven activities help the WBG articulate an improved line-of-sight to high level outcomes. Among these activities are themes that represent corporate commitments, such as Gender, Climate, and Jobs and Economic Transformation (JET). These activities identify the 26 Zwart, R. 2017. 27 The African Development Bank’s (AfDB’s) approach to tracking country results combines an Alignment Matrix that demonstrates high-level contributions to national development objectives with a Performance Matrix that tracks project outcomes as well as policy dialogue, coordination activities, knowledge products, portfolio management and crosscutting issues such as gender, green growth, and fragility. The Asian Development Bank (ADB) is in the process of reforming its approach to country programs. Proposals under consideration include the inclusion of qualitative narratives describing how ADB engagement will contribute toward making progress on a few prioritized SDGs without quantitative indicators or targets and that will be periodically revised. In addition, it would track quantitative indicators, drawn from ADB operations implemented during the cycle. 28 World Bank, 2020b. 13 high-level outcomes that Bank support should aim towards (mechanism 1), presenting tangible quantitative and qualitative measures to track progress at the global, country, and operational levels (mechanism 2). Taken together they help reinforce the model through which the WBG aims for outcomes (see section V) by identifying opportunities to scale-up support, combine lending and non-lending, manage prominent risks, and close knowledge gaps. Importantly, these activities provide the basis to continue reinforcing the line of sight from WBG engagements to the 2030 Agenda and the twin goals. 35. Activities 8 to 10 help place outcome risk at the center of results programming. For example, the Bank is improving its conceptual approach to assess risk to development outcomes and identify opportunities for adjusting guidance to better manage risks and results in tandem. This would not only facilitate the inclusion of more precise metrics to track progress at the global, country, and operational levels (mechanism 2), but also facilitate program implementation, particularly in volatile country contexts like FCVs (mechanism 4). In bringing risks and results monitoring together, these activities have the potential to deliver both strategic and efficiency gains in country programs. 36. Activities 11 to 16 propose actions to enhance evidence-based learning. These activities comprise the testing of new methodologies to help the Bank and its client countries to articulate and track high-level outcomes at the country level through a longer time horizon (mechanism 4). They also include the modernization of data collection approaches for core operational priorities to facilitate capturing high-level outcomes through new techniques, use of technology and adaptation to country circumstances, deepening the understanding of thematic outcomes (mechanism 3). Complementing the new WBG Strategic Framework for Knowledge29, these activities will help operations and country engagements refine their understanding of outcome trajectories for high-level outcomes mapped to corporate priorities. 37. Finally, activities 17 to 19 strengthen the outcome orientation of country teams. They include activities that revamp and strengthen training to WBG staff on techniques to articulate and track high-level outcomes at thematic and country levels (mechanisms 2 and 4). They also repurpose the Results Measurement and Evidence Stream (RMES) to provide a strong platform for peer learning on outcome orientation and knowledge sharing across the WBG to facilitate outcome orientation at all levels (all four mechanisms). Collectively, these activities will ensure that staff have the capacity to apply an outcome orientation lens in their efforts to help partner countries meet their development goals. 38. Many activities have already made progress. Activities underway are advancing the WBG’s efforts to articulate an improved line of sight to higher level goals. For example, the IDA- RMS and Corporate Scorecard indicators have been revised to better align with SDGs and corporate priorities. In addition, IFC is now closely monitoring Upstream engagements by introducing a new metric in the Corporate Scorecard to measure the 5-year cumulative investment pipeline. The application of structured diagnostic tools (SCDs and CPSDs) is helping to fill important knowledge gaps and to identify the most impactful investment priorities in support of the JET agenda and IFC’s 3.0 Creating Markets Strategy. Risk and results approaches have been integrated in select training modules (e.g., modules of the Advanced Investment Project Financing (IPF) Academy and the Core IPF academy), making outcome risk more central to results 29 World Bank. 2020e. Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework. World Bank. Washington, DC. 14 programming. The remapping of technical staff focused on country services to the Regions is helping improve staff responsiveness to country problems and solutions. IFC’s creation of Upstream departments is enhancing IFC’s ability to create a medium-term investment pipeline to open and create new markets. The Bank is strengthening the outcome orientation of operations teams through the provision of outcome-focused skills development to staff, including task teams, task team leaders and experienced staff via self-paced and facilitated learning. Thus far in FY21, 955 staff have participated in workshops/academies, 6,752 staff have participated in Clinics, and eLearning modules have reached 3,345 participants. These activities are jointly contributing to an increased focus on high-level outcomes across multiple dimensions of the Bank’s work. 39. Ultimately, all these activities must add value to the country level, the foundation of WBG work. Grouped together they form an integrated set of initiatives to support existing systems, processes, and skills, building incremental changes towards greater outcome orientation. They reinforce both the what and how of the WBG’s definition of outcome-orientation, building on the steady progress made over the past two decades towards greater results orientation. V. Adjusting the country engagement model to account for high-level outcomes 40. A key part of the WBG’s efforts to improve outcome-orientation across the institution involves defining and implementing a refined country engagement approach. Using lessons from the outcome pilots, the IEG evaluation on Outcome Orientation and the WBG Strategic Framework for Knowledge30 (see box 1), the WBG is adjusting its country engagement guidance to articulate more explicitly and purposefully a line of sight to selected high-level outcomes via public and private solutions. The purpose of these adjustments is to ensure that the country engagement approach captures the WBG’s contribution to high-level outcomes and usefully informs decision-making on country engagements, in line with IEG’s recommendations.31 Following consultations with stakeholders, Management considers it more viable to adjust existing systems rather than creating new products and processes ex novo. Ensuring that the approach is implementable at scale with adequate rigor, Management will roll out the new guidance, with adjusted templates, training, and hands-on support in FY22. 30 World Bank. 2020e. Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions : A Strategic Framework. World Bank. Washington, DC. 31 Independent Evaluation Group, 2020a. 15 Box 1. Key lessons from Bank outcome pilots, IEG evaluation on outcome orientation, and the WBG Strategic Framework on Knowledge 1. Reporting on high-level outcomes requires a longer perspective than the one that had been used in CLRs. High- level outcomes take several cycles to materialize. 2. Identifying the pathways to high-level outcomes ex ante can facilitate design, implementation, and evaluability of WBG contributions. 3. High-level outcomes are at the core of policy dialogue but were less central in CPFs and CLRs given the existing country engagement guidance. 4. The WBG’s contributions go significantly beyond lending, including ASA, convening power, and knowledge sharing, yet CLRs use to focus disproportionally on lending. 5. Stakeholders described the Bank’s support for strengthening policies, systems, and capabilities as its most lasting contribution to the desired high-level outcome. 6. Opportunities exist for greater prioritization and selectivity of ASA, yet flexibility to adjust them is essential. Source: Operation Policy and Country Services (OPCS) adaptation taking information from Outcome Pilots, Independent Evaluation Group, 2020a and World Bank, 2020e. 41. With a more explicit connection to higher-level outcomes, the CPF results framework will be the cornerstone of the WBG’s results architecture at the country level. Key adjustments, several of which are detailed in subsection (b) below, can reinforce the role of the results framework as an integral part of outcome orientation. Focusing country programs exclusively on high-level outcomes could be misleading, as no individual agency can be realistically held accountable for achieving those outcomes, and information on progress toward those outcomes without context is neither useful for managing a program, nor adequate for learning and evaluation. Rather a CPF results framework with intermediate outcomes at the program level complements greater focus on contributions to high-level outcomes. Importantly, a results framework combined with a clear narrative can connect the dots, providing a more rigorous line of sight from the lending and non-lending portfolio to country results that ultimately lead to high- level outcomes. Furthermore, it can help provide a reality check on the anticipated pathways to high-level outcomes, tracking progress on strengthening institutions where relevant to CPF objectives. 42. The value of the CPF results framework comes not only from the information it generates, but from the processes it triggers to support the way the WBG aims for outcomes. Development of the results framework and its associated results chain is the primary touchpoint for many important conversations. First, it is used to strengthen the synergies between different instruments and sectors in the program and across the Bank, IFC, and MIGA with a shared purpose around meaningful results, and to ensure prioritization is driven by results. Second, it shapes conversations with government in a coherent narrative and helps clients coordinate initiatives across ministries and programs. And third, it is used to coordinate with other development partners and make connections across programs. It has also been the spark to develop partnerships for data collection in many countries. 43. In addition to the key items described above, WBG guidance and IEG research over the years has highlighted many other ways in which results frameworks for country programs can be useful and important.32 Table 1 below lists several. 32 See Roberts, Dawn; Khattri, Nidhi. 2012. Designing a Results Framework for Achieving Results: A How-to Guide. Washington, D.C.: World Bank Group. 16 Table 1: Value of results frameworks for country programs WBG teams have found the country program-level results framework valuable in the following ways: • Helps focus the program on specific outcomes • Develops a narrative for communication • Builds consensus and ownership around a strategic approach • Establishes an evidence-based approach to M&E • Measures progress toward a strategic objective that cuts across interventions • Facilitates harmonization of efforts with partners • Coordinates support across the public and private sectors • Provides framework for selectivity Source: World Bank staff adaptation from “Roberts, Dawn; Khattri, Nidhi. 2012. Designing a Results Framework for Achieving Results: A How-to Guide. Washington, DC: World Bank. 44. The WBG’s adjusted country engagement guidance will offer teams a more flexible approach to articulate and monitor their contribution to selected high-level outcomes, including in the results framework. The envisioned changes will provide a more structured approach for setting high-level outcomes anchored in the country’s development priorities, and articulating the WBG’s collective contributions to those outcomes, including through indirect pathways and through coordinated Bank-IFC-MIGA support via the MFD approach. These adjustments while relevant for all clients, are of particular significance in FCV countries (see section VI). The following dimensions summarize the forthcoming changes: (a) Focusing on a few high-level outcomes 45. The WBG will improve the line of sight to a few high-level outcomes over time in the CPF narrative. Using analysis from the SCD and CPSD, assessment of WBG comparative advantage, and country development priorities, and building on the selectivity framework, the CPF will articulate the results chains from the program towards selected CPF objectives and upwards to a limited set of high-level outcomes. This narrative will describe more clearly the role and complementarities of investments, policies, and institutions, and contributions from the public and private sectors. It will more deliberately articulate the centrality of institutional strengthening and market creation to the CPF objectives and WBG contributions to high-level outcomes. And it will convey how the Bank, IFC, and MIGA can help coordinate public and private action to yield transformative high-level outcomes through the MFD approach. The program will continue to situate WBG support for the selected priorities in the context of contributions from other development partners, the private sector, and civil society. The strategy will explain how the WBG plans to leverage partnerships and play a convening role in tackling the selected development challenge(s). It will also recognize that as targeted outcomes rise to a higher-level and depend on a larger scope and scale of changes, the degree of attribution to WBG support will diminish. 46. The time horizon of this high-level outcome narrative will extend beyond the CPF cycle, as long as it remains relevant as a high-level development goal that the WBG seeks to contribute to, based on country priorities and demands for WBG engagement, SCD findings, and other strategic considerations. This extended planning period will acknowledge that progress to high-level outcomes becomes evident over time, often with a long run up of improving policies, developing systems, and building capabilities. At the same time, the CPF objectives will continue to be assessed within the current cycle, with an expectation that many CPF objectives would be carried over from cycle to cycle. Changes to CPF objectives would be explained in the 17 context of the targeted high-level outcomes—for example due to new knowledge about a strategic approach to achieving a high-level outcome, a shift in responsibilities among partners, or progression to the next phase along the pathway to the high-level outcome. This will better align the country program instruments with the WBG’s business model of helping client countries achieve high level-outcomes over time, while still maintaining a strong degree of accountability within the CPF cycle. High-level outcomes, while usually articulated across multiple CPF cycles, can also be adjusted through a forward-looking CLR process and a clear justification for change. (b) Connecting the results framework to high-level outcomes 47. The CPF results framework will be adjusted to match the new CPF line-of-sight narrative, with three primary changes. First, the results framework will reach all the way up from the planned program to high-level outcomes, while acknowledging that these outcomes will require contributions from many partners. Second, Management is reflecting on options to better use the results framework to track progress in strengthening institutions—policies, systems, and capabilities. Third, Management is considering approaches to identify indicators that are better matched to CPF objectives without increasing the data collection burden on clients. This may take the form of more purposefully encouraging teams to draw on ‘sentinel’ indicators, those proxy indicators that serve as bellwethers and have outsized importance in explaining a program pathway towards high-level outcomes, based on available evidence. (c) Assessing progress on the pathway towards high-level outcomes 48. At the end of the CPF cycle, the CLR will contextualize and put in perspective the results achieved under the CPF objectives—including results attributable to projects and interventions pre-dating the current CPF cycle—and progress observed towards the selected high-level outcomes. This assessment will provide teams the opportunity to look at the bigger picture and reflect on the effectiveness of the program in context. The CLR will endeavor to explain why or why not things are moving in the right direction and formulate lessons for the next CPF, including on whether and how the selected high-level outcomes or their pathways should be adjusted. Attention will be paid to efforts that address institutional risks, such as support to improving policies, systems, and capabilities. Findings will rely on a mix of quantitative and qualitative evidence and may include feedback from partners and stakeholders. The CLR will also assess the extent to which WBG support was outcome oriented. (d) Learning for high-level outcomes 49. In line with the WBG Strategic Framework for Knowledge, the WBG will identify key knowledge gaps, including to achieve the selected high-level outcomes, in a more structured way to the extent possible. This will include helping the government to diagnose development challenges, capture evidence about what works and what does not work, conduct research, and use evidence to plan and implement reforms. A more structured approach to identifying and filling knowledge gaps will help ensure that the Bank is generating the evidence needed to inform strategic decision-making and to shape WBG investments and government policies to be more effective. The CLR may reflect on progress in filling key knowledge gaps during CPF implementation. (e) Adapting for high-level outcomes 50. While the WBG has processes to facilitate and capture changes at both the project and country levels, the outcome pilots showed that adaptive decision making often happens 18 organically, on an ongoing basis and is informed by a range of evidence sources. In some cases, learning from impact evaluations or results monitoring data point to the need for adaptations. In others, beneficiary feedback, discussions with clients, country team reflections on what is working and what is not, and information shared by key stakeholders are critical sources of evidence and are more commonly relied upon for strategic adaptations. Ideally multiple data points are triangulated to inform adaptive decisions. 51. In the way the WBG aims for outcomes, adaptation and course-corrections can be strategic or tactical. Strategic adaptations involve higher level changes, for example adding, removing, or adjusting a development objective in response to new knowledge or emerging constraints, priorities, or opportunities. In the WBG context, these strategic adaptions are made at the country program level and reflected in the PLR and CLR. For example, in response to increasing urbanization and a shift in the concentration of poverty from rural to urban areas, the Mongolia 2019 PLR broadened one of the country strategy objectives to reflect the WBG’s engagement to promote economic opportunities in urban areas (in addition to rural areas).33 Tactical adaptations focus on course corrections to navigate implementation challenges and ensure the achievement of existing objectives. In the Peru pilot, an impact evaluation found that including child growth check-ups as a condition for cash transfers works well as an incentive and leads to better nutrition in targeted populations. These decisions are typically made at the project level and reflected in the ISRs and ICRs. IFC Country Strategies, which feed into CPFs and shape IFC’s priorities for investment, advisory and upstream operations at the country level, are reviewed internally on a 6-month informal basis, and every 18 months on a formal basis with the opportunity to change strategic direction. 52. As one piece in a larger puzzle, the revised country engagement guidance will support ongoing efforts to better capture and bridge these different types of adaptations. The intervention logic for selected priorities will connect the CPF objectives to high-level outcomes. The recent introduction of theories of change at the project level is already helping to make the connections between PDOs and high-level outcomes more evident. The WBG Strategic Framework for Knowledge also envisions enhancements to the ICR and CLR to systematically capture lessons learned. Under the new guidance, periodic and intermediate reviews during CPF implementation will note shifts in priorities, circumstances, or evidence; discuss changes in contributions of partners towards high-level outcomes; and use rapid and light-touch monitoring on progress toward high-level outcomes to evaluate options for shifting the program’s course. Some of these reviews may continue to be informal, allowing a candid discussion on progress and encouraging new ideas to reconsider expected pathways to the selected high-level outcomes and adjust the program as needed. 53. Table 2 illustrates the key differences in the enhanced approach to country engagement, taking the support to the Kenya energy sector pilot case as an example. Under the new approach, the higher-level outcome that the WBG is aiming for will be specified in the CPF. The narrative will describe how the WBG expects to contribute to that high-level outcome over time through public and private solutions, with special attention to key milestones along the pathway to stronger institutions. 33 World Bank Group. 2019c. Performance and Learning Review of the Country Partnership Strategy for Mongolia for the Period FY13-FY18. World Bank. Washington, DC. November 13, 2019. 19 Table 2. An illustration of the new approach in the Kenya energy sector Current approach New approach Narrative Assertion that policy advice and Description of program situated in the context of a long- technical assistance will help term engagement, noting achievements to date and the improve functioning of electricity expected pathway to high-level outcome. Specific market, and investments will milestones of institutional strength—for example ability of modernize selected transmission regulator to set cost-reflective tariffs, use of market and distribution networks. analysis to inform transmission investment decisions, etc.—described in the CPF narrative. High-level Not specified Explicit—universal access to reliable electricity outcome Learning and PLR reflects adjustments already Regular opportunities to reflect with client on progress adapting made. toward achieving high-level outcome and to consider adjusting the approach and the WBG engagement in that context, including filling knowledge gaps. Assessing CLR assesses effectiveness in CLR assesses effectiveness in achieving CPF objectives effectiveness achieving CPF objectives during during CPF period, reflects on contributions to progress on CPF period, focused on direct selected high-level outcomes, and draws lessons that linkage to specific investments. inform the next phase of the engagement. Source: Operation Policy and Country Services (OPCS) adapted from World Bank. 2014. Kenya - Country Partnership Strategy for the Period FY14 – FY18. World Bank. Washington, DC. 54. This new outcome-oriented lens on country engagement cuts across the full range of country engagement products—from SCDs to CPFs to PLRs and CLRs. Throughout, teams and clients will maintain a clear focus on high-level outcomes that matter, using that focus when developing and reconsidering strategic approaches, correcting course during implementation, and reflecting on contributions. VI. Applying the new country engagement guidance to FCV 55. Fragile situations demand more adaptability to dynamic circumstances and realism in objective-setting and will benefit specially from the changes to the country engagement guidance. While circumstances on the ground change suddenly in FCVs, actions in response to the crisis of the moment need to consider the long run implications of policies to avoid unsustainable development paths. This requires anchoring the country engagement on selected long-term pathways to high-level outcomes, while facilitating the adjustment of the lending program to respond to contextual priorities, such as natural disasters and conflict. They will be coupled with adjustments to the evaluative framework (in line with measure 5 of the WBG’s Strategy for Fragility, Conflict, and Violence 2020–2025,34) to ensure that the right incentives are provided. The Liberia example illustrates the value added in the CPF preparation when additional flexibility is granted in the FCV context (box 2). 34 World Bank, 2020d. 20 Box 2. FCVs need additional flexibility The Liberia CPF (2019-2024) is grounded in an analysis of drivers of conflict and exclusion and targets the problems of weak governance, inadequate economic and social inclusion, breakdown of social cohesion, youth unemployment, gender inequality, and regional disparities. The strategy integrates lessons relevant to Liberia’s fragile context from the CLR, including the need to apply a sound post-conflict and fragile-country lens and the importance of remaining focused on medium-term goals even in the face of a crisis. • The CPF identifies the foundational importance of strengthening public administration and envisions substantial knowledge and capacity development support. • Improving the transparency, accountability, and efficiency of public institutions features as a primary objective and is integrated throughout the strategy. • Given the fluid context, the CPF includes biannual portfolio reviews to regularly monitor risks and adapt to changing circumstances and priorities. • The Results Framework includes outcome-oriented indicators that capture improvements to people’s well-being, such as impacts on jobs and income. Source: World Bank. 2020c. Liberia - Country Partnership Framework for the Period FY19 - FY24. World Bank. Washington, DC. 56. Sustainable development in FCV settings necessitates addressing the drivers of fragility and strengthening sources of resilience. This requires a sustained engagement over a long timeframe, as progress can be slow and does not always follow a linear path. Adjustments to the country engagement guidance will help maintain such focus by specifying and tracking selected high-level outcomes to eliminate the drivers of fragility over multiple CPF cycles. The narrative intervention logic of the CPF will give due consideration to the logical linkages among elements of the program across focus areas and sectors that contribute to tackling these drivers over time. 57. The WBG’s comparative advantage in FCVs is grounded in its long-term support to strengthening institutions. Institutional capacity deficits in FCV contexts increase the risk of instability and reduce resilience to disruptive shocks. But building strong, capable institutions is generally a long-term process that extends beyond a single CPF cycle. Adjustments to the CPF guidance will make the strategic approach to building and strengthening institutions and the investment climate over the long term more central to the WBG’s country program. It will also track more carefully key milestones along that path, no matter whether an output or outcome, quantifiable or not. 58. Because of increased risk and uncertainty in FCVs, enhanced adaptability in design, implementation, and evaluation is paramount to aim for and report on high-level outcomes. The political, stakeholder, social and related institutional capacity risks underlying fragility and conflict require frequent adjustments and course corrections along the pathway to achieving high- level outcomes. In these contexts, effective implementation requires managing risk and results in tandem. Several activities in the outcome roadmap aim at making the articulation of risks and results trade-offs more prominent. 59. Weak data and statistical systems in FCVs require customized monitoring with rapid feedback loops. Many of the short-term results of activities that build resilience are difficult to measure through traditional results frameworks. Adjustments to the CPF guidance, coupled with activities identified in the Outcome Roadmap to improve data collection, will help build-in more 21 frequent and light-touch monitoring activities (using a mix of quantitative and qualitative methods) to better inform adaptations in response to rapidly changing contexts. Given the prevalence of a multitude of stakeholders in FCV settings, and their combined contribution towards high-level outcomes, partnerships around collecting data and measuring results throughout the process would be critical. VII. Reinforcing incentives for high-level outcomes 60. Better articulating WBG contributions to high-level outcomes can reinforce positive behaviors identified in the outcome pilots, facilitating a more homogeneous approach to results across countries. The task is no longer one of creating a culture of results in operational teams, as it was two decades ago. Steady positive trends in outcome ratings for operations and CPFs showed that teams aimed for clear outcomes.35 The outcome pilots further indicated that this resolve is sustained across extended periods of time to achieve outcomes that matter. And they do so even if the existing country engagement guidance may incentivize them in a different direction.36 As highlighted in recent IEG evaluations, while teams practice this model well, the associated results system “does not capture the WBG’s contribution to country outcomes well.”37 So, this is now an agenda of ensuring that existing country engagement guidance keeps up with the outcome orientation culture that predominates in country teams. 61. To make the WBG results systems relevant to the way the Group aims for high-level outcomes, the proposed changes seek to reinforce three key behaviors in country teams. As shown in figure 4, the first behavior is the need for teams to combine short- and long-term perspectives more explicitly during CPF design. This would be instrumental for ensuring that programmatic decisions are driven by a focus on achieving high-level outcomes, over multiple cycles. The second behavior is the use of the information generated by the results system and analytical work as part of the battery of evidence that informs mid-course corrections. For this to happen, that information will be made more relevant and country management will provide more frequent spaces for reflection. Finally, CPF completion should be a unique opportunity to reflect on progress towards high-level outcomes to inform future engagements. As demonstrated in the outcome pilots and confirmed in IEG evaluation on Outcome Orientation,38 country teams have outcomes in mind across the country engagement cycle but those outcomes are not explicitly articulated, tracked, and reported in country strategy products, the aim of the proposed adjustments. 35 Independent Evaluation Group, 2020b. 36 World Bank. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. 37 Independent Evaluation Group, 2020a. 38 Independent Evaluation Group, 2020a. 22 Figure 4: Desired behaviors to reinforce country teams’ outcome-orientation Source: World Bank Staff review of behaviors that contribute to outcome orientation. 62. Avoiding shortcomings of previous attempts to articulate high-level outcomes requires thinking about the incentive system that both accompany and is embedded in the proposed adjustments. Incentives are broadly defined as “the things that motivate or encourage one to do something.” One common way that development organizations look at incentives is by the timing of the desired effect along the project/country engagement product cycle. Ex ante incentives for outcome orientation aim to ensure a strong project/CPF design, one that is based on a nuanced understanding of the problem to be addressed, is underpinned by a logical, evidence- based results chain, and includes a clear articulation of the outcomes to be achieved. Ex post incentives, on the other hand, are created by processes that evaluate and rate performance and results after project completion. The timeframe for WBG projects/CPFs and frequent staff rotations may limit the ability of ex post incentives to have a significant impact on staff behaviors.39 Designing effective incentives to increase the WBG’s outcome orientation is not straightforward because incentives can inadvertently encourage a compliance mind-set or undermine positive norms, exactly the opposite of what is needed to encourage an outcome orientation. Also, incentives are not static and homogenous; they vary across the organization, and individuals respond differently to the complex and dynamic web of interacting incentives that they confront. 63. The proposed changes to the country engagement guidance, as well as the broader outcome roadmap, combine ex ante and ex post incentives towards high-level outcomes. The revised guidelines aim to better align incentives for outcome orientation throughout the country engagement process, from CPF design to CLR review. The need to articulate a line of sight to high-level outcomes at CPF design and the multiple opportunities for discussion, debate, and contestability built into the WBG’s review processes strengthen ex ante incentives for outcome orientation. The proposed changes also reinforce ex post incentives by taking stock during the PLR and/or the CLR of the program’s outcome orientation and the country’s progress along the pathway to the targeted high-level outcome. 64. Incentives can also be extrinsic or intrinsic, with the latter being predominant in WBG’s culture. Extrinsic incentives use external rewards and punishment (carrots and sticks) to 39 Marie Moland Gaarder & Ulrich Bartsch, 2014. "Who cares about development outcomes? Market failures and the role of the evaluation function," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(4), pages 361-377, December. 23 motivate desired behaviors, for example tying salary increases to performance ratings. Recognition and visibility are also common extrinsic incentives. Intrinsic motivation refers to the enjoyment, meaning, or sense of accomplishment one derives from an activity. As a mission-driven organization that plays a preeminent role in international development, the WBG attracts staff with strong intrinsic motivation to help client countries achieve development outcomes. Most staff are drawn to the WBG due to their commitment to its twin goals and a desire to contribute to meaningful change. A survey conducted on incentives for knowledge sharing in the Bank found that staff feel a commitment to see their projects succeed, even if they are moving on to another role.40 The survey also confirmed the critical role of intrinsic motivation within the Bank’s incentive landscape, with more respondents identifying intrinsic rewards (82%) and a sense of ownership toward a project (82%) as incentivizing staff to engage in desired knowledge sharing behaviors than extrinsic rewards (62%).41 Staff with intrinsic motivation towards results are guided by the intention of doing the right things and doing them right, usually responding to the sense of direction imparted by Senior Leadership. As it continues advancing implementation of the Outcome Roadmap, Management is paying increasing attention to providing the right signals towards greater outcome orientation. The Bank, for example, is ensuring that review meetings interrogate the strength of evidence underpinning the program’s results chain and discuss expected contributions to high-level outcomes. 65. Better articulating WBG support to high-level outcomes, accompanied by reinforced incentives, injects new meaning into matters that would be problematic in the absence of such articulation. Table 3 below illustrates how adopting an “outcome mindset” reorients and gives greater purpose to key features of the WBG’s business model that are usually assessed assuming an “output mindset”. For example, with an outcome mindset, CPF results frameworks gain increased relevance, as they provide an empirical basis to substantiate the WBG’s contributions to high level outcomes. Mid-course corrections are not understood as discrete events, but rather as a continuous process of strategic and tactical adaptations driven by a line of sight to high-level outcomes which are formally recorded at discrete touchpoints for accountability. And the push for approvals, commitments, and deliverables is seen as a vehicle, rather than an obstacle, to achieving results, given leadership signals on the importance of aiming for high-level outcomes. In such a context, building a solid package of training, surge support, guidelines and approaches to allow staff to aim for high-level outcomes from early stages of project and program conception onwards, as presented in the Outcome Roadmap, reinforces teams’ intrinsic motivation towards results. 40 World Bank. 2019b. World Bank Knowledge Sharing and Reuse Incentive Framework . Washington DC: World Bank. 41 World Bank, 2019b. 24 Table 3: Shifting from an output mindset to an outcome mindset Assuming an Output Mindset Adopting an Outcome Mindset Results frameworks impede focus on long-term Results frameworks provide realism to claims of outcomes as they collect short-term project data contribution to high level outcomes Approvals, commitments, and deliverables overshadow Approvals, commitments, and deliverables underpin the drive for results the drive for results Mid-course corrections are discrete events that do not Mid-course corrections happen continuously at happen with sufficient regularity operational and strategic levels Extrinsic and ex post incentives are more effective to Intrinsic and ex ante incentives are more effective to reinforce the drive towards outcomes reinforce the drive towards outcomes Source: World Bank staff. VIII. Conclusion and next steps 66. In summary, the paper highlights the following three key messages: 67. First, the model of how the WBG aims for high-level outcomes is generally sound. In the IEG evaluation on Outcome Orientation,42 IEG recognized that “the model of how the Bank Group aims for outcomes in its client countries is sound, and, for the most part, teams practice this well”. This was confirmed by the outcome pilots. 68. Second, opportunities exist to adjust the country engagement guidance to better articulate contributions to high-level outcomes and its associated behaviors. The outcome pilots showed that the most important WBG contributions to these outcomes emerge over multiple CPF cycles, are driven by strengthened institutions, and were not well-articulated in SCDs, CPFs and CLRs. The proposed adjustments, which are relevant for all country contexts, including FCV countries where more flexibility is needed, include: • The SCD will articulate pathways to key high-level outcomes on the way to the twin goals, and the CPF will select, applying standard selectivity filters, a few high-level outcomes that the WBG will support. These high-level outcomes will usually carry across CPF periods providing for continuity of focus and a line of sight not bound by relatively short periods. • Periodic and intermediate reviews during CPF implementation would note any shift in priorities, circumstances, or evidence, and discuss any mid-course corrections in implementation, to maintain a focus on achieving identified high-level outcomes. • At the end of the CPF cycle, the CLR will contextualize and put in perspective the results achieved on the CPF objectives and the progress observed on the selected high-level outcomes. It will be brought forward to better inform the preparation of new CPFs. 69. Third, the WBG’s roadmap to improve outcome orientation is under implementation and progressing. Current activities combine actions from multiple workstreams, including the follow-up to the four outcome pilots in Indonesia, Kenya, Pakistan and Peru; the IEG evaluation 42 Independent Evaluation Group, 2020a. 25 on Outcome Orientation;43 and the measures in the WBG’s Strategy for FCV44 that pertain to outcome orientation in FCV contexts. Across all levels, efforts are underway to move indicators from outputs to outcomes, balance risks and results, and build the skills and tools to measure high- level outcomes. While the WBG is learning from other development partners’ current practices, this is frontier work across the spectrum, and the WBG is playing a leading role. 70. Next steps. Management plans to conduct further discussions to facilitate adjustments to the country engagement model that are practical and feasible. Adapting guidance and templates to articulate the WBG contribution more explicitly to high-level outcomes will enhance the model of how the WBG aims for outcomes. This will complement other methods of outcome tracking with more nuanced country level results. Given the novelty of the approach, and to ensure that the adjustments are relevant for all client segments, Management will initiate the changes in CPFs starting in FY22. The initial roll-out will be accompanied by dedicated training, clinics, and Management communications to reinforce the importance of the changes. After this period, Management will distill lessons and propose further adjustments as needed based on experience. 43 Independent Evaluation Group, 2020a. 44 World Bank, 2020d. 26 Annex A: Outcome Orientation Roadmap CODE Paper – August 2020 Progress to date – March 2021 Next Milestone Timeline Pathway 1. Articulating an improved line-of-sight to higher-level goals, including WBG twin goals and SDGs 1. Implement WBG Climate Change Bank climate co-benefits target raised from 28% to Incorporate climate outcome indicator in From Jan. 2020 for IDA Action Plan 2016-2020, and initiative 35% in December 2020; retrospective completed operations with 20%+ climate co-benefits. and July 2021 for IBRD retrospective review; continue to in February 2021; Climate Change Action Plan Launch Country Climate and Development From July 2021 meet climate commitments embedded 2021-2025 initiated. Report. in IDA19 Policy Commitments and Improve metrics to capture resilience to By June 2022 IBRD capital increase package. climate shocks, impact on Green House Emissions (GHE), and mitigation and adaptation outcomes. 2. Continue to implement WBG Initiated revision of guidance and staff training to Finalize guidance and staff training. By June 2022 Gender Strategy and gender strengthen Bank's ability to track sector outcomes Develop approach to realize cross- By June 2022 commitments embedded in IDA19 in closing gender gaps. operational synergies to address country- Policy Commitments and IBRD Increased share of IDA operations linking gender level gender gaps. capital increase package; initiate gaps to actions tracked in the results framework to Gender Strategy Mid-Term Review 62% percent in FY20. 3. Deepen focus on WBG Jobs and Increased use of Systematic Country Diagnostics Continue to increase application of Ongoing Economic Transformation (JET) in (SCDs) and Country Private Sector Diagnostics structured diagnostic tools. operations and country strategies, in (CPSDs) to help fill knowledge gaps towards high- Improve techniques and systematize efforts By June 2023 line with IDA19 Policy level outcomes and identify the impactful to measure JET-related outcomes (including Commitments. investment priorities to support JET agenda. job impacts). 4. Implement the WBG Strategic Approval of the WBG Strategic Framework for Develop and implement institution-specific FY22 Framework for Knowledge to Knowledge. actions necessary to realize the vision of the enhance the relevance, quality, and Strategic Framework. impact of WBG knowledge products. Integrate results 'narratives' on the impact of FY22 WBG ASA knowledge products in IDA RMS and Corporate Scorecards. 5. Expand use of Bank IT systems and Developed IT system to track and archive annual Create a repository of Corporate Results By June 2022 platforms to track outcomes at all IDA RMS and Scorecard data more Indicators (CRIs) to establish linkages with levels. systematically. corporate priorities and facilitate analysis over time. 6. Align WBG development objectives Revised IDA19 RMS and 2020 Corporate Align IDA20 RMS and 2021 Corporate December 2021 and corporate strategies--via Forward Scorecard indicators to better align with SDGs and Scorecards to IDA20 Policy Commitments 27 CODE Paper – August 2020 Progress to date – March 2021 Next Milestone Timeline Look, IDA and IBRD commitments, corporate priorities around gender, climate, as well as the WBG response to the COVID and others--with the 2030 Agenda. fragility, jobs, and governance. crisis. 7. Lead an MDB dialogue to explore Conducted benchmarking of outcome orientation Exercise convening role for the outcome From July 2021 opportunities for synergies and cross- practices in MDBs and selected bilateral partners. orientation agenda across the MDB institutional learning on outcome community by exploring opportunities for orientation. stocktaking and learning. Pathway 2: Placing outcome risk at the center of results programming 8. Articulate conceptual approach to Updated Board via upcoming QBRR risk chapters. Test approaches to consider results and risk From July 2021 assess development outcome risk in jointly in project design and implementation. Bank programs and adjust guidance. 9. Calibrate Bank’s Systematic Two rounds of operational Implementation Status Continue the iterative process with Ongoing Operational Risk Rating Tool Results Report (ISRs) with SORT risk ratings calibration expected to continue through (SORT) based on new guidance. recorded in line with guidance. several ISR cycles. 10. Build more integrated focus on risk Consideration of risk and results being integrated Finalize updates to the Core IPF Academy By June 2022 and results in training and outreach into select training modules (e.g., modules of the and to the Advanced IPF Academy. Include work on operations at the Bank. Advanced IPF Academy, Core IPF academy), and greater results focus in tailored outreach on outreach on risk calibration. risk. Pathway 3: Enhancing evidence-based learning in operations 11. Revise country engagement Completed three outcome pilots (Kenya, Pakistan, Adjust country engagement guidance to By end FY21 guidance and templates to facilitate Peru) and initiated a fourth (Indonesia); explored track high-level outcomes, including specific the articulation, monitoring and options in consultation with CMUs and TTLs. approaches in FCV countries. reporting of WBG contributions to Roll out new training and direct support to From July 2021 high-level outcomes. country teams. 12. Explore possible WBG Developing practical methodologies to measure Finalize the methodology for measuring By June 2022 methodologies (including through how knowledge products, capacity building and contribution of knowledge products to outcome pilots) to track indirect institutional reform support development outcomes. pathways to country-level outcomes outcomes. Test the methodology for integration into By June 2023 to be incorporated in self-evaluation standard M&E frameworks and reporting. and reporting tools. 13. Set the foundations to help WBG Management Action Record (MAR) process Issue second WBG self-evaluation under By September 2021 shift for higher outcome orientation reformed, with self-evaluation informed by an new model, with more explicit identification in implementing recommendations outcomes-based narrative covering all outstanding of outcomes to which IEG evaluations from IEG evaluations. IEG evaluations together. contributed. 14. Test and implement in several Five project/ portfolio monitoring tools integrated Scale-up toolkit in support of project Ongoing countries data collection tools to into a toolkit, including Iterative Beneficiary implementation across regions, including for strengthen country M&E systems, Monitoring, Portfolio Footprint, Project Targeting COVID-19 applications. better target interventions, and Index, Survey of Well-being via Instant and 28 CODE Paper – August 2020 Progress to date – March 2021 Next Milestone Timeline establish impact on poverty by the Frequent Tracking, and Rapid Response Phone Bank. Surveys. 15. Complete WBG FCV strategy and Ongoing consultations between OPCS, IEG, and Define revisions in evaluative framework at By June 2021 and Q1 FY22 develop and implement Bank FCV FCV on updating the evaluative framework. operational and country level in FCV respectively M&E Practice Note to support countries. operational teams. Developing a practice note on M&E in FCV Finalize Practice Note. By June 2021 countries, with a focus on high-level outcomes. Rollout Practice Note and deliver training. From July 2021 16. Enhance WBG data management to Modernizing data collection approaches for Explore new ways to use artificial By June 2022 improve outcomes, decision-making, operational priorities to help capture outcomes intelligence and 'big data' in data collection selection, and prioritization. through new techniques, use of technology, and and real-time reporting. adaptation to country circumstances. Pathway 4: Strengthening the outcome orientation of operations teams 17. Continue to provide outcome-focused In FY21 to date, 955 staff participated in Targets for staff learning by the end of the By June 2021 skills development at the Bank workshops/academies, 6,752 staff in clinics, and FY are 1,500 participants of Workshops/ through self-paced and facilitated 3,345 in eLearning modules; eLearning was Academies; 10,000 participants of Clinics; learning to key audiences of staff, launched for TTLs in Q2; over 1,600 hours of and 6,000 participants of eLearning. including task teams, TTLs and operations learning in academies and clinics were eLearning for Specialists is under By June 2022 experienced staff; increase and delivered by WBG subject-matter experts, development. enhance PforR Academy learning including 61% from Regions, 30% from OPCS, offering: launch new PforR and 9% from Global Practices. eLearning for TTL and Specialist tracks. MIGA has dedicated learning seminars with IEG, to disseminate findings and lessons learned from the ex post evaluation. 18. Launch the Bank’s M&E Gateway, a M&E Gateway launched in Q1 FY21 with inputs Continue to populate and promote M&E Ongoing one-stop-shop for staff to find easily from various GPs. Central repository of WB Gateway. corporate resources on M&E and impact evaluations being created, so that Archive in a dedicated system all Bank- By June 2022 results management. operational teams can easily utilize evidence supported impact evaluation in accessible generated from impact evaluations to aim for way to facilitate learning from evidence. outcomes to aim for outcomes. 19. Refocus WBG Results Discussing among the WBG opportunities for Launch an "Outcome Academy" to share By June 2022 Measurement and Evidence repositioning and broadening the membership of good practices on outcome orientation from Stream to advance knowledge and RMES to serve as a platform to identify inside and outside the WBG with operational reflection on outcome orientation and knowledge and opportunities to advance outcome and country teams. position WBG as a thought-leader on orientation in programs. outcome orientation. 29 Annex B: Outcome Orientation During COVID19 Response The WBG is working both to respond to the crisis situation and to maintain a line of sight on the long-term, aiming purposefully for selected high-level outcomes. A few examples illustrate the WBG’s efforts during the pandemic. The Bank continued to support better informed policies around gender that would help strengthen and respond to the gender equality gains lost in Latin America and the Caribbean (LAC) during the pandemic. The LAC COVID-19 High Frequency Monitoring project conducted high-frequency phone surveys in three waves from May to August 2020, involving 13 countries in LAC to collect information and track the impacts of the crisis on households’ wellbeing. This real-time monitoring looked at the impacts of the COVID-19 crisis on labor, behavior, health, coping mechanisms, and public interventions. The study raised awareness concerning the unequal effects of the pandemic on women, and it helped produce better-informed policy making by prompting discussion on the unequal adverse effects of the pandemic on women. A balance of crisis management and keeping the line of sight on outcomes by investing in strengthening health systems has remained the Bank’s priority during the pandemic in Yemen. The Yemen COVID-19 Response Project (YCRP) provides the country with critical resources, which helped Yemen respond to the pandemic. The project supports early detection and screening for the virus, provides essential medicines and medical equipment to treatment centers, and builds the capacity of human resources to respond to the pandemic in Yemen. The YCRP will play a significant role in deploying vaccines by supporting deployment costs to administer the vaccine to the population. At the same time, the project will continue supporting strengthening health systems, improving national laboratories, strengthening surveillance capacity, conducting case management of COVID-19 cases, and ensuring environmental and social safeguards. Learning from past experiences with the Ebola crisis, Sierra Leone with the help of the Bank is building a more resilient, equitable, and personalized education system in response to the COVID-19 pandemic. The Bank’s Free Education (FREE) Project, an IDA grant, supported the initial implementation of the Government’s COVID-19 education response: distance learning for students, of which half are girls; sensitizing campaigns regarding COVID-19 prevention, gender- responsiveness, and back-to-school arrangements; school safety protocols and psychosocial support guidelines to primary and secondary schools, as well as training teachers; hygiene and safety products; annual School Census 2020; and a legal and policy framework to support the inclusion of all children in Sierra Leone. This has been supplemented by a series of the Bank’s analytical and advisory support. A Bank supported community-driven development approach in Afghanistan became a key mechanism to quickly monitor the situation of the vulnerable and help them in a timely manner during the pandemic. The ongoing Charter Afghanistan Project (CCAP) and a recent parallel emergency project – Relief Effort for Afghan Communities and Households (REACH), both supporting a community-driven development approach, helped Afghanistan better respond to community needs at the time of the pandemic. Through the Bank’s support, 90 percent of households in the country under the government’s “Dastarkhan-e-Milli” program, are benefitting. This amounts to an estimated 4.1 million households with incomes of $2 a day or less. The 30 forthcoming Early Targeting Action Warning for Food Security (ENETAWF “resilience” in Dari) project will establish a national disaster response and social protection early warning, early finance, and early action system. IFC through the on-lending arm of Tanzania’s National Microfinance Bank is able to reach out to support micro, small and medium enterprises (MSME). (Tanzania Working Capital Solutions (WCS) COVID NMBTZ project). The WCS is designed to provide funding to existing IFC client banks in emerging markets that will then extend new trade-related or working capital loans to companies whose cash flows have been disrupted by the global outbreak of the COVID- 19 pandemic. The Tanzania macroeconomic environment has been challenging over the past few years and GDP growth has slowed down following the COVID-19 pandemic and is expected to worsen following the emerging variants of the virus, which is expected to impact banking sector liquidity and asset quality and pose a significant risk to the recovery of the MSME sector. The Project falls under a facility, whereby IFC, acting as Implementing Entity of IDA Private Sector Window (PSW) Blended Finance Facility, is expected to support working capital loans in PSW eligible countries with a pooled first loss guarantee of up to US$215 million. MIGA, at a time of severe economic stress and uncertainty due to the COVID-19 pandemic, provided guarantees to free up regulatory capital, allowing for the expansion of a German bank’s green small and medium-size enterprises (SME) loans across eight Eastern European countries. Albania, Bosnia and Herzegovina, Georgia, Kosovo, Moldova, North Macedonia, Serbia, and Ukraine. MIGA’s support helped lay the foundation for sustained green growth in the impacted countries as MIGA guarantees supported climate finance SME transactions. Approximately 15 percent of MIGA’s support was directed in Kosovo, an IDA and FCS country. The WBG’s long-term relationship and in-depth country knowledge and know-how on global best practices prompted the Government of Vietnam to seek WBG assistance for a comprehensive package of policy advice during the pandemic. At the invitation of the Government of Vietnam, the WBG prepared a series of policy notes and was able to influence the Government’s response to the COVID-19 crisis, starting with the strengthening of health programs and shifting the course of fiscal and monetary policies to respond to the emergence of new vulnerabilities in the most affected sectors such as tourism and transport, and which has been assessed as one of the most effective in the world. More concretely, the authorities adopted the sequencing strategy advocated in the policy notes. The Bank, in close coordination with some other development partners, has prepared a policy framework that could be used to provide budget support if needed by the Government to close an eventual fiscal gap. 31 Annex C: Summary Findings from Country Outcome Pilots This annex provides a summary of four retrospective case studies, conducted in Calendar Year 2020, using a contribution analysis methodology based on desk reviews and stakeholder consultations (the fourth case study is being completed in FY21). The cases are: (i) Kenya Access to Electricity, (ii) Punjab (Pakistan) Primary School Enrolment, (iii) Peru Child Stunting, and (iv) Improving Fiscal Management in Indonesia (work in progress). Kenya Access to Electricity Summary: The WBG played a critical role in expanding access to electricity to a significant share of the population, including in informal settlements, rural areas, and lagging counties. The Bank’s financing, support for strengthening institutions, and partnerships were key to bring about and sustain this change. Through guarantees, the Bank de-risked investments, and leveraged financing from other development partners and the private sector. The Context In the 1990s, Kenya suffered acute power generation capacity shortages, low investment and heavy dependence on public funding, and limited access to electricity. In 1993, approximately 11 percent of Kenyans had access to electricity, while generation stood at 776 Mega-Watt (MW) in 1996. The limited generation, coverage, and capacity of the transmission and distribution grid affected the quality-of-service provision, led to unnecessarily high electricity losses, and prevented the country from extending electricity service to a larger population. Following implementation of a series of reforms, Kenya has created a vibrant power sector. Key sector utilities operate on commercial principles on an arms’ length relationship with private shareholding in Kenya Power (KPLC, the power distribution company) and in KenGen (the generation company). Power generation capacity now stands at 2,819MW and includes a well- diversified mix, with close to 90 percent of energy being generated from clean sources (mainly geothermal, hydro, and wind) with the possibility of a fully green grid in the near future. By 2019, about 75 percent of Kenyans had access to either on or off-grid electricity sources and at least US$3 billion in private capital was mobilized in the power sector between 1997 and 2018. Going forward, to sustain and advance the gains made so far, more focus is needed to address affordability/equity, quality and reliability of supply, high electricity losses, and financial difficulties faced by KPLC. 32 World Bank Group Contribution The WBG’s contribution to access to electricity was through financial support, technical assistance, advisory services and analytics, capacity building, policy dialogue with the government, and providing leadership among other development partners within the sector. More broadly, support may be categorized under policy and regulatory reforms, prioritization of the electrification agenda, increased public sector investment in geothermal exploration and development, and increased private sector investment in off-grid energy sources. Under the policy and regulatory reforms, the WBG became a respected convener and coordinated the efforts of other development partners in the sector. The Bank was the government’s lead partner in the reform process and played a key role in identifying critical reforms and helping the client build capacity to implement them. When Kenya prioritized electrification as a key development agenda, the WBG acted as a key partner to help advance the government’s electrification program. Specifically, the WBG provided financial resources for energy expansion and modernization, and technical assistance for the development of the National Electrification Strategy and the geospatial plan. As the country increased public sector investment in geothermal exploration and development, the WBG was pivotal as the first partner to support geothermal energy development, which gave confidence to other partners and eventually to the private sector to invest. Among other things, the WBG supported exploration and drilling wells, de-risking investments through its instruments such as Guarantees, and development of the National Geothermal Strategy, Dec. 2017 (yet to be adopted). On increased private sector investment in off-grid energy sources, the WBG played a lead role in articulating the need for the government to include off-grid renewables in the official energy mix, catalyzing private sector expansion of solar home systems. 33 What we were told (before the Outcome Orientation study) Available records show that the Bank invested approximately $2.5 billion in Kenya’s energy sector over the 20-year period and was one among many partners. Access to electricity was a key government priority, and good progress has been made overall – reaching 75 percent access rate through grid and off-grid options. The Government increased access in informal settlements, public institutions such as schools and hospitals, rural areas, and the lagging counties in the North and North Eastern part of the country. While this was a success story, there was no clear articulation of how gains in the sector, including access, had been achieved over the period and what the WBG’s contribution, among other players, had been. What we found out (after the Outcome Orientation study) The magnitude of outcomes was derived from long-term and sustained engagement of the WBG in policy reform, institution building, and mobilizing investment. Because of the short- term horizon of the Country Partnership Framework (CPF) cycles and the need for measuring direct and attributable outcomes, major contributions of the WBG engagement over a long-term period were not appropriately captured in the CPF closing reports. The Bank’s engagement in Kenya spanned 20 years in three phases, each of which focused on a different aspect of the reform and provided relevant support to the government to achieve the intended outcomes. Moreover, this long-term engagement involved coordination and partnerships with other development partners and the private sector. Feedback from a wide range of stakeholders made clear the importance of WBG contributions in helping to build and strengthen institutions. A key feature of the Kenyan energy sector reforms is the strong performance of the two publicly listed power companies, KPLC and KenGen. Following unbundling, reduction of political influence, the introduction of robust policy and legislation, and the advent of performance contracts, KPLC became a top performer in the region (despite the recent challenges that the company is facing). Assessments of long-term support depend on a systematic way of capturing long-term results. Periodic assessments and capturing of results through consultations with a wide range of stakeholders is necessary to document the long-term contributions especially in policy reform and institution building. The Bank regularly reflected with key client counterparts on what was working and what was not to inform course corrections. Accepting risks to demonstrate what works helped leverage contributions from other partners. Lessons learned from previous investments informed improved strategies in new investments. The WBG started with supporting the initial reform process, building institutions, and making pivotal investments. Each successive investment or reform program tried to address limitations or remaining binding constraints from the previous period or predecessor operation. For example, while initial operations mainly focused on reforms, addressing energy shortages, and creating generation capacity, subsequent projects invested in energy expansion, continuous improvements in efficiency of energy utilities, distribution, etc. The most recent project investments have focused on modernization of the sector, universal access, and transmission of electricity, as key areas that still need to be addressed. The compound effect of the WBG’s support has contributed to the transformation of Kenya’s energy sector. 34 Punjab (Pakistan) Primary School Enrolment Summary: The government of Punjab, in collaboration with the Bank, was able to achieve long- term impact in Punjab’s education system. The Bank used its entire arsenal of instruments, including investment projects, analytical work, and convening power. A linear path was not followed to reach the education outcome, it was rather an indirect pathway, in which the Bank brought a system-wide approach for education reform built on a shared vision, strengthened institutions to implement it, produced data to correct course and otherwise guide implementation based on evidence, and provided stability and mitigated risks with long-term funding and support. The Context Punjab (Pakistan) was able both to increase primary school enrollment and also almost achieve gender parity. Pakistan had close to 23 million children (aged 6-16) out of school in 2001, making it the country with the second largest out of school population after Nigeria according to official administrative data. Punjab, containing over half the population of Pakistan, contributed close to half of the total out of school population. In 2001, only 64% of children (age 6-10) attended school, which was somewhat higher than other provinces (Sindh: 56%, Balochistan: 40%, Khyber Pakhtunkhwa: 47%). The overall primary school enrollment rate for the province was much below the South Asia average of 76%. There was also a substantial gender gap in Punjab, with primary school age girls (age 5-10) being 8 percentage points less likely to be enrolled and adolescent girls (age 10-16) 16 percentage points less likely to be enrolled. Two decades later, after a long-term Bank engagement in Punjab, primary school enrollment has increased 21 percentage points to 85 % and gender parity has been almost achieved, even as Punjab’s population grew from 74 million to 110 million. Punjab now contributes just over a third of the estimated 19 million children that are out of school (age 6-16). By 2000, there was a realization that earlier initiatives in Punjab province fell short of reaching impact on primary school enrollment at scale. Key constraints were the lack of an overall governance structure and implementation capacity at various levels that could help encourage reform-oriented governance, which is sustainable in the longer run. All this required a more visionary and long-term engagement for the education sector that could survive changes in governments and priorities of development partners. 35 World Bank Contributions The Bank has supported the Punjab education sector combining technical assistance, investment projects totaling US$1.5 billion and support for implementation of evidence-based reforms. In retrospect, the Bank’s support over this period can be divided into two phases: (a) a comprehensive system-wide approach involving governance and finance reforms, and (b) prioritizing programs with a results-based focus. During the first phase, between 2001-2008, the Bank teams worked with the government of Punjab in shifting the education dialogue towards a comprehensive system-wide approach involving governance and finance reforms to align political leadership, finance, and education sectors behind the achievement of universal primary education (millennium development goals (MDG) goal 2). Analytical work on private school subsidies, a Public Expenditure Review (PER), and an impact evaluation on a conditional cash transfer program for girls’ enrolment fed into government policies and built evidence for implementing new strategies. In addition, the Bank financed a longitudinal panel survey following a set of rural schools in Punjab, which provided a deep understanding of the challenges facing school enrolment and quality. The Bank financed four Development Policy Operations (DPO) to support a series of reforms which targeted critical policies supporting the supply and demand sides of education, the delivery of education services, and monitoring mechanisms. At the end of this period, primary enrollment reached 77%. During the second phase, from 2009 until today, the Bank’s involvement shifted to supporting priority programs with a results-based focus through Investment Loans with Disbursement Linked Indicators. The Investment Project Financing (IPFs) with disbursement linked indicators (DLIs) contributed strongly to sector dialogue, as these became embedded in an overall program focused on predefined results, political ownership, and donor coordination. The 36 focus in the dialogue also shifted more decisively from access to quality of primary education (SDG goal 4). Continued impact evaluations on school vouchers in public-private partnership schools, private school participation, and teacher incentives fed into policy dialogue and allowed evidence-based decisions to expand the government’s strategy. At the end of this phase, primary school enrollment reached 85% (with girls enrollment at 84%, just 3 percentage points below the boys enrollment rate of 87%). The Bank’s long-term engagement and presence brought legitimacy and stability in the government educational agenda and was able to address institutional, financial, and sector challenges allowing the sustained increase of primary school enrollment. What we were told (before the Outcome Orientation study) Before the outcome pilot study was conducted, the story about the Bank’s contribution towards the Punjab primary school program was the following: (a) The Bank has supported the education sector combining technical assistance, investment projects totaling US$ 1.5 billion, and support for implementation of evidence-based reforms, over 20 years, and it was one among many partners. (b) The impact on girl enrolment was not fully articulated, as the Bank’s contribution towards the indirect pathway was not reviewed over the long-term in any of the project documents. Moreover, (c) while progress in school enrolment was well known, an intractable gender gap in the out-of-school population remained. What we found out (after the Outcome Orientation study) The outcome pilot study provided a deeper understanding of the Bank’s overall contribution towards the Punjab primary schooling program, with the following additional information: First, girls primary school enrollment increased substantially from 60% in 2000 to 84% in 2019, and the gender gap was significantly reduced. While project documents and data sources showed an increase in girls enrollment, the long-term results that the program had supported – and a reflection on why that happened - are not to be found anywhere in project documents, which typically reflect on shorter term changes. Second, the Bank investments were paired with technical support and high-level policy dialogue for strategic planning and impact evaluations to clarify the role of stipends in incentivizing girl enrollment. For example, a conditional cash transfer (CCT) program for girls attending school (grades 6-10) was introduced in the 16 districts of Punjab with the lowest literacy rates. This program aimed to incentivize families to keep their daughter in school, particularly during the time when girls would traditionally get married early. Over time, this program matured with families receiving 1,000 Pakistani Rupee (PKR) per month for keeping a girl in school. In addition, female teachers were recruited to attract more girl students, with 50 percent of the teachers hired under site-specific fixed term contracts being female. As a testament to the joint effect of improving both supply and demand interventions, the gender gap in enrolments narrowed considerably. According to key stakeholders interviewed from the Bank, the government, development partners, and researchers, these different initiatives were all key to incentivizing school enrollment. Such qualitative information was missed in Bank documents. Third, client and partners highlighted the Bank’s role in reducing risks, strengthening educational planning and leveraging financing from others. The Bank’s first pathway to increase primary school enrollment was through improving governance to enhance accountability and financing. There was also a deliberate effort to build a coalition of reform minded actors, in the government, the bureaucracy, and civil society. Even in the early 2000s, there were no accurate 37 data on schools and teachers, most of the education expenditure was spent on teacher salaries, and there was a slow transfer of funds and procurement, which were managed at a central level. By 2004, the government established an independent monitoring and evaluation system in the Project Management and Implementation Unit (PMIU), reporting directly to the Chief Secretary and later to the Chairman Planning and Development. The decision to place it outside the education bureaucracy was to allow for trustworthy and robust data from regular school visits by independent monitors. The information, coupled with a high-level policy dialogue, allowed for an evidence- based design of education policies regarding girls’ enrollment, missing facilities, and teacher absenteeism, among other things. The monitoring system also allowed a better tracking of various programs implemented in the sector. These critical contributions to strengthen the government’s ability to deliver effective services were not adequately highlighted in program documents as they were not easily quantifiable, extend into political economy issues, and would normally be considered implicit to the way in which the Bank aims for outcomes. Peru Child Nutrition Summary: The Bank formed a close partnership with the government on the journey to reducing child stunting, providing multisectoral support from governance and finance to health and social protection. The Bank helped the government in identifying pathways to outcomes. It started with helping to generate evidence that would enable the government to build evidence-based policies, and then supporting government initiatives through policy lending and project financing. Throughout the engagement, continuous analytical support was provided in parallel, including randomized control trials and process evaluations that fed into strengthening institutions, and into program quality. Studies suggest that in Peru, while economic growth alone explains up to half of the reduction in child stunting (Galasso and Wagstaff, 2016), the remaining half largely resulted from the design and operations of Peru’s multisectoral nutrition policy and program (Acosta, 2011). The Context Peru demonstrated a remarkable achievement in lowering child stunting in the past decade. With aim towards SDG2, Peru had been able to reduce child stunting rates to 13.1% by 2016, from a level of 31.3% in 2000. A significant reduction in child stunting rates was seen from 2008 -- a 15 points reduction in child stunting in 8 years (2008-16). The momentum in child stunting reduction has continued; by 2018, Peru’s child stunting rate (at 12.3%) was closer than ever before to the Latin American regional average (9.3%). Since 2007, Peru has saved more than 377,000 children from being stunted. Given its approach, Peru’s child stunting program of 2000 had a significant impact among marginalized groups, although differences still exist in child stunting between consumption quintiles (2010-2018) and between urban-rural areas (2005-16). Peru lacked a comprehensive strategy to address child stunting in the early 2000. A new approach had to be considered to address child stunting, and it had to complement existing programs on poverty alleviation and food assistance. It is sometimes believed that the supply- driven food supplementation program of the 1980s and 1990s was a strategy to address child stunting, but it was not; that program was developed to incentivize children (those who were of school-going age and those who were among the early childhood development group) to attend schools. Additionally, the program did not reach the desired population, as it suffered from fragmentation, poor design, and ill targeting. The international scientific community had come to 38 the conclusion that reducing chronic malnutrition had to be addressed with complementary services to affect behavior and increase access to food and to health, water and sanitation services. Peru’s child stunting crisis called for an urgent development of a national strategy to address the country’s own specific challenges related to child stunting. Civil society advocacy groups, support from international partners, and learning from international experiences helped to articulate an evidence-based blueprint for a nation-wide program adoption. The Bank (along with other partners) conducted analytical studies to show that a multisectoral approach was required to combat child stunting. There was a shift in direction on the pathways to reducing child stunting from one on hunger to one on public health. Fighting child stunting – a national agenda in the 2006 election – received the highest level of commitment and was a sustained political priority under four successive governments. The government integrated the fight against child stunting into an early childhood development approach targeted to the poor population (CRECER, 2007), under the leadership of the Presidential Council of Ministers (PCM). In 2007, a multisectoral policy focusing on results and performance was prepared for the nutrition program by the Ministry of Economy and Finance (MEF), and in 2009, cross-cutting anti-poverty and social inclusion policies were integrated under the new Ministry of Development and Social Inclusion (MIDIS), launched in 2011. The final program was a mix of a universal approach plus targeted interventions (in terms of increasing demand of services using social protection interventions, plus special focus for the “integrated” health and nutrition package in regions with the highest stunting rates). World Bank Contributions Key intermediate outcomes helped move the agenda towards high-level outcome (reduction in child stunting). From 2000 to 2016, the Bank provided lending and non-lending support of $700 million and $4.5 million respectively through a series of development policy lending (DPLs) and IPFs including technical assistance, and support through a continuous policy dialogue and 39 evidence building through advisory services and analytics (ASAs). Demand-side initiatives were supported by the -- Juntos – CCT and the health insurance programs, while supply-side initiatives were supported by health sector reform projects building the health and nutrition delivery system, and the governance aspects of results-based budgeting (RBB) for social sectors with regions and districts through the MEF. A careful assessment of the situation on the ground resulted in a diagnosis of the problem, which was then translated into the design of a result-based operation financed by a programmatic series of DPLs, supporting a careful transtion away from food supplementation towards integrated multisectoral support. It was the integration of the different instruments as well as the careful design of the operations that ultimately made a significant positive contribution to reducing stunting in Peru. What we were told (before the Outcome Orientation study) Before the outcome pilot study was conducted, the story about the Bank’s contribution towards the Peru Child Stunting program was well documented in various papers and a book on its success over the last decade. Peru also has hosted various country delegations to learn from their experience. It included the following: (a) The success of the child stunting program was in part driven by changes to the CCT program, but also in large part by the multisectoral efforts, combining health, social protection, WASH, and to a lesser account education sector. (b) The Bank’s downstream lending efforts were very important contributions to the ultimate outcome. Not much was said in any of the documents about the Bank’s contribution to long-term institutional strengthening towards high-level outcomes. What we found out (after the Outcome Orientation study) The outcome pilot study (using earlier documentation and conducting new ones) provides a deeper understanding of the Bank’s overall contribution towards Peru’s child stunting program, as the particulars had not been emphasized in previous documents as they had different purpose (supporting counterparts efforts). The study was able to describe in more detail the following additional information: First, the Bank’s support for institutional changes over time (unique ID Card for children, social monitoring strategy, Results-based Budgeting) was seen as critical to long-term sustainability. For example, the Bank sustained a program of technical assistance to help the MEF in establishing RBB for the social sectors. In subsequent years, successive Bank’s PERs assessed the distributional impact at subnational levels to further refine RBB. An IEG evaluation noted that “the Bank’s technical assistance in this process helped the MEF adopt RBB and a focus on children’s chronic malnutrition more quickly.” While stakeholders had a clear understanding of the scope of Bank contributions, existing Bank documentation on CPFs did not articulate these efforts and achievements in details as they had not yet been the purpose of any report. This study contributes as it highlights this aspect of the story. It was reconstructed retrospectively through the outcome pilot. Second, the Bank contributed decisively to efforts to build evidence. For example, the Bank conducted a randomized control trial and a process evaluation to help improve beneficiary coverage and quality of the health and nutrition services. These evaluations helped address shortcomings in the Juntos CCT program and were thus instrumental in addressing chronic malnutrition. In general, this learning agenda was not identified in CPFs, in part because it emerged organically to address implementation shortcomings. The contribution of these investments to development outcomes were also not systematically assessed as they were primarily about how 40 rather than what results were achieved. The pilot study found that the evaluations served both to strengthen strategy and implementation of programs, and to convene partners and leverage additional financing for the program. Improving fiscal management in Indonesia Summary: The Bank has been a close partner of Indonesia in supporting improved fiscal management since after the Asian financial crisis. Indonesia has built strong institutions of public financial management and undertaken important fiscal policy reforms, especially in reducing inefficient fuel subsidies. These have contributed to macro-fiscal discipline and a gradual reprioritization of spending towards areas such as social assistance and infrastructure. The reform agenda on fiscal management is not complete, but the Bank continues to work on all these issues and explore new openings for reforms. The Context Over the last two decades, Indonesia has improved its fiscal management. Prudent fiscal management has played an important role in supporting macroeconomic stability and growth, which averaged 5.3 percent over the period. Growth has supported poverty reduction, which has fallen from 19.1 percent in 2000 to 9.4 percent by 2019. After the 1997 Asian financial crisis, Indonesia was in a very weak macro-fiscal position with high debt, limited fiscal space, and low budget credibility. Poorly targeted fuel subsidies (3.9 percent of gross domestic product (GDP) or 20 percent of the budget in 2001) were a source of macro-fiscal volatility and a costly fiscal burden, crowding out more productive spending. Public financial management suffered from weak accountability without proper checks and balances, a high degree of centralization, unreliable data, opaque decision making, limited transparency, and a national budget that was “just one large spreadsheet”.45 The government of Indonesia at the time resolved to address the weaknesses in its macro- fiscal policy and related institutions. It embarked on a long-term program to put in place a more stable and sustainable macroeconomic framework, in part by revamping its fiscal and public financial management institutions. These efforts would also be supplemented during episodes of heightened fiscal pressures by reforms aimed at expanding fiscal space and improving the quality of spending. Public financial management (PFM) reforms aimed to increase budget credibility and performance. Overall, the quality of PFM has improved as shown by sustained improvements in Public Expenditure and Financial Accountability (PEFA) assessments from 2007 to 2017. Average PEFA scores in Indonesia now exceed basic levels consistent with good international practice. Fuel subsidy reforms aimed to create space for more productive and pro-poor spending. While the agenda remains unfinished and fuel subsidies have crept up again, the overall trend over the past 20 years has been downward. The reported fuel subsidy reached 0.3 percent of GDP or 1.8 percent of the budget in 2020, with an additional implicit subsidy provided through energy State-Owned Enterprises (SOEs).46 The reforms provided fiscal space for increased spending on 45 Based on stakeholder interviews. 46 Energy SOEs have been compelled to price their fuel at a fixed retail price below cost since 2017, but the implicit subsidy is estimated to be well below the explicit budgetary subsidy. 41 infrastructure, health, and social assistance. Social assistance increased from 0.3 percent of GDP in 2004 to 1.0 percent of GDP by 2020 with better targeting and increased benefits. World Bank Contributions The Bank supported improvements in fiscal management through a combination of technical assistance (TA), fiscal-policy- and PFM-focused DPLs, and IPFs focusing on productive spending. Since the Asian financial crisis, the Bank has invested heavily in its engagements on fiscal sustainability and budget credibility. It has implemented six programmatic DPLs and two IPFs with core components targeted at this policy area. In addition, large TA programs focused on macro-fiscal, poverty, and PFM47 further helped anchor the teams’ engagement with the government on strengthening fiscal management capacity. The Government Financial Management and Revenue Administration Project (GFMRAP), launched in 2004 with a decade- long implementation horizon, laid the groundwork for an integrated financial management information system for the Ministry of Finance (MoF), and the establishment of a Treasury Single Account (TSA). The PFM Multi-Donor Trust Fund (MDTF), now in its third phase since 2007, is supporting the MoF to improve efficiency, effectiveness, capacity, and accountability of Indonesia’s revenue administration and expenditure management, at both central and subnational levels. This resulted in substantial annual savings due to the implementation of the treasury single account and in an unqualified audit opinion of the central government’s financial statements from 2016 to 2019. On the revenue side, it contributed towards a significant increase in the number of registered individual and business taxpayers and their annual filing ratio (including e-filing). 47 The Support for Enhanced Analysis and Monitoring of the Indonesian Economy (SEAMIE), Support for Enhanced Macroeconomic and Fiscal Policy Analysis (SEMEFPA, both Australia), PFM MDTF (Canada, European Union, the Netherlands and Switzerland) and Partnership for Knowledge-based Poverty Reduction (PKPR, Australia). 42 On fuel subsidy reform, the Bank’s role has been discretionary and increasing over time. The Bank has recommended energy subsidy reforms since the 1990s, for example, through Public Expenditure Reviews, policy notes, and technical papers. In 2005, the Bank was advising the recently elected government’s taskforce on the design of compensatory social spending and providing communication training for ministers. In 2014-15, the Bank’s support to the presidential transition team was similar to 2005 but also expanded to assess the fiscal and distributional impact of subsidy reforms. DPLs from 2006 to 2013 supported reforms to improve targeting and expand social protection coverage in response to fuel price reforms. Fiscal DPLs between 2016 and 2019 and social protection operations helped cement compensatory spending, setting a path to increased coverage and benefits over time. More recently, the Bank has been involved in the design of a planned liquified petroleum gas subsidy reform, which the government has postponed due to the pandemic. What we were told (before the Outcome Orientation study) Before the outcome pilot study was conducted, the prevailing narrative within the Bank about its contribution towards Indonesia’s fiscal sustainability and budget credibility was the following: (a) Bank support was critical to the success of the two major phases of fuel subsidy reforms. Strong advocacy by the Bank, especially in the leadup to the second phase, helped to trigger government reform efforts. The Bank was able to offer the government invaluable technical expertise on the macroeconomic and poverty impacts of reforms, and on social protection design. The specific details of these contributions, their timeline, the factors that put the Bank in a position to offer those contributions, and whether external stakeholders agreed with the Bank’s views, were all unclear prior to the study. (b) The Bank was critical in supporting a large-scale transformation of PFM in Indonesia following the Asian financial crisis. After the end of the International Monetary Fund’s (IMF) structural adjustment program in the early 2000s, the Bank quickly became the government’s preferred and leading partner on PFM reform. The Bank helped restructure the MoF’s PFM, implement transformative new fiscal and public finance legislation, and set up critical PFM reforms such as: performance-based budgeting, efficient treasury single account, Financial Management Information System (FMIS), accrual accounting and reporting systems that were missing. These activities were made possible by long-running trusted Bank engagement, several PFM-focused development policy operations, strong bilateral donor support channeled through flexible trust funds, and a large client-facing PFM team. What we found out (after the Outcome Orientation study) The study broadly confirmed that the Bank, along with other partners, made a substantive positive contribution to the government's achievements on PFM and fuel subsidies. Government counterparts and other development partners recognized these important contributions. The Bank benefited from and built upon the contributions of other partners, such as the foundations set by IMF programs or the Asian Development Bank’s (ADB’s) support for the State Accountability and Audit Law 2004 implementation. The Bank’s contributions were underpinned by a broad set of factors including direct and indirect pathways such as technical expertise, in-country staff presence, and long-term 43 trusted relationships built over a period. Government counterparts highlighted the Bank’s ability to offer cross-country expertise grounded in local experience. This was enabled by an extensive Bank staff footprint in Indonesia.48 This footprint – the largest among development partners – also made it possible for the Bank to provide a level of support that other partners sometimes could not.49 The study revealed a history of close long-term working relationships between the Bank and government counterparts, which has likely served as a foundation for ongoing trust. The Bank’s contribution to long-term outcomes can be better understood by considering a broad spectrum of Bank engagements over time, rather than specific activities during critical episodes. 48 Indonesia hosts the Bank’s largest client-facing office and largest client-facing macro-fiscal and PFM teams. 49 Based on interview remarks from a development partner. 44 Selected References Bengo, I., Arena, M., Azzone, G., Calderini, M. 2016. Indicators and Metrics for Social Business: A Review of Current Approaches. Journal of Social Entrepreneurship, 1 (2), 1-24. Binnendijk, A. 2000. Results Based Management in Development Co-operation Agencies: A Review of Experience, OECD DAC Working Party on Aid Evaluation. Independent Evaluation Group. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. World Bank, Washington, D.C. ________. 2016b. Results and Performance of the World Bank Group: An Independent Evaluation. World Bank, Washington, DC. February 21, 2017. ________. 2020a. The World Bank Group Outcome Orientation at the Country Level: An Independent Evaluation. World Bank, Washington, DC. October 16, 2020. _______. 2020b. Results and Performance of the World Bank Group: An Independent Evaluation. World Bank. Washington, DC. November 30, 2020. Moland Gaarder, M., Bartsch, U. 2014. "Who cares about development outcomes? Market failures and the role of the evaluation function," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(4), pages 361-377, December. Multilateral Organization Performance Assessment Network. 2016. The World Bank Institutional Assessment Report. Organization for Economic Cooperation and Development/Development Assistance Committee. 2006. Sourcebook on Managing for Development Results. OECD Publishing, Paris. Organization for Economic Cooperation and Development. 2017. Results in Development Cooperation, Provider Case Studies: World Bank Group. Roberts, Dawn, Khattri, Nidhi. 2012. Designing a Results Framework for Achieving Results : A How-to Guide. Washington, D.C.: World Bank Group. Vähämäki, J. and C. Verger. 2019. "Learning from Results-Based Management evaluations and reviews". OECD Development Co-operation Working Papers, No. 53, OECD Publishing, Paris. World Bank. 2003. Country Assistance Strategies: Retrospective and Future Directions Operations. Policy and Country Services. World Bank. Washington, D.C. March 12, 2003. ________. 2005. Results Focus in Country Assistance Strategies: A Stocktaking of Results-Based CASs. Operations Policy and Country Services. World Bank. Washington, D.C. February 23, 2005. ________. 2009. Country Assistance Strategies: Retrospective and Future Directions. Operations Policy and Country Services. World Bank. Washington, DC. September 1, 2009. ________. 2014. Kenya - Country Partnership Strategy for the Period FY14 – FY18. World Bank. Washington, D.C. May 08, 2014. ________. 2019a. Development Results in the World Bank Group . Executive Directors’ Seminar. Outcome Orientation in the World Bank Group. July 29, 2019. ________. 2019b. World Bank Knowledge Sharing and Reuse Incentive Framework . World Bank. Washington, D.C. ________. 2019c. Performance and Learning Review of the Country Partnership Strategy for Mongolia for the Period FY13-FY18. World Bank. Washington, DC. November 13, 2019. ________. 2020a. Focusing on Outcomes and Managing Risks: Helping Countries achieve the SDGs. Document for discussion at a meeting of the Committee on Development Effectiveness on April 22, 2020. ________. 2020b. Strengthening Outcome Orientation: An Update. Document for discussion at a meeting of the Committee on Development Effectiveness on August 31, 2020. August 21, 2020. ________. 2020c. Liberia - Country Partnership Framework for the Period FY19 - FY24. World Bank. Washington, D.C. October 26, 2018. 45 ________. 2020d. Strategy for Fragility, Conflict, and Violence 2020–2025. World Bank. Washington, D.C. February 27, 2020. ________. 2020e. Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework. World Bank. Washington, D.C. Zwart, R. 2017. Strengthening the Results Chain: Synthesis of case studies of results-based management by providers, OECD Development Policy Papers, No. 7. 46