World Bank Group Support to Health Services Achieve me nts a nd Challe nge s AN INDEPENDENT EVALUATION © 2018 International Bank for This work is a product of the staff of The RIGHTS AND PERMISSIONS Reconstruction and Development / The World Bank with external contributions. The material in this work is subject World Bank The findings, interpretations, and to copyright. Because The World 1818 H Street NW conclusions expressed in this work Bank encourages dissemination Washington, DC 20433 do not necessarily reflect the views of of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they noncommercial purposes as long as full represent. attribution to this work is given. Attribution—Please cite the report as: World Bank. 2018. World Bank The World Bank does not guarantee Any queries on rights and licenses, Group Support to Health Services: the accuracy of the data included in including subsidiary rights, should be Achievements and Challenges this work. The boundaries, colors, addressed to Independent denominations, and other information World Bank Publications, The World Evaluation Group. shown on any map in this work do not Bank Group, 1818 H Street NW, Washington, DC: World Bank. imply any judgment on the part of The Washington, DC 20433, USA; fax: World Bank concerning the legal status 202-522-2625; e-mail: pubrights@ of any territo y or the endorsement or worldbank.org. Cover Photo: janews / acceptance of such boundaries. Shutterstock.com. World Bank Group Support to Health Services ACHIEVEMENTS AND CHALLENGES AN INDEPENDENT EVALUATION Careful observation and analysis of program data and the many issues impacting program efficacy reveal what works as well as what could work better. The knowledge gleaned is valuable to all who strive to ensure that World Bank goals are met and surpassed. contents abbreviations  vi acknowledgments  vii overview  viii management response  xix management action record  xxv 1. Health Services and Development  1 Health Achievements and Challenges 2 World Bank Group Strategies 4 Intervention Logic and Evaluation Methodology 6 Report Structure 12 2. World Bank Group Support in Countries  15 the Results and Performance of 17 Portfolio Characteristics World Bank Group 2015 Aspects of Project Design 21 Alignment with Country Needs 23 Project Outcome Ratings 24 AN INDEPENDENT EVALUATION Achievement of Project Development Objectives 26 Selected Delivery Mechanisms for improving Health Services 32 World Bank Support for Pandemic Preparedness and Control 34 Synergies and Complementarities 37 3. World Bank Group Support to Global Partnership Programs  44 The Global Health Landscape 45 4. Conclusions and Recommendations  50 The Drivers of Universal Health Coverage 50 Country-Level Support to Health Services 52 The World Bank Group’s Role in GPPs Careful observation 54 and analysis of program data Recommendations54 and the many issues impacting program efficacy reveals what References  works as well as 56 what could work better. The Boxes knowledge gleaned is valuable to all who strive Box 1.1. Health, Nutrition, and Population Global Practice: Updated Priority Directions and Organizational Model  7 to ensure that World Box 2.1. The Health Results Innovation Trust Fund  35 met and Bank goals are surpassed. Box 2.2. Lessons Learned from Early Initiatives to Create Markets for Health  38 Independent Evaluation Group | World Bank Group iii Figures Figure 1.1. Evolution of World Bank Engagement in Health 5 Figure 1.2. The Evaluation Intervention Logic 9 Figure 2.1. Evolution of World Bank Group Health Services Portfolio, FY05–16  17 Figure 2.2. World Bank Project Financing: Commitments and Number of Projects, FY05–16 19 Figure 2.3. Recipients of IFC Investment and Advisory Services 20  ctivities Supported by World Bank–Financed Projects  Figure 2.4 A 21 Frequency of Project Development Objectives, by Institution  Figure 2.5  27 Figure 3.1. The Network of World Wide Web Co-Citation of Aid Organizations 46 Tables Table 1.1. World Bank Support to Health Services, FY05–16  8 Table 2.1. World Bank Health Services Project Financing: Financing Sources  18 Appendixes Appendix A. Methodological Approach Appendix B. Portfolio Analysis Appendix C. Assessment of Global Partnerships Programs Appendix D. Analysis of Service Delivery and Behavior Change Appendix E. Case Study Analysis of Selected Interventions Appendix F. Case Study Analysis of Selected Countries Appendix G. Social Network Analyses Appendix H. 2014 Reform Effort Survey iv World Bank Group Support to Health Services | Contents abbreviations AIDS acquired immune deficiency syndrome AS Advisory Services ASA Advisory Services and Analytics CCT conditional cash transfer DAC Development Assistance Committee DAH Development assistance for health EBRD European Bank for Reconstruction and Development FCV fragility, conflict, and violence GFF Global Financing Facility GP Global Practice GPAI Global Program on Avian Influenza GPP global partnership program HIA Health in Africa HIV human immunodeficiency virus HNP Health, Nutrition, and Population HRITF Health Results Innovation Trust Fund IBRD International Bank for Reconstruction and Development IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation IHP+ International Health Partnership IS Investment Services M&E monitoring and evaluation MDG Millennium Development Goal MDTF multi-donor trust fund MIGA Multilateral Investment Guarantee Agency MS+ moderately satisfactory or better OECD Organisation for Economic Co-operation and Development PBF performance-based financing PDO project development objective Independent Evaluation Group | World Bank Group v PPP public-private partnership RBF results-based financing SDG Sustainable Development Goal SGBV sexual and gender-based violence SNA social network analysis S+ substantial or better SWAP sectorwide approach UMIC upper-middle-income country UN United Nations UNICEF United Nations Children’s Fund USAID U.S. Agency for International Development WHO World Health Organization All dollar amounts are U.S. dollars unless otherwise indicated. vi World Bank Group Support to Health Services | Abbreviations acknowledgments This evaluation was prepared by an Independent Evaluation Group (IEG) team led by Antonio Giuffrida and Maria-Elena Pinglo (co-task team leaders) and included Anna Aghumian, Marco Boggero, Cristiano Buizza, Javier Lanza Ferrera, Gisela Garcia, Victor Malca, Eduardo Fernandez Maldonado, Maria Gabriela Padrino, Daniel Williams, Catherine Seya, Zukhra Shaabdullaeva, Nazumi Takeda, and Mercedes Vellez. Aline Dukuze was responsible for administrative aspects. Interventions case studies were prepared by Anna Aghumian, Scott Featherston, Willy De Geyndt, Cecilia Jade Kern, Hiroyuki Hatashima, and Mercedes Vellez. Country case studies were prepared by Erik A. Bloom, Tanu Duworko, Kavita Mathur, Muhammod Abdus Sabur, Asita De Silva, Judith Twigg, and Raluca Zoitanu. Social network analyses were produced by Michele Coscia, Katsumasa Hamaguchi, and Jozef Leonardus Vaessen. The IEG framework for the evaluation of service delivery was applied to this evaluation by Susan Caceres and Judith Hahn Gaubatz, and the framework for the evaluation of behavior change by Maria Ann Dumpert and Ann Flanagan. The team is grateful to various managers and staff of the World Bank and the International Finance Corporation as well as country directors, sector staff, government officials, development partners, and beneficiaries in case study countries for their input and time. The evaluation also benefited from the guidance of Jozef Leonardus Vaessen (adviser, Office of the Director-General). The evaluation was conducted under the supervision of Soniya Carvalho (acting manager), Emanuela Di Gropello (manager), Marie Gaarder (former manager), and Auguste Tano Kouame (director), and under the overall direction of Caroline Heider (Director-General, Evaluation). Peer reviewers for this evaluation were Pedro Pita Barros (professor of economics at Universidade Nova de Lisboa), Mead Over (senior fellow at the Center for Global Development and former Lead Health Economist in the Development Research Group of the World Bank Group), William Savedoff (senior fellow at the Center for Global Development), and Leslie Faye Stone (lead economics specialist at the Office of Evaluation and Oversight of the Inter-American Development Bank). Independent Evaluation Group | World Bank Group vii highlights 1 The World Bank Group has adopted health- related goals at the country and global levels, actively participating in the global call for universal health coverage and heeding Overview Sustainable Development Goal 3 while reaffirming its focus on “investing in people.” 2 World Bank Group support to health services is aligned overall with countries’ health needs and the key drivers of universal health coverage. 3 The performance of World Bank–financed closed and evaluated health projects has improved significantly during FY05–16, but the performance of the health portfolio remains slightly below the World Bank Group overall average. International Finance Corporation (IFC) Investment and Advisory Services in health perform better than their overall IFC comparators. 4 World Bank project financing and IFC investments supporting health services seldom monitor and evaluate all dimensions of quality (structure, process, and outcomes) relevant to the intervention. Additionally, the projects’ distributional impacts are rarely monitored and evaluated when specific disadvantaged population groups are identified as beneficiaries. viii 5 The potential of private provision to contribute to universal health coverage is constrained by the limited success of integrating it with public financing. Progress toward private provision will require joint efforts by the World Bank and IFC. 6 The World Bank Group performance in pandemic preparedness and response has improved through successive pandemic outbreaks, but World Bank Group support is not fully mainstreamed into operations and policy dialogue. The World Bank’s experience suggests that to respond to and control a pandemic threat effectively, efforts must be integrated with the client country’s health system and sustained over time. 7 The complexity of the global development landscape, the appearance of new actors, and the proliferation of partnerships globally require an improvement in the strategic focus of the World Bank Group’s participation in global partnership programs. Independent Evaluation Group | World Bank Group ix A L A R G E S H A R E of the world’s population today has access to health services, but the health needs of a great segment of the global population remain unmet. In 2013, more than 400 million people worldwide were not receiving at least one of the seven essential health services identified as priority areas in the Millennium Development Goals. Demographic and epidemiological changes, the increasing importance of noncommunicable diseases, the effects of climate change and natural disasters, and the surge of pandemic threats compound an already challenging situation in many countries. Challenges to health services are exacerbated further in countries facing fragile and conflict-affected situations. The global community has given itself the goal of reaching universal health coverage. In 2014, the World Bank joined a global coalition of more than 500 leading health and development organizations that called for acceleration in universal health coverage to ensure that everyone, everywhere can access quality health services when needed without being forced into poverty. The 2015 United Nations General Assembly embraced universal health coverage among the targets for Sustainable Development Goal 3 (SDG3): “ensure healthy lives and promote well-being for all at all ages.” The World Bank Group’s recent strategic focus acknowledges the importance of health services to achieve the twin goals. Ending extreme poverty and promoting shared prosperity in a sustainable way requires expanding service delivery for the poor, or the bottom 40 percent. In “Forward Look: A Vision for the World Bank Group in 2030: Progress and Challenges,” presented to the Development Committee in March 2017, the management of the World Bank Group institutions stated that “investing in people” (of which health is an important aspect) is one of the three ways through which the twin goals ought to be pursued. The World Bank Group Human Capital Project—the focus of the World Bank Group president’s speech at the 2017 annual meetings—confirms the central role that health will continue to play in the World Bank Group’s strategic focus. The drivers of universal health coverage have been articulated in the World Bank Group’s strategic focus in the health sector for about two decades. Since the 2007 health strategy, World Bank Group support to health services has focused more on results while sustaining its emphasis on the needs of the poor, health systems performance, and sustainable health financing. The “One World Bank Group” approach, which was presented in the 2013 World Bank Group’s corporate strategy and aspires to provide the best development solutions—regardless of whether they are public or private— was reflected in the 2015 joint World Bank and International Finance Corporation (IFC) “Technical Briefing to the Board: Joint World Bank Group Approach to Harnessing the Private Sector in Health.” The joint approach promotes the World Bank Group’s unique role in helping client countries achieve universal health coverage and the twin goals by harnessing the private sector. The Health, Nutrition, and Population (HNP) Global Practice (GP), created in 2014, adopted the mission of better connecting global and local expertise within the World Bank Group to assist client x World Bank Group Support to Health Services | Overview countries in accelerating progress toward universal health coverage through financial protection, service coverage, and healthy societies. To this end, the HNP GP has adopted a number of programs and initiatives. These include the Global Financing Facility in Support of Every Woman Every Child, a global coalition, launched in July 2015; the Power of Nutrition, a global partnership launched in 2015; the Global Tobacco Control Program, launched in July 2015; and the Pandemic Emergency Financing Facility, a quick-disbursing financing mechanism approved in July 2017. This evaluation aims to assess the roles and contributions of the World Bank Group in supporting health services in client countries. The evaluation also seeks to provide lessons and recommendations for achieving greater development effectiveness in future support to health services. Health services include all services dealing with the diagnosis and treatment of disease or the promotion, maintenance, and restoration of health. The evaluation’s intervention logic recognizes that the World Bank Group contributes to universal health coverage through its support to health services. The World Bank Group supports health services directly through project financing and Advisory Services and Analytics (ASA). The World Bank Group also supports health services indirectly by engaging in global partnership programs (GPPs) that bring like-minded development partners together to move toward shared objectives. This evaluation aims to fill an evaluative evidence gap in the health sector. It is the first comprehensive health sector evaluation carried out by the Independent Evaluation Group (IEG) since 2009. This evaluation also complements the 2014 IEG health financing evaluation, which examined how World Bank Group support to revenue collection for health, pooling of health funds and risks, and health financing reforms have improved equity in health financing and service use, financial protection, and efficiency. The evaluation uses a mixed method approach and triangulates evidence from various data sources. Evaluation methodologies include (i) development of the evaluation’s intervention logic; (ii) a portfolio review of World Bank Group–financed projects and activities analyzing project design features, results, indicators, and drivers of success and failure; (iii) intervention case studies of selected delivery mechanisms (conditional cash transfers [CCTs], performance-based financing [PBF], and the private provision of publicly financed health services) and of the World Bank’s response to pandemic outbreaks; (iv) six country case studies, three involving field missions (Bangladesh, Liberia, and Romania) and three involving desk-based studies (Brazil, the Philippines, and the Republic of Yemen); (v) the analyses of the health GPPs the World Bank Group is currently engaged with; and (vi) social network analyses to assess interactions between the World Bank Group and development partners in health. The Evaluation Portfolio The evaluation portfolio comprises two overlapping groups of projects. First, for assessing the evolution of the World Bank Group’s support, the evaluation portfolio includes all of the World Bank Group’s financed projects and activities supporting health services approved during the period from Independent Evaluation Group | World Bank Group xi fiscal year (FY)05 to FY16. This group comprises about 31 GPPs, 619 World Bank–financed projects with $22.8 billion in commitments, 1,033 World Bank ASA with a value of $262.9 million, 124 IFC Investment Services (IFC IS) projects with a commitment of $2.7 billion, and 67 IFC Advisory Services (IFC AS) with a commitment of $71.4 million. To compare the evolution of projects’ design features, the evaluation breaks down this portfolio into projects that are closed (but not necessarily evaluated) and projects that are still open. Second, for assessing project performance over the evaluation period, the evaluation portfolio includes 259 World Bank–financed projects, 28 IFC investments, and 16 IFC AS projects that were approved, closed, and evaluated through self-evaluation and IEG validation during FY05–16. Multilateral Investment Guarantee Agency projects are not included in this portfolio; thus recommendations apply only to the World Bank and IFC. The performance of World Bank–financed health projects has improved markedly during the evaluation period. In FY05–16, 71 percent of these projects received ratings of moderately satisfactory or better (MS+), which is just 2 percentage points below the performance of the entire World Bank portfolio. Starting in FY10, the health services portfolio performed better than the overall World Bank portfolio, with 74 percent of projects achieving a rating of MS+, compared with 71 percent for the complete World Bank portfolio. This is a marked improvement. The 2009 health evaluation, Improving Effectiveness and Outcomes for the Poor in Health, Nutrition and Population. An Evaluation of World Bank Group Support Since 1997, found that the performance of the health portfolio lagged significantly behind the overall World Bank portfolio from FY05–07 to FY07–09. Much of the improvement since FY10 was driven by a turnaround in health project performance in the Africa region. IFC investments in health services perform better than the overall portfolio. About 75 percent of IFC IS projects are rated MS+, a significantly better performance than the overall IFC investment portfolio at 57 percent. About 64 percent of IFC AS projects are rated satisfactory or better, which is above the overall IFC AS portfolio of 58 percent. Focusing on Client Needs The evaluation finds that World Bank Group support to health services shows good alignment with the health needs and priorities of client countries. World Bank project financing shows a positive correlation with both the overall burden of disease and the relative importance of disease burden in the client countries. The health focus is also generally consistent with country epidemiological transitions and income levels. World Bank–financed projects supporting health services focus primarily on disease prevention activities and on the primary care level. Support for maternal and child conditions, which disproportionately affect the poor, is strong and has increased over time from 39 percent to 60 percent between closed and open projects. The focus on prevention, primary care, and maternal and child care is stronger in low-income countries and fragility, conflict, and violence situations. These trends confirm the improvements made since the 2009 IEG health sector evaluation. xii World Bank Group Support to Health Services | Overview IFC’s Investment and Advisory Services projects largely reflect the health priorities of middle-income countries. IFC focuses more on noncommunicable diseases and general health. IFC’s support is divided more evenly among the three levels of care, but it largely focuses on treatment, with slightly more emphasis on secondary care. IFC’s support through investments has mostly concentrated on lower- and upper-middle-income countries. Most support goes to hospitals, clinics, and pharmaceutical companies with a high concentration in large markets. IFC’s investments in the health sector show a high frequency of repeat support to client networks. Although additionality in repeat interventions is still needed, the evaluation finds that realization of incremental additionality seems to diminish over consecutive operations. Similar to IFC IS, IFC AS assistance is concentrated on lower-middle-income countries. Public-private partnerships (PPPs) are represented heavily in IFC advisory support, accounting for about 69 percent of total projects. About 13 percent of IFC AS is now mapped to the HNP GP, which manages the Health in Africa Initiative. Focusing on the Key Drivers of Universal Health Coverage The World Bank Group is addressing all the key drivers of universal health coverage, though with different levels of emphasis across the key drivers. The most frequent objectives sought by World Bank Group–financed projects are the improvement of access, quality, or health systems. Objectives related to health outcomes and equity are pursued less frequently. The design of recent projects approved toward the end of the evaluation period appears to incorporate good practice features such as higher focus on quality of health services and results-based approaches. This is a welcomed development. However, there is room for further improvements, and the findings and recommendations of this evaluation can help guide the World Bank’s future health services support to maximize health outcomes. Improve access. The World Bank has made substantial contributions to improving access to health services. It is an objective in 54 percent of World Bank–financed projects approved during the evaluation period and was achieved in 70 percent of the 259 evaluated projects. Projects containing CCT and PBF interventions perform better in improving access (81 percent and 87 percent of access objectives were rated positively, respectively). IFC also contributed to improved access to health services; it is an objective in 88 percent of the investment projects and was achieved in 73 percent of its evaluated projects. However, because of the limitation of the monitoring frameworks of IFC projects, it is not possible to determine if they contributed to expanding coverage or to improving availability and use among those who were already covered. Improve quality. Quality improvement is an objective in 27 percent of World Bank–financed projects. Over time, World Bank–financed projects show, greater emphasis on improving the quality of health services (from 18 percent among closed projects to 44 percent among open projects), but they show only partial success. Only 46 percent of quality improvement objectives in evaluated projects Independent Evaluation Group | World Bank Group xiii have been rated positively (moderately satisfactory and above). IFC projects frequently included quality improvement objectives, but with a declining trend (from 27 percent among closed projects to 19 percent among open projects). Moreover, IFC quality improvement objectives and indicators focus on a narrow aspect of quality (structure). World Bank–financed projects show limited capacity to monitor and evaluate all relevant aspects of quality: structures, processes, and outcomes. Projects containing PBF interventions show better results in quality improvement and present stronger monitoring and evaluation frameworks and examples where all relevant quality dimensions are monitored with appropriate indicators. It is desirable to expand these good practices to the entire health services portfolio. Strengthen health systems. This is the second-most common project objective and is included in about 37 percent of World Bank–financed projects. However, the presence of this type of objective has been decreasing over time, even if health systems–strengthening activities are identified in about 90 percent of projects. Additionally, about 55 percent of such objectives were rated positively. Projects are more likely to achieve their health systems–strengthening objectives if the scope of the objective is well defined and if it is an area where the World Bank has accumulated significant experience. Improve equity. Few World Bank–supported projects have objectives that explicitly aim at improving equity. However, there seems to be an implicit equity focus in many projects. In fact, the majority of World Bank–financed projects (64 percent) target specific disadvantaged population groups (often the poor). IFC also strives to invest in clients with a strong focus on corporate social responsibility. However, the distributional impact of World Bank and IFC projects is unknown. World Bank Group– financed health services projects should be able to monitor and evaluate their distributional impacts when specific disadvantaged population groups are identified as the beneficiary. This would be advisable regardless of the project having an explicit equity objective. Improve health outcomes. Explicit objectives toward health improvement are present in 29 percent of World Bank–financed projects but with a reduced frequency over time (from 35 percent among closed projects to 16 percent among open projects) and in only 1 percent of IFC projects. About half of the World Bank–financed projects reached the desired health improvement objectives. Projects supporting CCT and PBF interventions performed better. The limited success is partially the result of the use of indicators (for example, mortality rate) that may not be sufficiently sensitive over the project life span and are subject to attribution challenges. The literature on clinical indicators for outcome aspects of health care quality improvement offers examples of indicators that are better suited to monitor and evaluate the impact of World Bank Group projects on health outcomes. Service Delivery Mechanisms: Public-Private Interaction and the Use of Incentives The World Bank Group has had limited success in integrating private provision with public financing. The World Bank and IFC articulated a joint approach to support universal health coverage in client xiv World Bank Group Support to Health Services | Overview countries with integrated public and private sector solutions. However, IFC investee companies continue to face challenges in blending private provision with public financing to improve access for the underserved. The main reasons for these challenges include the limited availability of public resources and capabilities; underdeveloped private markets for health services; and inadequate regulation and regulatory enforcement. Opportunities for synergy and collaboration between the World Bank and IFC in health have not been fully seized, despite the strong potential. The sample of six country case studies also found limited instances of complementarities between institutions. The experience of the Health in Africa Initiative shows that complementarities and synergies between the public and private sectors are difficult in low-capacity and low-resource settings, and that collaboration between World Bank and IFC teams is difficult without unified leadership at the country level. The HNP GP has recently appointed a global solutions lead for private sector engagement to facilitate the joint World Bank–IFC approach in health. This is an encouraging step. This evaluation confirms that the World Bank Group has improved its capacity to generate evidence on the effectiveness of financial incentives, such as CCTs and PBFs, although the use of the evidence generated in country-level support remains a work in progress. This is a significant improvement from the situation identified in the 2009 health sector evaluation, when IEG recommended boosting investments and incentives. The Health Results Innovation Trust Fund (HRITF) has been instrumental in stepping up the generation of evidence around results-based financing (RBF) in health through impact evaluations. The HRITF is now partnering with the Global Financing Facility to build country institutional capacity to scale up and sustain RBF within national health strategies and systems. Response to Pandemic Outbreaks The World Bank Group’s performance in pandemic preparedness and control has improved with successive pandemic outbreaks. The World Bank supported 63 countries under the Global Program on Avian Influenza Control and Human Pandemic Preparedness during 2006–13. However, the World Bank failed to sustain this effort. However, the World Bank was a vital member of the global coalition that fought the Ebola outbreak in West Africa in 2014–15. It quickly mobilized financial resources needed to stop the outbreak, and helped restore basic health services in the three West African countries affected by the pandemic. The Pandemic Emergency Financing Facility, approved in May 2016, is expected to accelerate the release of funds to respond to future outbreaks. A key lesson that has emerged from World Bank Group support in pandemic situations is that capable health systems are a necessary ingredient to mount a successful response to deadly virus outbreaks. They require adequately staffed health services, a supply of essential personal protective equipment, capacities for laboratory diagnosis, clinical management, and surveillance for quick diagnosis and rapid contact tracing. Independent Evaluation Group | World Bank Group xv Preparedness is the first line of defense. The commitment made under the 18th Replenishment of International Development Association (IDA) to support about 25 IDA countries in developing pandemic preparedness plans and frameworks for health emergency preparedness, response, and recovery is a step in the right direction. Synergies and Collaborations with Development Partners in Selected Countries The evaluation assessed the World Bank Group’s roles in country-level partnerships: its financing role, coordination function, and provision of technical assistance and knowledge. The six countries selected as case studies show partnerships of different complexity and different levels of in-country institutional capacity. The evaluation found that the World Bank Group’s financing role is more important, as a percentage of country total health expenditure, in the three countries with lower institutional capacity and more complex development partner networks. In the three countries with higher institutional capacity and relatively less complex development partner networks, financing was significant only during a crisis, and the World Bank responded relatively well when a crisis occurred. The World Bank’s coordination role is stronger in the countries with low capacity and complex development partner networks. In these countries, HNP GP leadership has been increasingly proactive in strengthening the World Bank’s coordination role. It is leveraging relevant GPPs, such as the Global Financing Facility and the UHC2030 platform (formerly the International Health Partnership). However, the evaluation identified opportunities to improve communication among donors in the field in all countries. In all six countries, the World Bank Group is recognized as a leader in providing technical assistance and knowledge in select areas relevant to health services. For example, the World Bank has supported the use of incentives (that is, CCT and PBF) and health PPPs in several countries. As could be expected, the World Bank Group is not usually seen as the overall knowledge leader in health at the country level. In the three countries with high government capacity, the World Bank Group chose more narrow areas of focus to add value—for example, monitoring and evaluation, targeting, and reorganization of service delivery networks. In these countries, IFC has been instrumental in the development of PPPs. In the three countries with low government capacity, the World Bank Group focused on addressing unique systemic constraints in a timely manner by providing, for example, financial support during the Ebola outbreak and support to mobile health. Making a Difference on the Global Stage The global health landscape has become more complex. This complexity arises primarily from the increase in the multiplicity of actors. The size and diversity of funding instruments adds to the intricacy. Development assistance for health has increased from $6 billion in 1990 to more xvi World Bank Group Support to Health Services | Overview than $37 billion in 2015. Although most of the increase in funding comes from members of the Organisation for Economic Co-operation and Development’s Development Assistance Committee, other important players are philanthropic foundations. Global partnerships and multi-donor trust funds increasingly complement traditional single-donor funding. The World Bank Group plays a central role in online interactions among global development actors. The evaluation used social network analysis to visualize how organizations operating in the global health landscape interact online. The World Bank relates directly or through GPPs with almost all relevant actors, it has a central position in the network, and it is potentially able to spread information effectively through the online network of development actors. The World Bank Group plays multiple and distinct roles in GPPs. The most common roles are as founding partner, governing partner, implementing partner, and trustee. World Bank support is often critical at the formative stage of global and regional partnerships. As a governing partner, the World Bank Group contributes to aligning partners’ priorities. As an implementing partner, it leverages partners’ resources to complement its own resources. The World Bank Group’s role as a trustee is limited to a few GPPs, among which are the Global Fund to Fight AIDS, Tuberculosis and Malaria, and Gavi, the Vaccine Alliance. Although the World Bank Group’s more recent global engagements in health seem to be selected more carefully, its overall global partnership portfolio includes partnerships that may have lost their relevance or have overlapping mandates. Some progress was made in selectivity in global partnerships in the past decade, but the evaluation found that there is room for achieving greater strategic alignment between the World Bank Group’s involvement in global partnerships and its institutional focus and comparative advantage. Recommendations Recommendation 1. Improve measurement of the quality of health services and the distributional effects of health services projects. The monitoring and evaluation framework of World Bank Group projects should include (i) appropriate indicators of the relevant dimensions of health service quality—structure, process, and outcomes, and (ii) the measurement of improvements of beneficiaries relative to nonbeneficiaries. Recommendation 2. Strengthen World Bank and IFC synergy to support public-private interactions in client countries to contribute to SDG3 and universal health coverage. (i) For World Bank, work with IFC to strengthen the planning, regulatory, and accountability arrangements for public-private interactions, and (ii) for IFC, work with the World Bank to crowd-in public financing for privately delivered services. The World Bank Group’s newly launched Maximizing Finance for Development approach, aimed at mobilizing finance for development by focusing on upstream reforms where necessary to address market failures and other constraints to private sector investment, can be applied to achieve greater synergies between the public and private sectors. Independent Evaluation Group | World Bank Group xvii Recommendation 3. For sustainable capacity to address pandemics, systematically integrate preparedness plans and governance frameworks for pandemic control within the client country’s own health system in World Bank Group-financed projects and advisory services. Building on the commitment made under IDA18 to support health emergency preparedness, response, and recovery, the management of the World Bank Group institutions could seek to ensure that the World Bank’s project financing and ASA are not one-off responses outside the client country’s health system. Recommendation 4. Enhance the strategic alignment and selectivity of World Bank Group engagement in ongoing and future GPPs. A strategic review should apply clear selectivity criteria that reflect the World Bank Group’s comparative advantage and the broader global development agenda. It can inform the selectivity and relevance of ongoing and future GPPs, and a more effective use of resources needed for engaging in partnerships. xviii World Bank Group Support to Health Services | Overview management response M A N AG E M E N T O F T H E WO R L D B A N K G R O U P welcomes the Independent Evaluation Group (IEG) report reviewing the Bank Group’s support to health services over more than a decade (FY05–16). The report covers many topics, covering both World Bank operations and International Finance Corporation (IFC) investments, as well as selected subject areas such as pandemic preparedness, performance-based financing, and public-private integration. Management thanks IEG for its collaborative engagement with staff over the course of the evaluation. World Bank Management Response The 2015 United Nations General Assembly embraced universal health coverage (UHC) among the targets for Sustainable Development Goal number 3 (SDG3). The Bank Group acknowledges the importance of health services to achieving its twin goals. The Bank Group’s Human Capital project, along with other leading global initiatives related to health, confirms the pivotal role that health will continue to play in the Bank Group’s strategic focus. Unprecedented progress has been achieved during the evaluation period. For example, globally, the number of child deaths was reduced at a faster rate than at any other time in history, and the rate of reduction is now faster in Sub-Saharan Africa than in any other region. Yet despite similar progress on several fronts, the recent World Bank/ World Health Organization Global Monitoring Report estimates that annually at least 100 million people are pushed into poverty because of high health expenditures, and two billion people still lack access to essential health services. World Bank management appreciates IEG’s general finding that Bank Group support to health services is overall aligned with countries’ health needs and the key drivers of UHC and that the outcome performance of World Bank–financed closed projects has improved significantly during the review period. Management broadly concurs with the report’s recommendations and suggestions for further improvements. Many aspects highlighted by the report have been among the focus areas of the World Bank’s recent and ongoing efforts to strengthen and improve the resilience of health systems. These aspects include the following: Increased evidence-based approaches. As the evaluation noted, global health is a particularly complex and growing field of practice. The pace of innovation is accelerating as demand for health care grows. To be more strategic in this space requires a firm foundation of evidence-based approaches and decision-making. The World Bank has explicitly sought to establish catalytic links between financing and service delivery and has stepped up its effort to establish robust monitoring and evaluation systems. To strengthen the link between spending and outcomes, the World Bank has designed and implemented performance-based financing across a large set of low- and middle- income countries and has invested heavily in impact evaluations that are beginning to shed critical insights. Pandemic preparedness and response. The evaluation notes that “World Bank pandemic performance improves with successive pandemic outbreaks.” Since the end of the evaluation period the World Bank has been at the forefront of developing and successfully launching a number of innovative initiatives to strengthen pandemic preparedness and response: the Pandemic Emergency Financing Facility (PEF), which can immediately release resources to a country that fulfills the eligibility Independent Evaluation Group | World Bank Group xix criteria and requests financing; the Coalition for Epidemic Preparedness Innovations, an alliance to finance and coordinate the development of new vaccines to prevent and contain infectious disease epidemics; and the use of Catastrophe Deferred Drawdown Option (CAT-DDO) in both International Bank for Reconstruction and Development (IBRD) and International Development Association (IDA) countries to cover the risk of an epidemic outbreak. Engagement of private sector. The evaluation highlights the challenge of integrating private provision of health services into public financing. In this regard the Bank Group has made some notable progress. Large health projects—for example, the System Enhancement for Health Action in Transition Project in Afghanistan—have successfully engaged nonstate actors for service delivery at national scale. The Global Financing Facility for Women and Children is an example of a global initiative that systematically engages the private sector to provide higher quality, improve the efficiency of and equitable access to essential health services, and more explicitly link these private sector initiatives to public financing. Management would like to emphasize a few aspects related to the broader context that have implications for accelerating progress toward UHC. Improving quality of services. Quality is indeed an important aspect that requires even more attention and innovation. The World Bank has increasingly shifted its efforts from constructing buildings and buying drugs to tackling the structural problems of limited country capacity and poor service quality, with an increased focus on measuring quality in the development objectives of health projects. A few examples would help illustrate the World Bank’s increased attention to quality of services. First, the World Bank has intensified its effort to strengthen measures of quality of care that go beyond structural improvements and shed light on knowledge and implementation challenges. In particular, the Primary Health Care Performance Initiative is a global monitoring initiative, co-led by the World Bank and the World Health Organization, that brings greater attention to innovative ways of measuring the performance of health systems, including special attention to measures for high- quality services. Second, the World Bank’s work across many countries to establish a purchaser- provider split has enabled purchasing agencies to use toolkits and, increasingly, client feedback to prioritize quality outcomes in health facilities. Third, measures of quality are also embedded in the results-based operations that constitute roughly 38 percent of the overall World Bank health, nutrition and population (HNP) portfolio. Fourth, many health projects support quality improvements to enable primary health care facilities to receive accreditation and qualify for performance-based contracting with purchasers (for example, health insurance). Finally, the World Bank has established a Quality of Care Community of Practice whose primary objective is to help HNP operational staff understand interventions that can help improve quality of services and identify indicators that can measure the several dimensions of quality. Greater attention to equity. The World Bank has put a great deal of effort and resources into bringing attention to distributional issues. First, financial protection is an explicit goal of UHC, and the World Bank was instrumental in including one SDG indicator to measure the proportion of people falling into poverty because of high out-of-pocket health expenditures. Second, over the evaluation period the World Bank has focused more systematically on strengthening surveillance systems and programs for diseases that predominantly affect the poorest and that have been relatively neglected, such as tuberculosis. Third, in addition to targeting through geography and targeting diseases that predominantly affect the poor, the World Bank’s overall support has been more integrated, focusing on programs that yield maximum benefit for the poorest segments of the population. For example, xx World Bank Group Support to Health Services | Management Response the majority of projects focusing on maternal and child health, for which the World Bank has been providing increasing support, are pro-poor. A UHC Health Systems Global Solutions Area has recently been established in HNP Global Practice (GP) to facilitate the systematic integration of health financing, service delivery, and private sector policy issues into the GP’s operational processes. Greater efficiency and effectiveness in situations of fragility, conflict, and violence. Given the increasing importance of fragility, conflict, and violence (FCV) for the Bank Group, management believes that the experience of health service provision in complex FCV environments (involving war and displacement, or famine and cholera responses, for example) merits further analysis to derive lessons with broader applicability. The IEG evaluation could not capture the increasing support provided to refugees and internally displaced people (IDPs). For instance, in the Middle East and North Africa Region, the World Bank is providing support to help Jordan and Lebanon strengthen their health systems to address the issues posed by the growing number of refugees. In the Republic of Yemen, the health project (along with other projects in the social fund/cash transfer program) are supporting Yemenis, including IDPs. With the IDA18 Replenishment and the newly launched IDA18 sub-window for refugees, these areas remain a key priority for the World Bank. Although this evaluation could not capture the increasing support provided to refugees and IDPs, management hopes these initiatives will be captured in the IEG’s forthcoming evaluation on forced displacement. Comments on Recommendations Recommendation 1: Improve measurement of the quality of health services and the distributional effects of health services projects. As part of overall effort to improve monitoring and evaluation and ensure that the right mix of indicators is used, management remains committed to strengthening measurement of quality and distribution of health services. Monitoring and evaluation quality is also being closely monitored at the sector level by the Portfolio Monitoring Group. Specific indicators to measure the quality of services and the impact on beneficiaries versus nonbeneficiaries depend on the nature of the operation and what the operation is trying to achieve. The World Bank has a large number of operations that are addressing health issues that are related to quality of services and that disproportionately affect the poor, the disadvantaged, and the underserved. The feasibility of measuring several dimensions of health service quality and the distributional impacts varies in each context of project design and implementation capacity, and thus such measuring may not apply or be feasible for all projects. Recommendation 2: Strengthen World Bank and IFC synergy to support public-private interactions in client countries to contribute to SDG3 and UHC. The World Bank has been exploring options to ensure the successful implementation of the Maximizing Financing for Development (MFD) approach, in partnership with IFC and the Multilateral Investment Guarantee Agency. The World Bank and IFC work together on issues related to strategy, and they strive to create and seize operational opportunities. In particular, the Global Lead for Harnessing the Private Sector in the HNP GP interacts regularly with IFC, including providing significant input into the new IFC health strategy and giving feedback on all new IFC projects in health. The former “Health in Africa” team, originally based in IFC and now fully integrated into HNP GP, continues to work on Independent Evaluation Group | World Bank Group xxi better public-private dialogue and reduction of red tape, creating new opportunities for private sector engagement in several African countries. Recommendation 3: To develop sustainable capacity to address pandemics, systematically integrate, in World Bank Group–financed projects and advisory services in health services, awareness and preparedness plans and governance frameworks for pandemic control within the client country’s own health system. Strengthening country resilience by consolidating public health capacities as integral elements of strong health systems is part of work toward UHC and is at the core of the World Bank’s engagement in resource-constrained low-income countries. With a number of health sector projects in IDA countries in advanced stages of preparation, the World Bank is well positioned to meet the country preparedness targets set under IDA18. Following the 2014 Ebola outbreak, and recognizing of the need to strengthen institutional capacities for disease surveillance and control, African Heads of State have established the Africa Centers for Disease Control and Prevention (CDC) as a regional network to lead, integrate, and strengthen the continent’s public health institutions, capacities, functions, and partnerships to detect and respond to disease threats and outbreaks. The World Bank is working to mobilize substantive support for the Africa CDC, alongside partners such as the US Centers for Disease Control and Prevention, the World Health Organization, and the African Development Bank. The PEF has been put in place and is expected to support countries with financial resources for early surge responses. In addition, the World Bank has included disease outbreaks as eligible catastrophes for CAT-DDO to both IBRD and IDA countries. Finally, the World Bank is also leveraging global partners to consolidate financial and technical assistance for pandemic preparedness to IDA countries as a central element of UHC. Recommendation 4: Enhance the strategic alignment and selectivity of World Bank Group engagement in ongoing and future global partnership programs. The World Bank has increasingly consolidated its participation in existing global partnership programs (GPPs). Decisions to create or join new GPPs are based, first, on the World Bank’s comparative advantages and priorities as a development finance institution, and then on such factors as whether the GPP has a robust strategic engagement framework and there is sufficient policy development and implementation capacity at the country level. Under a resource-constrained environment, the World Bank has put more resources in fewer and larger GPPs that build on the World Bank’s comparative advantages—for example, the Global Financing Facility (GFF), PEF, and Power of Nutrition GPPs—and fewer resources on older GPPs, some of which have overlapping mandates. Withdrawal from such older GPPs has been done in a consistent but gradual manner to minimize disruption and sustain support to other partners, when such support is requested and adds value. Management agrees that a targeted strategic review would be helpful to map all existing GPPs and explicitly identify the type of engagement the World Bank should have in each of them, as well as guiding principles for current and future engagements to ensure optimal allocation of resources. Recognizing the importance of partnerships to find sustainable solutions to health challenges and the complex landscape of external actors, management will apply such guiding principles for engagement with flexibility and judgment. xxii World Bank Group Support to Health Services | Management Response IFC Management Response IFC management thanks IEG for a comprehensive report on the Bank Group support to health services. It particularly appreciates IEG’s constructive engagement during the evaluation process and the encouraging messages regarding IFC’s role. The report provides an effective overview and analysis of a period that was characterized by substantial change in the sector and in the Bank Group itself. To achieve UHC and the Bank Group’s twin goals, the Bank Group’s concerted efforts are of paramount importance. IFC is committed to using its resources to maximize the private sector contribution to the UHC goal, including through enhanced collaboration with the World Bank and other private and public partners. Before responding to the recommendations, IFC management wishes to address three points. First, IFC appreciates the report’s articulation of the key challenges faced by the private sector in health care, including the limited availability of public resources and capabilities, underdeveloped private markets for health services, and weak regulatory regimes. IFC plans to work with the World Bank’s HNP and other partners to overcome these challenges. Second, IFC is working on various initiatives that address the issues raised in the report and is collaborating closely with the World Bank’s HNP on some of them. For example, the two institutions are working together to develop an approach to MFD in health care. Key to MFD is to leverage sustainable private sector solutions and reserving public resources for where they are needed most or where private engagement is not optimal. As will be discussed in more detail, IFC has developed a health care service quality assessment tool that will assist health care facilities to improve the quality of care they provide. IFC is also exploring ways to improve access to healthcare by disadvantaged groups through the use of blended finance—–for example, investments that can be made alongside the GFF in support of Every Woman Every Child or the IDA Private Sector Window in IDA and fragile and conflict-affected countries—and potentially developing an impact platform to reach BOP (Base of Pyramid) patients and students in the health and education sectors. Third, IFC management notes some challenges encountered during the assessment, primarily a lack of evidence from IFC’s system for monitoring development impact. The implementation of Anticipated Impact Measurement and Monitoring will help address the challenges of results measurement and tracking of IFC activities. IFC uses metrics that can be reasonably tracked on an ongoing basis. In- depth evaluations of interventions whose impact cannot be tracked on an ongoing basis complement regular results tracking and monitoring. This includes evaluating the impact that IFC engagements on a project or sector level have on the broader health care system and public health. The report’s main conclusions and recommendations are well defined, and IFC is broadly aligned with them. Regarding recommendation 1, IFC management is pleased to share that it is working to incorporate new indicators of the relevant dimensions of health service quality into its new impact assessment and results tracking system (Anticipated Impact Measurement and Monitoring). These indicators are being piloted in a process that includes benefitting from interactions with IEG colleagues. They will measure the quality of IFC’s health care investments, including the processes for ensuring quality, such as the accreditation status of facilities, and the outcomes, such as infection rates. The new indicators will also include enhanced metrics on the structural determinants of the quality of health care. Appropriate indicators will be selected depending on the nature of the project being evaluated. In terms of measuring improvements that beneficiaries experience relative to nonbeneficiaries, IFC agrees on the importance of obtaining information on nonbeneficiaries for those Independent Evaluation Group | World Bank Group xxiii projects where this information is relevant to the development impact that the project is claiming. However, IFC notes that implementing systematic and continuous monitoring of such information across all projects is not considered to be a realistic approach because of resource implications. At the same time, few IFC clients collect information that would be important for analysis of this kind to protect the confidentiality of their patients’ personal information. Therefore, IFC management is considering evaluations of selected projects, focusing on the specific outcome metrics targeted by the IFC project. Regarding recommendation 2, IFC remains committed to working with the World Bank on initiatives to expand private provision. As recognized in the report, the later years of this evaluation period coincide with a series of positive changes that led to closer coordination between the World Bank and IFC. These largely occurred after the Bank Group HNP Roadmap exercise in 2015 that developed a joint Bank Group approach to harnessing the private sector in health. The World Bank and IFC also have shared experiences in the health sector in Africa and other programs. IFC will seek to capitalize on these coordinated engagements, which have been tested over the last few years. In particular, in the context of IFC’s 3.0 strategy, IFC will pursue synergies with the World Bank, including through the Cascade/MFD. Collaborative engagements and research are contemplated that focus on countries, with an emphasis on IDA countries, and in thematic areas such as how to promote better quality of care and how to further reach BOP beneficiaries. On quality, IFC envisages assistance to help private health care facilities take the first steps toward accreditation by assessing their clinical quality, governance, and patient safety through a recently piloted health care service quality assessment tool. Other possible joint efforts include promoting the availability of social health insurance in developing countries, Managed Equipment Services partnerships, and a broad range of policy work to ensure that public-private partnerships (PPPs) in health services help governments achieve their development goals in a responsible and sustainable manner. Finally, IFC concurs with recommendations 3 and 4. To the extent that IFC engages with pandemic preparedness and response, it will work with the World Bank to support the Bank Group in its capacity enhancement. IFC management will also collaborate with the World Bank on a review of the strategic alignment and selectivity of Bank Group engagement in GPPs. xxiv World Bank Group Support to Health Services | Management Response management action record Improve Quality and Distribution IEG FINDINGS AND CONCLUSIONS Quality. World Bank Group–financed project objectives show greater emphasis over time on improving the quality of health services, but limited capacity to monitor all the relevant aspects of the desired quality improvement. IFC quality improvement objectives and related indicators focus on the narrow aspect of structural quality. World Bank project financing is only rarely able to monitor all the relevant dimensions—structure, process, and outcomes—and the underlying links or theory of change. Some World Bank–financed projects, such as those adopting performance-based financing, present stronger monitoring and evaluation (M&E) frameworks for quality improvement. The World Bank has produced analytical works and is engaged in global initiatives aiming at improving the M&E of quality of health services (see the Primary Health Care Performance Initiative; Das, Hammer, and Leonard 2008; Smith and Nguyen 2013). This indicates the opportunity to improve the M&E framework of World Bank project financing and IFC investments seeking to improve quality of health services. Equity. Most World Bank projects identify the specific population groups with coverage gaps who are expected to benefit from interventions (often the poor). However, even when the beneficiaries are identified, the projects’ M&E rarely measure improvements in relative terms (that is, comparing beneficiaries with nonbeneficiaries), thus the distributional impacts (as well as the contribution to universal health coverage and shared prosperity) are seldom measured. IFC focuses on large markets and networks, which suggests the potential for systemic impact, but also in this case, the distributional impact of IFC projects is rarely specified in project interventions. IEG RECOMMENDATIONS Recommendation 1. Improve measurement of the quality of health services and the distributional effects of health services projects. The monitoring and evaluation framework of World Bank Group projects should include (i) appropriate indicators of the relevant dimensions of health service quality—structure, process, and outcomes, and (ii) the measurement of improvements of beneficiaries relative to nonbeneficiaries. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE As part of overall effort to improve M&E and ensure that the right mix of indicators is used, WB Management remains committed to strengthening measurement of quality and distribution of health services. M&E quality is also being closely monitored at the sector level by the Portfolio Monitoring Group. Specific indicators to measure the quality of services and impact on beneficiaries versus nonbeneficiaries depend on the nature of the operation and what the operation is trying to achieve. The WB has a large number of operations that are addressing health issues related to the quality of services and that disproportionately affect the poor, the disadvantaged, and the underserved. The feasibility of measuring several dimensions of health service quality and the distributional impacts varies in each context of project design and implementation capacity, and thus such measuring may not apply or be feasible for all projects. IFC management is pleased to share that it is working to incorporate new indicators of the relevant dimensions of health service quality into its new impact assessment and results tracking system (AIMM). These indicators are being piloted in a process that includes benefitting from interactions with IEG colleagues. They will measure the quality of IFC’s health care investments, including the processes for ensuring quality, such as the accreditation status of facilities, and the outcomes, such as infection rates. The new indicators will also include enhanced metrics on the structural Independent Evaluation Group | World Bank Group xxv determinants of the quality of health care. Appropriate indicators will be selected depending on the nature of the project being evaluated. In terms of measuring improvements that beneficiaries experience relative to nonbeneficiaries, IFC agrees on the importance of obtaining information on nonbeneficiaries for those projects where this information is relevant to the development impact that the project is claiming. However, IFC notes that implementing systematic and continuous monitoring of such information across all projects is not considered to be a realistic approach because of resource implications. At the same time, few IFC clients collect information that would be important for analysis of this kind to protect the confidentiality of their patients’ personal information. Therefore, IFC management is considering evaluations of selected projects, focusing on the specific outcome metrics targeted by the IFC project. xxvi World Bank Group Support to Health Services | Management Action Record Integrate Public and Private Sectors IEG FINDINGS AND CONCLUSIONS Public-private interactions. The evaluation identifies missed opportunities in integrating World Bank and IFC support to health services. The World Bank Group’s strategy to better assist governments with integrating public and private health sectors within their broader health care systems was articulated a decade ago and restated in 2015 However, to date, follow-through has not matched intent. The evaluation found, from the selected countries, missed opportunities in Brazil, and in Romania, where the public-private partnerships (PPPs) went ahead with little coordination with the World Bank. A recent IEG analysis found ample opportunities for better synergies between IFC and the World Bank in supporting health PPPs. The experience of Health in Africa shows that coordination between the World Bank’s upstream policy support and IFC support to small and medium health service providers remained weak until all HIA activities were transferred to the same management. The World Bank’s support to articulate private service provision and public financing is still limited, and IFC investee companies face challenges in integrating with public financing to improve access for the underserved. The main reasons are limited availability of public resources and capabilities; underdeveloped private markets for health services, including difficulties in making true price comparison between the public and private sectors; and inadequate regulation, including enforcement. The World Bank Group has taken steps to address this through the creation of a new unit to lead the implementation of the private health sector roadmap, but challenges remain because of the limited expertise and resources to support country teams and governments. The cascade approach to mobilize finance for development offers an opportunity to enhance public-private synergies. IEG RECOMMENDATIONS Recommendation 2. Strengthen World Bank and IFC synergy to support public-private interactions in client countries to contribute to Sustainable Development Goal 3 and universal health coverage. (i) For World Bank, work with IFC to strengthen the planning, regulatory, and accountability arrangements for public-private interactions, and (ii) for IFC, work with the World Bank to crowd-in public financing for privately delivered services. The World Bank Group’s newly launched Maximizing Finance for Development approach, aimed at mobilizing finance for development by focusing on upstream reforms where necessary to address market failures and other constraints to private sector investment, can be applied to achieve greater synergies between the public and private sectors. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE The World Bank has been exploring options to ensure a successful implementation of the MFD approach, in partnership with IFC and MIGA. The World Bank and IFC work together on issues related to strategy and strive to create and seize operational opportunities. In particular, a Global Lead for Harnessing the Private Sector in the HNP GP interacts regularly with IFC, including providing significant input into the new IFC health strategy and giving feedback on all new IFC projects in health. The former “Health in Africa” team, originally based in IFC and now fully integrated into HNP GP, continues to work on better public-private dialogue and reduction of red tape, creating new opportunities for private sector engagement in several African countries. IFC remains committed to working with the World Bank on initiatives to expand private provision. As recognized in the report, the later years of this evaluation period coincide with a series of positive Independent Evaluation Group | World Bank Group xxvii changes that led to closer coordination between the World Bank and IFC. These largely occurred after the Bank Group HNP Roadmap exercise in 2015 that developed a joint Bank Group approach to harnessing the private sector in health. The World Bank and IFC also have shared experiences in the health sector in Africa and other programs. Going forward, IFC will seek to capitalize on these coordinated engagements, which have been tested over the last few years. In particular, in the context of IFC’s 3.0 strategy, IFC will pursue synergies with the World Bank, including through the Cascade/MFD. Collaborative engagements and research are contemplated that focus on countries, with an emphasis on IDA countries, and in thematic areas such as how to promote better quality of care and how to further reach BOP beneficiaries. On quality, IFC envisages assistance to help private health care facilities take the first steps towards accreditation by assessing their clinical quality, governance, and patient safety through a recently piloted health care service quality assessment tool. Other possible joint efforts include promoting the availability of social health insurance in developing countries, Managed Equipment Services partnerships, and a broad range of policy work to ensure that public-private partnerships (PPPs) in health services help governments achieve their development goals in a responsible and sustainable manner. xxviii World Bank Group Support to Health Services | Management Action Record Mainstream Pandemic Preparedness IEG FINDINGS AND CONCLUSIONS Pandemics. The World Bank Group performance in pandemic preparedness and response has improved through successive pandemic outbreaks, but World Bank Group support to pandemic risk management, mitigation, and preparedness is not fully mainstreamed into operations. Under the Global Program on Avian Influenza Control and Human Pandemic Preparedness and Response, the World Bank supported pandemic preparedness and response efforts in 63 countries during 2006–13, but it failed to sustain efforts. As a central member of the global coalition that fought the Ebola virus outbreak in West Africa, the World Bank quickly mobilized the financial resources required to fight the spread of infection, restore basic health services, and reactivate the economy. A key lesson emerged: capable health systems are necessary to mount a successful response. They require adequately staffed health services, a supply of essential personal protective equipment, capacities for laboratory diagnosis, clinical management, and surveillance for quick diagnosis and rapid contact tracing. The Pandemic Emergency Financing Facility, approved in May 2016, may accelerate the release of funds to respond to future outbreaks. However, preparedness is the first line of defense. Country health systems’ pandemic risk management capacity should be strengthened. The commitment made under the 18th Replenishment of IDA to support about 25 IDA countries in developing pandemic preparedness plans and frameworks for health emergency preparedness, response, and recovery represents an opportunity to leverage World Bank experience in pandemic preparedness and response with International Bank for Reconstruction and Development financing. IEG RECOMMENDATIONS Recommendation 3. For sustainable capacity to address pandemics, systematically integrate preparedness plans and governance frameworks for pandemic control within the client country’s own health system in World Bank Group-financed projects and advisory services. Building on the commitment made under IDA18 to support health emergency preparedness, response, and recovery, the management of the World Bank Group institutions could seek to ensure that the World Bank’s project financing and Advisory Services and Analytics are not one-off responses outside the client country’s health system. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Strengthening country resilience by consolidating public health capacities as integral elements of strong health systems is part of work toward UCH and is at the core of the World Bank’s engagement in resource-constrained low-income countries. With a number of health sector projects in IDA countries in advanced stages of preparation, World Bank is well positioned to meet the country preparedness targets set under IDA18. Following the 2014 Ebola outbreak and recognizing the need to strengthen institutional capacities for disease surveillance and control, African Heads of State have established the Africa CDC as a regional network to lead, integrate, and strengthen the continent’s public health institutions, capacities, functions, and partnerships to detect and respond to disease threats and outbreaks. The World Bank is working to mobilize substantive support for the Africa CDC, alongside partners such as the US Centers for Disease Control and Prevention, the World Health Organization, and the African Development Bank. The PEF has been put in place and is expected to support countries with financial resources for early surge responses. In addition, the World Bank has included disease outbreaks as eligible catastrophes for a CAT-DDO to both IBRD and IDA countries. Finally, the World Bank is also leveraging global part- Independent Evaluation Group | World Bank Group xxix ners to consolidate financial and technical assistance for pandemic preparedness to IDA countries as a central element of UHC. To the extent that IFC engages with pandemic preparedness and response, it will work with the World Bank to support the Bank Group in its capacity enhancement. xxx World Bank Group Support to Health Services | Management Action Record Enhance Partnerships IEG FINDINGS AND CONCLUSIONS Partnerships. World Bank Group participation in global partnership programs (GPPs) has contributed to aligning partner objectives toward health Millennium Development Goals (MDGs) and Sustainable Development Goal (SDGs) and leveraging resources, but there are opportunities for improvement. The World Bank Group has often participated in GPPs at the request of other partners who value its convening capacity to align partners with shared objectives (usually MDGs and SDGs), its strong ability to manage and execute trust funds, and country presence. Although engagements in GPPs have become more selective and aligned with sector and corporate strategies over the evaluation period, the additionality of some partnerships remains weak and some mandates overlap. The absence of a strategy that defines World Bank Group priorities in its global-level engagement with health GPPs does not allow the assessment of the value-added of each partnership engagement and the worth the World Bank Group brings to them. A strategic review could define clear selectivity criteria for current and future engagements, delineate the division of labor among partners, clarify expectations, and ensure adequate resourcing for representation and participation in the GPPs’ governance. IEG RECOMMENDATIONS Recommendation 4. Enhance the strategic alignment and selectivity of World Bank Group engagement in ongoing and future GPPs. A strategic review should apply clear selectivity criteria that reflect the World Bank Group’s comparative advantage and the broader global development agenda. It can inform the selectivity and relevance of ongoing and future GPPs and a more effective use of resources needed for engaging in partnerships. Agree. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE The World Bank has increasingly consolidated its participation in existing global partnership programs (GPPs). Decisions to create or join new ones are based, first, on the World Bank’s comparative advantages and priorities as a development finance institution, and then on such factors as whether the GPP has a robust strategic engagement framework and there is sufficient policy development and implementation capacity at the country level. Under a resource-constrained environment, the World Bank has put more resources in fewer and larger GPPs that build on the World Bank’s comparative advantages—for example, the GFF, PEF, and the Power of Nutrition GPPs—and fewer resources on older GGPs, some of which have overlapping mandates. Withdrawal from such older GPPs has been done in a consistent but gradual manner to minimize disruption and sustain support to other partners, when such support is requested and adds value. Management agrees that a targeted strategic review would be helpful to map all existing GPPs and explicitly identify the type of engagement the World Bank should have in each of them as well as guiding principles for current and future engagements to ensure optimal allocation of resources. Recognizing the importance of partnerships to find sustainable solutions to health challenges and the complex landscape of external actors, Management will apply such guiding principles for engagement with flexibility and judgment. IFC management will collaborate with the World Bank on a review of the strategic alignment and selectivity of Bank Group engagement in GPPs. Independent Evaluation Group | World Bank Group xxxi 1 highlights 1 A large share of the world’s population has access to health services, and child mortality has significantly declined. However, at least 400 million people, most of them living in developing countries, do not receive all the Health essential health services. Lack of resources, Services and poor articulation of the roles of the public and Development private sectors, weak governance, health market failures, and fragility, conflict, and violence situations contribute to coverage gaps. 2 The World Bank Group has joined the global coalition for universal health coverage, which aims to ensure that all people receive quality health services when needed without suffering financial hardship. 3 World Bank Group strategies related to health services emphasize the needs of the poor, the impoverishing consequences of catastrophic health spending, health system performance, and financing. Over time, strategies have put more emphasis on results and called for a joint World Bank and IFC approach to harness the private sector to achieve universal health coverage and the World Bank Group’s twin goals. 4 The World Bank Group supports health services through supply-side interventions, demand-side interventions, and health system–strengthening activities, which aim to improve access to, quality of, and equity in the use of health services. 1 H E A LT H S E R V I C E S are an important component of any health system and are arguably its most visible part. These include services dealing with the diagnosis and treatment of disease or the promotion, maintenance, and restoration of health. They include personal and nonpersonal health services. Providing health services is the most visible function of any health system to both users and the public. Service provision refers to the way in which inputs such as money, staff, equipment, and drugs are combined to allow the delivery of health interventions. Improving access, coverage, and quality of services depends on these inputs being available, on the ways services are organized and managed, and on incentives influencing providers and users.1 It is recognized that it takes more than health services to improve health, nutrition, and population outcomes. A range of personal, social, economic, and environmental factors are known to influence health. This evaluation focuses primarily on health services rather than on broader health, nutrition, and population outcomes. Health Achievements and Challenges Today, a large share of the world’s population has access to health services. For example, the vaccination coverage of diphtheria-tetanus-pertussis has reached 84 percent of one-year-olds. On the reproductive and maternal health front, 73 percent of live births take place in the presence of a skilled birth attendant. Because of increased access to health care, there has been a significant reduction in the child mortality rate of children under 5 from 62 per 1,000 live births in 2005 to 40 per 1,000 live births in 2015 worldwide. Reduction was more pronounced in Sub-Saharan Africa International Development Association (IDA) countries, where child mortality rates (per 1,000 live births) dropped from 127 to 82. Despite these achievements, as of 2013, at least 400 million people, most of them living in developing countries, were not receiving the essential health services needed to achieve the Millennium Development Goal (MDG) targets (WHO and World Bank 2015). Universal health coverage is a target of the Sustainable Development Goals (SDGs). However, currently most health systems in developing countries are dramatically undersupplied in the areas of financial resources allocated to health, availability of infrastructure, and health workforce. Total health expenditure (by government and households) represents only 6 percent of gross domestic product among low- and middle-income countries compared with 12 percent in Organisation for Economic Co-operation and Development (OECD) countries.2 Low-income countries have 2.5 physicians per 10,000 people compared with 28.7 among high-income countries (WHO 2015a). Furthermore, according to World Bank Group and World Health Organization (WHO) research, 6 percent of people in low- and middle-income countries are tipped into or pushed further into extreme poverty because of health spending. Thus, the global community has enshrined in the universal health coverage goal the aspiration that all people should 2 World Bank Group Support to Health Services | Chapter 1 receive the quality health services they need and that the use of such health services should not expose them to financial hardship. Challenges to health services are exacerbated in poor countries and those affected by fragility, conflict, and violence (FCV). The destruction of the health infrastructure and the displacement of people can produce outbreaks of communicable diseases (for example, polio and cholera), heighten the impact of noncommunicable diseases, and increase the burden for those injured in conflict. For example, “more people are estimated to have died from the breakdown of the health system than directly from the fighting in Syria” (World Bank 2017b). Supporting key health services in FCV situations is a humanitarian priority but also essential to preventing global pandemics and key to supporting recovery, peace, stabilization, and long-term development (Haar and Rubenstein 2012). People living in FCV situations are also exposed to higher risk of sexual- and gender-based violence (SGBV; UNHCR 2011) and poverty. Appropriately, the commitment to poverty eradication under the 18th Replenishment of IDA (IDA18) implies that the share of support to health services delivered in FCV situations will increase. Different but interrelated trends, including demographic and epidemiological changes, are reshaping health needs and priorities. From about 7 billion today, the global population is projected to reach 9.3 billion by 2050 and 10.1 billion by 2100.3 Demographic trends show populations are aging in almost all low- and middle-income countries. Decreasing fertility trends in Sub-Saharan Africa follow a similar trajectory occurring elsewhere, but separated by several decades. In Africa, 76 percent of deaths are still attributable to communicable, maternal, neonatal, or nutritional causes, compared with 25 percent of total deaths in developed countries. Noncommunicable diseases, such as cardiovascular disease, cancers, diabetes, and respiratory diseases, make up a large proportion of deaths in low- and middle-income countries, but attention to this focus area is limited. The risk factors for noncommunicable diseases are associated with urbanization and altered lifestyles, especially smoking, physical inactivity, air pollution, unhealthy diet, and excessive alcohol use. The past decade also saw the resurgence of global pandemic threats. The H1N1 flu pandemic that emerged in Mexico and the United States in 2009 and the West Africa Ebola virus outbreak in 2014 demonstrated the global nature of pandemic threats.4 Surveillance and laboratory capacity, maintained by strong national and regional public health institutes, are essential components of functioning health systems and provide the foundation for resilient public health systems. In addition, most of the recent pathogens with pandemic potential are of zoonotic origin, which requires action and coordination in both veterinary and human public health areas. All health systems are mixed, with variation in the roles of public and private sectors across countries and health services. Almost all systems define a set of health services that are publicly funded.5 Economic theory indicates three distinct rationales for the public financing of health services: (i) to ensure the optimal provision of public goods or services with large externalities (efficiency); (ii) to subsidize consumers too poor to buy health insurance or health services out of pocket (equity); and (iii) to correct failures in the health insurance market (efficiency and equity; Musgrove 1996).6 A desirable mix of public and private service provision in a particular country context will depend Independent Evaluation Group | World Bank Group 3 on the availability of providers, the technical capacity of government to regulate and purchase health services, how responsive individuals’ decisions are to public actions, and political economy considerations (Filmer, Hammer, and Pritchett 2002; McPake and Hanson 2016; Pita Barros and Siciliani 2011). In addition, considerable room exists for the private sector (including the nonprofit sector) to offer additional health services beyond those publicly funded. This plurality of systems makes it difficult for countries to understand the roles each actor should play. Innovation in health services and cutting-edge health technologies are helping to shape delivery methods and approaches to improving people’s health and well-being. The health care industry has experienced a proliferation of new services, ways of working, and technologies that have enhanced life expectancy, quality of life, and diagnostic and treatment options as well as the efficiency and cost-effectiveness of the health care system. Innovation can expand opportunities and provide value to all stakeholders (for example, the use of telemedicine solutions to cover hard to reach populations). Opportunities arise due to better understanding of human behaviors and how individuals and groups respond to communication, incentives, and information (for example, the expanding use of incentives to improve both the delivery of and demand for health services; Flanagan and Tanner 2016). The rise in the number and scope of actors has made the global health landscape more complex and intertwined. Greater funding has characterized this evolution. Development assistance for health (DAH) has increased from $6 billion in 1990 to more than $37 billion per year in 2015.7 Global partnership programs (GPPs), such as the Global Fund to Fight AIDS, Tuberculosis and Malaria, have scaled up programs for human immunodeficiency virus (HIV) and acquired immune deficiency syndrome (AIDS), malaria, and tuberculosis. Gavi, the Vaccine Alliance has expanded access to vaccines. Although most of the increase in funding comes from governments that are members of the OECD Development Assistance Committee, nonmembers, such as the Bill and Melinda Gates Foundation, account for about one-quarter of all development aid for health. World Bank Group Strategies World Bank health strategies have put greater emphasis on results and called for a more focused approach on the World Bank Group’s comparative advantage. The 1997 Health, Nutrition, and Population (HNP) sector strategy steered the World Bank toward improving outcomes for the poor, protecting the population from the impoverishing effects of illness, enhancing the performance of health systems, and securing sustainable health financing (see figure 1.1). In 1998, the International Finance Corporation (IFC) created the Health Care Best Practice Group and adopted its frontier country strategy to steer resources toward high-risk, low-income countries. The current World Bank Group health sector strategy, approved in 2007, embraces previous objectives, but also calls for concentrating contributions on the World Bank Group’s comparative advantage (health system strengthening, financing, and economics), selectivity in engagement with global partners, and support for country readiness to prevent and address the rapid onset of pandemics (Fair 2008; World Bank 2007b). 4 World Bank Group Support to Health Services | Chapter 1 FIGURE 1.1 Evolution of World Bank Engagement in Health | 2017–15: Universal Health Coverage Focus 2013: World Bank Group Corporate Strategy 2007: New Strategy for Health, Nutrition, and Population Results • Universal Health • Twin goals of ending Coverage – use of extreme poverty and quality health care 2000: Millennium increasing equity and Development Goals • Close of the 1997 services, when shared prosperity needed, for all, strategy’s objective and • Work as a “One World 1997: Health, Nutrition, means with increased including financial Bank Group”: scaling up protection – included and Population focus on governance and collaboration among the Strategy Endorsement of health- results among the targets for World Bank, IFC, and Sustainable 1998: IFC Health Care related Millennium • World Bank aims to MIGA, to identify the Development Goals: reposition itself with focus Development Goal Best Practice Group best solutions, • Increased financing for on comparative number 3 regardless of whether • Health, Nutrition, and Areas of emphasis: single disease/single advantages to improve they are public or intervention programs, health outcomes Population Global • Improving outcomes for private Practice: new the poor often within weak health • Joint World Bank and IFC systems organizational model • Financial protection in health approach to harnessing and priority directions • Enhancing health systems • Increased development the private sector in health assistance for health updated performance • Sustainable health financing • New health partnerships: Gavi and Global Find Note: Gavi = Gavi, the Vaccine Alliance; Global Fund = Global Fund to Fight AIDS, Tuberculosis and Malaria; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency. MDGs put health at the center of the development community’s agenda. Three of the eight MDGs targeted health outcomes directly (that is, reduce child mortality, improve maternal health, and combat HIV/AIDS, malaria, and other diseases), and health greatly influences results for other MDGs (for example, eradicate extreme poverty and hunger; promote gender equality and empower women; and develop a global partnership for development). The increased global attention to health brought an increase in DAH and the establishment of global partnerships such as Gavi and the Global Fund. In this period, links between veterinary and human public health were strengthened. In 2008, the World Bank, together with WHO, the World Organization for Animal Health, the Food and Agriculture Organization, and the United Nations (UN) System Coordinator for Avian and Human Influenza, prepared the global strategy “One Health” to fortify country systems in veterinary and human public health areas and the bridges between them. The World Bank Group’s more recent strategies emphasize synergies among World Bank Group institutions and partners to achieve the twin goals of ending extreme poverty and boosting shared prosperity. The World Bank Group’s 2013 corporate strategy emphasizes the “One World Bank Group” approach to find the best development solutions, regardless of whether they are public or private (World Bank 2013). The joint World Bank and IFC roadmap to health recognized that universal health coverage cannot be achieved without the private sector, and that the World Bank Group is uniquely positioned to help client countries harness the private sector in achieving universal health coverage and the twin goals (World Bank 2015). The roadmap established changes at the strategic level with the aim to achieve universal health coverage, such as moving away from input-based Independent Evaluation Group | World Bank Group 5 operations to support broader policy reforms and system changes (so that governments can become better stewards of the health systems), and enhance coverage and strengthen service delivery to ensure equitable access to quality. The subsequent creation of the HNP Global Practice (GP) in 2014 aimed at better connecting global and local expertise within the institution to assist client countries accelerating progress toward universal health coverage (World Bank 2016b). Through its priority programs and initiatives (see box 1.1), it was notably instrumental in stewarding the global community away from a focus on vertical diseases priorities (at the core of the MDGs) toward an integrated universal health coverage goal for the SDGs. IDA, IFC, and the Multilateral Investment Guarantee Agency (MIGA) are expected to step up their collaboration further to leverage resources of the private sector and development partners to maximize financing for development. As part of the IDA18 replenishment, the Private Sector Window was created to mobilize private sector investment in IDA-only countries, with a focus on poverty eradication and fragile and conflict-affected situations through IFC- and MIGA-led transactions. (World Bank 2017a, 2017d). Moreover, in March 2017 the Forward Look: Progress and Challenges Report introduced the cascade approach as a concept to guide the World Bank Group’s efforts and comparative advantages to mobilize finance for development (World Bank 2017e). Intervention Logic and Evaluation Methodology This evaluation covers World Bank Group support to health services from fiscal year (FY)05 to FY16. The evaluation portfolio is presented in table 1.1.8 It is worth noting that most World Bank support to health services comprising the evaluation portfolio and the related findings refers to a period prior to the introduction of the new HNP GP operational model (that is, 87 percent of World Bank–financed projects and 78 percent of Advisory Services and Analytics (ASA) projects supporting health services were approved before FY15). The evaluation assesses the World Bank Group’s contribution to health services both in countries and at the global level as follows:    The World Bank Group’s in-country support for health services includes World Bank project financing and ASA, IFC Investment Services (IFC IS), and IFC Advisory Services (IFC AS). In the six countries for which case studies were conducted, the evaluation assesses the entire mix of support to health services provided over the evaluation period to identify synergies and complementarities among instruments and institutions delivering the support, and between the World Bank Group and development partners that also support in-country health services.    The evaluation considers, at the global level, World Bank Group participation in health GPPs9 to bring together partners’ efforts toward common health objectives, leverage resources, and share knowledge and solutions among partners. This evaluation is performed under the strategic engagement area of the Independent Evaluation Group (IEG) that covers service delivery for the poor, which has been aligned with the World Bank Group’s “Forward Look” Investing in People pillar. The evaluation also uses a service delivery lens 6 World Bank Group Support to Health Services | Chapter 1 Box 1.1 |  Health, Nutrition and Population Global Practice—Updated Priority Directions and Organizational Model The Health, Nutrition, and Population (HNP) Global Practice (GP), created in July 2014 as one the 14 GPs created under the World Bank’s new operating model, adopted the mission of assisting countries in accelerating progress toward universal health coverage through financial protection, service coverage, and healthy societies. HNP GP is organized in Regional teams led by practice managers under the overall leadership of two directors and one senior director, as well as global solutions leads for financing, service delivery, population and development, nutrition, health societies/public health, decision and delivery science, and private sector engagement. The following make up the HNP GP priority programs and initiatives: T n  he Global Financing Facility in Support of Every Woman Every Child is a global coalition launched in July 2015 to seize the opportunity to change the course of financing for the Sustainable Development Goals and improve the lives of millions of women, children, and adolescents across the world. T n  he Power of Nutrition is a global partnership launched in 2015 that set up a fund to leverage investments from the private sector to complement International Development Association (IDA) resources to fight malnutrition and stunting. P n  andemics preparedness and health systems strengthening include pandemic preparedness in at least 25 countries under the 18th Replenishment of IDA scaled-up commitment; the West Africa Regional Disease Surveillance Systems Enhancement program; the East Africa Public Health Laboratory Networking Project; investment to contain antimicrobial resistance; and the Pandemic Emergency Financing Facility, an innovative fast-disbursing financing mechanism launched in May 2016 to provide a surge of funds to enable a rapid and effective response to a large-scale disease outbreak. T n  he Global Tobacco Control Program was launched in July 2015 to assist countries in designing tobacco tax reforms—a win-win policy measure to achieve public health goals by increasing prices, reducing smoking, and preventing initiation among youth, and to raise more domestic resources for investments that benefit the entire population. Source: World Bank 2016c and http://www.worldbank.org/en/topic/health. Independent Evaluation Group | World Bank Group 7 TABLE  1.1 | World Bank Support to Health Services, FY05–16 Ap- Commit- Commitments Open Closed or Evaluated Type of proved ments ($, mil- (as a % of the Projects Operationally Ma- Projectsa Support Projects lions) portfolio) (no.) ture Projects (no.) (no.) (no.) GPPs and 31 n.a. n.a. n.a. n.a. n.a. MDTFs World 619 22,756 5 204 415 259 Bank project financing World 1,033 262.9 7 n.a. n.a. n.a. Bank ASAs IFC 124 2,672 3 56 68 28 Investment Services IFC 67 71.4 2 37 30 14 Advisory Services Note: ASA = Advisory Services and Analytics; GPP = global partnership program; IFC = International Finance Corporation; MDTF = multi- donor trust fund; n.a. = not applicable. a. Evaluated through self-evaluation and Independent Evaluation Group validation. as well as a behavior change lens, complementing, in this regard, the recent evaluations on urban transport, and water supply and sanitation (see appendix D). The overarching question of the evaluation is, What are the roles and contributions of the World Bank Group in support of health services, and what can be done to enhance them? The evaluation captures the World Bank Group’s support to health services through four specific questions answered by assessing contributions to health services at the project, country, and global levels:    What has the nature, extent, and evolution of World Bank Group support to health services been in the past 10 years?    How relevant has World Bank Group support to health services been to the main health needs and priorities?   To  what extent has World Bank Group support effectively contributed to the achievement of its health goals?   What has the role of the Wrld Bank Group been in country programs and GPPs? The intervention logic spells out World Bank Group’s expected contribution to in-country health services coverage, universal health coverage, and ultimately, to better health outcomes (see figure 1.2). The evaluation’s intervention logic is based on the World Bank Group’s 2007 HNP strategy and its 2016 update (see World Bank 2016b, 4). It is also consistent with the health determinants 8 World Bank Group Support to Health Services | Chapter 1 FIGURE 1.2 | The Evaluation Intervention Logic IF THE WORLD BANK GROUP WHICH WILL ULTIMATELY AND EFFECTIVELY DELIVERS ITS IT WILL CONTRIBUTE TO IMPROVING COUNTRIES’ FORMULATES THE RIGHT IMPROVE INTERVENTIONS, HEALTH SYSTEMS, STRATEGIC DIRECTIONS, HEALTH OUTCOMES. DIRECTLY By funding investments in health services and sector strengthening Improved access to health through services By strategically answering • World Bank lending Equitable access country needs and priorities • IFC investments to affordable, By providing technical assistance and quality health Enhanced quality of health policy advice through services services • World Bank ASA • IFC Advisory Service AND by generating • Internal synergies Enhanced equity in use of – across World Bank instruments health services Countries Improved (lending, ASA) accelerate health – across World Bank Group progress outcomes institutions (World Bank, IFC) toward for people • External synergies universal in client By leveraging its comparative – with development partners Strengthened stewardship countries health advantages: (oversight of private and coverage • Capacity for large-scale public actors) implementation • Multiple financing instruments Strengthened domestic National • Country focus and presence INDIRECTLY health financing systems health systems • Capacity to engage private By being a convener of global strengthened health actors partnerships that • Experience in health sector • Fund R&D in disease-specific areas Enhanced information reforms • Generate knowledge as a Global and resource Public Good management • Provide a platform for global advocacy • Strengthen international cooperation Enhanced health service • Harmonize donors’ practices provision Independent Evaluation Group | World Bank Group Note: ASA = Advisory Services and Analytics; IFC = International Finance Corporation; R&D = research and development 9 literatures (see Murray and Frenk 2000; Kutzin 2008; Evans, Hsu, and Boerma 2013). The intervention logic distinguishes between universal health coverage objectives (improve access to, quality of, and equity in the use of health services) and health system strengthening, which comprises the means (that is, the policy instruments) to achieve universal health coverage (see Kutzin and Sparkes 2016). Coordinated policy and implementation across the health system is considered essential to achieve the desired health coverage objectives and long-term health outcomes. The evaluation’ s intervention logic recognizes the existence of feedback loops, sequencing, and timing among its objectives of improving access, quality, equity, and health systems. Equitable use of affordable and quality health services is determined by supply-side factors (such as service availability) and demand-side factors (such as behavior change interventions). On the supply side, accessibility has three dimensions: physical, financial affordability, and acceptability. Access to quality services depends on the level of resources available and how these resources are organized and managed. However, good quality and affordable health services may be accessible, but for a variety of reasons, people may not use available services. In these cases, factors affecting the demand or the behavior of users (for example, incentives, information campaigns, or addressing social or cultural norms) may play a role in improving use. Solving the effective coverage problem requires tackling both demand- and supply-side issues (Shengelia et al. 2005). For example, quality improvement can generate additional demand and expand use and coverage, through a feedback loop. The evaluation adopts the IEG evaluative framework for service delivery and behavior change, which, in turn, builds on the 2004 World Development Report Making Services Work for Poor People (World Bank 2003). The framework identifies accountability across and between citizens, government, and providers as critical condition for services to benefit the poor. Therefore, strengthening poor people’s voice can make health services work better for the poor. However, incentives and monitoring are needed to ensure that providers serve the poor. This includes better linking of spending and outcomes and making sure beneficiaries are involved in the planning and monitoring of health services (see appendix D). The evaluation’s intervention logic treats universal health coverage as a downstream goal. This goal is about ensuring that all people have access to the quality health services they need without suffering financial hardship.10 Therefore, it is aligned with the World Bank Group’s shared prosperity goal, which requires countries to expand service delivery for the poor and/or the bottom 40 percent (World Bank 2013, 18). The extent to which the World Bank Group can demonstrate achievement of the universal health coverage goal will depend on the robustness of projects’ monitoring and evaluation (M&E) frameworks and intervention logic. IFC supports private actors in the provision of health services. The private sector can provide quality and affordable health services, indicating a role for commercial providers in expanding health coverage (Prahalad 2006; Mackintosh et al. 2016). Through demonstration effects to other health service providers, IFC can help improve services beyond those directly supported (Rosenthal and Newbrander 1996). Generally, however, private providers can provide only a limited set of services 10 World Bank Group Support to Health Services | Chapter 1 and cannot offer comprehensive universal care (particularly preventive and promotive care) even at a primary care level (Morgan, Ensor, and Waters 2016). Therefore, a more effective channel to expand coverage among the poor, who have limited capacity to pay for health services, is through the integration of private provision with public financing (Hammer, Aiyar, and Samji 2007). In any case, the government’s role as the steward and regulator of the health system is key to ensuring public resources are used for the public’s benefit and protecting against predatory behavior by private providers (McPake and Hanson 2016). IFC AS can also address health systems’ capacity constraints related to private sector engagement, primarily in advising governments to structure public-private partnerships (PPPs). Finally, IFC projects can also have objectives beyond the health sphere, such as job creation, private sector development, and innovation. This evaluation’s intervention logic considers the role of World Bank Group participation in GPPs. The World Bank Group participates in these programs selectively with an expectation to mobilize collective action toward common objectives, to leverage resources, as well as to share knowledge among partners. Therefore, GPPs complement the World Bank Group’s country-based model. The evaluation assesses the overall relevance of these GPPs to the World Bank Group’s strategic priorities, as well as the role played by the World Bank Group in GPPs. In addition, the evaluation assesses two GPPs: the Health in Africa Initiative (HIA) and the Health Results Innovation Trust Fund (HRITF). This evaluation uses different data and methods to triangulate evidence to answer the evaluation questions (see appendix A for more details):   Portfolio  analyses of World Bank Group support to health services (see table 1.1), including three types of analysis (see appendix B): – Portfolio trends, association with Global Burden of Disease, and DAH data Project design features comprising all World Bank Group health services–supported projects –  approved during the FY05–16 period Project outcomes, achievement of objectives, drivers of success and failure, and statistical –  modeling of associated factors for approved and evaluated projects (self-evaluation and IEG validation) in the FY05–16 period.   Development  of the evaluation’s intervention logic (to spell out the expected contribution of the World Bank Group) and the construction of specific intervention-centric theories of change    Interventioncase studies of three delivery mechanisms (conditional cash transfers [CCTs], performance-based financing [PBF],11 and the public-private interactions) and of the World Bank’s response to pandemic outbreaks (see appendix E)    Case study analyses of six selected countries (Bangladesh, Brazil, Liberia, Romania, the Philippines, and the Republic of Yemen) to assess synergies and complementarities among World Bank–financed projects, between the World Bank and IFC, and among the World Bank Group and development partners (see appendix F). The six countries were purposely selected among those that received both World Bank and IFC support, based on the following principles: (i) coverage of Independent Evaluation Group | World Bank Group 11 income level and fragility, (ii) coverage of in-depth analysis of interventions, (iii) regional balance, and (iv) balance between high and low capacity countries to manage development assistance.    Analysisof health GPPs, which comprised all GPPs with which the World Bank Group is currently engaging, and two case studies (HIA and HRITF; see appendix C)    Social network analyses (SNAs) of the health sector in Liberia and the webometrics information of key providers of DAH (see appendix G). This evaluation builds on the 2009 evaluation of World Bank Group support to HNP. The previous evaluation noted that one-third of HNP public sector projects had less than satisfactory outcomes. It argued that the HNP portfolio did not have an adequate focus on the poor and priority issues for the poor, such as family planning and nutrition. In addition, the evaluation concluded that the World Bank Group did not focus sufficiently on improving the efficiency of health systems and did not systematically coordinate internally between the World Bank and IFC to leverage its sector dialogue on health regulatory framework and private sector participation in health. The evaluation found weak M&E in World Bank Group projects, and few, if any, impact evaluations. IFC (then a relatively new player in the health sector) grew its portfolio but remained concentrated largely on hospitals, which showed limited social impact and limited focus on innovative solutions to improve the health of the poor (for example, low-cost drugs and other health technologies, health PPPs, and health insurance; World Bank 2009). This evaluation also complements IEG’s 2014 health financing evaluation. That evaluation examined revenue collection, pooling of health funds and risks, and health financing reforms supported by the World Bank Group in 68 countries during FY03–12 (World Bank 2014a). The evaluation concluded that World Bank support was more successful when the teams drew on a variety of skills across sectors and where government commitment to reforms was strong. Second, World Bank support has helped raise or protect public revenues for health. Equity in pooling increased where the World Bank assisted governments in subsidizing compulsory contributions to various health insurance for low-income groups. However, increased pooling did not always lead to pro-poor spending, improved equity in service use, or greater financial protection. Support to reduce user payments was limited. Third, the World Bank has increased its focus on activity- or results-based payments supported by results-based financing (RBF) projects, but with little attention to the impact on costs and broader effects on the public sector. Fourth, the evaluation noted that an integrated approach that links health financing with public sector reforms was likely to be more effective than single-issue interventions because it builds the institutions needed for sustainability. This includes equitable revenue instruments, taking account of the overall public finance situation, moving toward compulsory pooling in insurance and national health systems, focusing on strategic purchasing, and giving attention to adverse effects in a broader public sector context. However, the evaluation noted that linking health financing to public finance requires strong collaboration across the World Bank Group to facilitate the dialogue at all government levels. 12 World Bank Group Support to Health Services | Chapter 1 Report Structure In this report, chapter 2 assesses World Bank Group support to health services in countries. It answers the following questions: What is the nature and extent of project-level support to health services over the past 10 years? How does support relate to country needs and priorities? And, to what extent and in what ways has health services support achieved its goals? The chapter presents the evolution of World Bank–financed projects and ASA, IFC IS, and IFC AS over the past decade. The chapter also evaluates the effectiveness of World Bank project financing, IFC IS, and IFC AS in achieving the World Bank Group’s health services objectives (for example, improving access to and quality of health services, strengthening health system functions, improving equity, and improving health outcomes). It also assesses the relevance of World Bank support to country health needs. To provide further evidence on the ways that health services support achieved its goals, the chapter continues with a review of selected delivery mechanisms; the World Bank support for pandemic preparedness and control; and the analysis of synergies and complementarities between World Bank–financed projects and ASAs, between the World Bank and IFC, and among the World Bank Group and external development partners. The chapter triangulates evidence derived from the portfolio analysis, the country case studies, and the intervention case studies of delivery mechanisms (CCT, PBF, and public-private interactions) and of the World Bank’s response to pandemic outbreaks. Chapter 3 assesses World Bank Group support to health services through GPPs. It answers the following questions: What is the nature and extent of World Bank Group support to health services at the global level through participation in GPPs over the past 10 years? And, what has been the role of the World Bank Group in health GPPs? The chapter uses SNA of webometrics to depict how organizations operating in the global health landscape interact. It presents the evolution of GPPs the World Bank Group has been engaging with and their alignment with World Bank Group’s sector and corporate strategies. The chapter also assesses the role the World Bank Group has played in these GPPs. Finally, it examines in more detail two GPPs (HIA and the HRITF) adapting the IEG’s evaluation framework for assessing global and regional partnership programs (World Bank 2007a, 2007d). The HRITF is selected as the largest multi-donor trust fund (MDTF) program in health housed in the World Bank, and HIA is the first IFC-led comprehensive initiative in the health sector to enable private sector participation in African countries. Recent GPPs and initiatives such as the Global Financing Facility (GFF), Power of Nutrition, Pandemics Emergency Financing Facility, and the Tobacco Control Program are mentioned (see box 1.1). However, because these programs are very recent, their coverage in the evaluation is limited. Chapter 4 presents the conclusions and recommendations of the evaluation. It summarizes the evaluative evidence and offers recommendations to enhance the World Bank Group’s contribution to health services. 1 See http://www.who.int/topics/health_services/en. Independent Evaluation Group | World Bank Group 13 2  ata are from the World Development Indicator Database, World Bank, Washington, DC (accessed August 15, 2017), D http://databank.worldbank.org. 3  nited Nations, Department of Economic and Social Affairs. World population prospects, the 2010 revision [cited 2013 U May 16]. http://esa.un.org/unpd/wpp/index.htm. 4  he global framework for health security is embodied in the International Health Regulations, which were revised in T 2005 and adopted by the World Health Assembly. 5  ublic funds include both mandatory income-related contributions to social security funds, as well as tax-based P contributions. 6 T  he division between what is publicly and privately financed depends upon the capacity to raise public funds and priorities. However, all health systems (even the United States’ health system, which is primarily based on private health insurance) have a core of health services that are publicly funded (Mackintosh et al. 2016). 7 T  he source for this information is the University of Washington Institute for Health Metrics and Evaluation website at http://vizhub.healthdata.org/fgh/. 8  he evaluation reviewed the portfolio of the Multilateral Investment Guarantee Agency, which provides political risk T insurance (guarantees) for projects in a broad range of sectors in developing member countries all over the world. The evaluation identified only one guarantee that covered health services in Turkey. 9  lobal partnership programs are programmatic partnerships in which the partners (i) dedicate resources toward G achieving agreed objectives over time; (ii) conduct activities that are global, regional, or multicountry in scope; and (iii) establish a new organization with shared governance and management unit to deliver these activities (World Bank 2016c). 10 The universal health coverage goal recalls the concept of equity “that individuals should have equal opportunities […]  and be spared from extreme deprivation in outcomes” introduced in the World Bank’s World Development Report 2006: Equity and Development (World Bank 2005, p 2). 11 Performance-based financing is a specific type of results-based financing (see Musgrove 2010). 14 World Bank Group Support to Health Services | Chapter 1 2 highlights 1 World Bank project financing in support of health services is increasingly addressing health priorities, such as maternal and child conditions, that disproportionately affect the poor, and World Bank adopting results-based financing approaches. Group Support International Finance Corporation (IFC) in Countries investments mostly concentrate on lower- and upper-middle-income countries and focus more on noncommunicable diseases and general health. 2 The performance of World Bank–financed closed and evaluated health projects has improved significantly over the evaluation period, both in terms of outcome rating and quality of the monitoring and evaluation framework. However, the performance of the health portfolio remains below the World Bank Group overall average, though by a small margin. IFC Investment and Advisory Services in health perform better than their overall IFC comparators. 3 World Bank health projects that have adopted financial incentives such as conditional cash transfers and performance-based financing interventions have shown better results than the overall health portfolio. 15 4 World Bank Group projects only seldom monitor and evaluate all relevant dimensions of health services quality: the structure, the process, and the outcomes. 5 The project’s distributional impacts are seldom monitored and evaluated when a World Bank or IFC project identifies specific disadvantaged population groups as the beneficiary. 6 The World Bank Group shows limited success in integrating private provision of health services with public financing, which requires joint World Bank and IFC efforts. 7 The World Bank’s experience in supporting pandemic prevention and control in client countries highlights the need to sustain efforts and integrate them with national health systems to stop outbreaks from spiraling out of control. 16 World Bank Group Support to Health Services | Chapter 2 Portfolio Characteristics D U R I N G F Y0 5 –16 , the World Bank Group committed $22.8 billion to support health services through World Bank project financing and $2.7 billion in IFC investments to client countries. The World Bank also delivered ASAs for a total value of $262.9 million, and IFC provided advisory services for a total value of $71.4 million. The World Bank Group’s health services portfolio represents about 4 percent of the overall portfolio in amounts approved during FY05–16. It shows a slightly upward trend over time and spikes in 2010, which followed the global financial crisis, and in 2015 (see figure 2.1). World Bank Project Financing and ASAs IDA supported about half of all World Bank health services projects comprising the evaluation portfolio (52 and 54 percent by commitment volume and number of projects, respectively). The share of IDA projects has increased over time (43 percent of commitment in closed projects versus 60 percent in open projects). Conversely, the share of International Bank for Reconstruction and Development (IBRD)-financed projects has decreased. Trust funds provided around 5 percent of World Bank commitment to health services, but supported around 29 percent of all health services projects (see table 2.1). Investment project financing represents 88 percent of all World Bank commitments to health services, development policy financing accounts for 10 percent, and Program-for-Results the remaining 2 percent. The evolution of World Bank support to health services shows different patterns depending on the income level of the recipient countries. World Bank financial support to health services in lower- middle- and upper-middle-income countries represents a small share of total health expenditure FIGURE 2.1 Evolution of World Bank Group Health Services Portfolio, FY05≠16 | 4,000 7% 6% 3,000 5% Percent of total US, millions 4% 2,000 3% 2% 1,000 1% 0 0% 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Year IFC (investments and AS) World Bank (project financing and ASAs) % of total commitments Linear (% of total commitments) Note: AS = Advisory Services; ASA = Advisory Services Analytics; IFC = International Finance Corporation. Independent Evaluation Group | World Bank Group 17 TABLE 2.1 | World Bank Health Services Project Financing: Financing Sources Total Commitments (percent) Total Projects (percent) Source All Projects Active Closed All Projects Active Closed IBRD 43 32 53 17 15 18 IDA 52 63 43 54 60 51 Trust funds 5 5 4 29 24 31 Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. (about 0.1 percent), but almost 13 percent of total DAH. As depicted in figure 2.2, panel a, World Bank support shows significant variability, throughout the evaluation period, with a spike in the volume of commitment in the year 2010 (but more stable trends in terms of the number of approved projects and approved ASAs). Conversely, World Bank financing to low-income countries represents a larger share of total health expenditure (almost 5 percent) but a smaller share, around 8.6 percent, of the total DAH. World Bank support to low-income countries has been somewhat less volatile both in terms of commitment volume and number of ASAs, but it shows, over time, a reduction in the number of approved projects’ financing (figure 2.2, panel b). A similar pattern is confirmed by the country case studies. The World Bank supports health services in about 70 percent of countries in FCV situations. World Bank support to FCV situations represents about 23 percent of projects and 11 percent of commitments approved during the evaluation period, but a smaller share of ASAs (8 percent of the total number of ASAs approved and 4 percent in terms of value). World Bank financial support to countries in FCV situations is about 1 percent of total health expenditure and about 8.5 percent of total DAH. The majority of World Bank support to health services in FCV situations is in Africa. The Democratic Republic of Congo is the country receiving the largest share of World Bank support provided in FCV situations (24 percent), followed by Sierra Leone and Liberia. Trust funds have a strong focus on FCV, as about 49 percent of their disbursements go to countries in FCV situations. The HNP GP delivered most World Bank–financed projects comprising the evaluation portfolio (64 percent of the projects and 83 percent of commitments). Other GPs with projects supporting health services are Social Protection and Labor; Social, Urban, Rural, and Resilience; and Macroeconomics and Fiscal Management, which collectively approved projects for $3.8 billion of commitment, representing 36 percent of projects and 17 percent of commitments. IFC Investment and Advisory Services Between FY05 and FY16, IFC approved 124 investment projects in the health sector, with total original commitments of $2.7 billion. South Asia had the largest share of projects with 23 percent, followed by East Asia and Pacific with 18 percent and Europe and Central Asia with 17 percent. The distribution of commitments indicates a similar pattern. IFC investments have mostly concentrated on 18 World Bank Group Support to Health Services | Chapter 2 FIGURE 2.2 World Bank Project Financing: Commitments and Number of | Projects, FY05–16 a. Lower-middle- and upper-middle-income countries 60 2,500 Commitment (US, millions ) 40 1,500 Number of projects 20 500 0 –500 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Year Commitment (US, millions) Number of projects Number of ASAs Linear (Number of projects) b. Low-income countries 80 3,500 3,000 Commitment (US, millions ) 60 2,500 Number of projects 2,000 40 1,500 1,000 20 500 0 0 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Year Commitment (US, millions) Number of projects Number of ASAs Linear (Number of projects) Linear (Number of ASAs) Note: ASA = Advisory Services and Analytics. lower-middle-income countries and upper-middle-income countries, with 40 percent and 36 percent of projects and 35 percent and 51 percent of original commitments, respectively. IFC’s support to countries in FCV situations and low-income countries is very limited, with the latter decreasing over time in favor of more support to upper-middle-income countries, largely mirroring IFC’s overall investment portfolio. Independent Evaluation Group | World Bank Group 19 Most IFC IS goes to hospitals and clinics, with high concentration in large markets, but its share has been decreasing. Hospitals and clinics account for 62 percent of commitments (61 percent of projects; see figure 2.3, panel a). Most of the support went to China, India, and Turkey, accounting for 69 percent of commitments (65 percent of projects) for hospitals and clinics. Over time, the share of commitments to hospitals and clinics has decreased (from 71 percent among closed projects to 54 percent among open projects) in favor of a higher share of commitments to pharmaceuticals and medicines (20 to 35 percent); (see appendix B). The application of the IEG service delivery and behavior change frameworks found that the rationale provided for IFC IS projects in the health sector emphasized how they are reacting to existing market opportunities and included far less discussion about how the project would contribute to health systems improvement. IFC IS in the health sector shows high presence of repeat support to client networks. Repeat client groups—that is, clients with whom IFC has two or more investments during the FY05–16 period— account for about 62 percent of commitment volume. This is higher than the share of repeat clients in IFC overall (at 48 percent). IEG reviewed a set of repeat interventions with 19 clients, comparing the ex-ante additionality of the first intervention with that of the second. Results show that in 68 percent of the cases, additionality remained the same or evolved into a different type of engagement—for example, from financial to nonfinancial additionality. An analysis of evaluated repeat interventions suggests that realization of incremental additionality seems to diminish over consecutive operations (see appendix B). Most IFC AS is concentrated in hospitals and clinics, but its concentration has decreased over time. IFC AS approved 67 projects in the health sector between FY05 and FY16, accounting for a total commitment of $71 million.1 Hospitals and clinics accounted for 45 percent of commitment volume FIGURE 2.3 Recipients of IFC Investment and Advisory Services | a. IFC Investment b. IFC Advisory Services Hospitals and Hospitals and 61% 62% 55% 45% Clinics Clinics Pharmaceuticals Fund 14% 25% 21% 28% and Medicine Other Medical 9% 19% Specialized 9% 4% Medical Laboratories 11% 6% Fund 5% 3% Health 6% 3% Financing Medical Laboratories 2% 2% Pharmaceuticals 2% 1% and Medicine Other Medical 2% 1% Specialized 3% 1% 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 % of Total Projects % of Total Commitment % of Total Projects % of Total Commitment 20 World Bank Group Support to Health Services | Chapter 2 and 55 percent of the number of projects, followed by funds, which accounted for 25 percent of all commitment volume (14 percent of projects; see figure 2.3, panel b). When comparing closed and open projects, IFC AS’s emphasis on hospitals and clinics has substantially declined from 70 to 41 percent (see appendix B). Projects were concentrated in Africa (21) and South Asia (20), which together accounted for 61 percent of all AS projects. PPP Advisory accounted for 69 percent of all projects. Of the remaining 31 percent, about 13 percent of advisory support (by number of projects) was mapped to the HNP GP (which manages HIA), 3 percent was mapped to the Trade and Competitiveness GP, and the rest (15 percent) to various IFC cross-cutting advisory areas. Aspects of Project Design The majority of World Bank–supported projects approved during the evaluation period aim to strengthen the supply of health services and the health system; demand-side activities are less common. Supply-side and health system–strengthening activities are included in about 90 percent of projects, and their presence is slightly increasing between closed and open projects. Support to health financing, health information management systems, and the use of financial incentives to health service providers, are the activities showing the largest increases over time pointing notably to an increase in the use of RBF approaches. Project documents usually cover the issue of sustainability of resources in World Bank health services projects in terms of fiscal space and fiscal sustainability, but they rarely address operational aspects related to maintenance of the service delivered (see figure 2.4). FIGURE 2.4 Activities Supported by World Bank–Financed Projects | 100 80 Percent 60 40 20 0 Any HF HMIS P&L STW Any CONT. EQUIP FI IC- INFRA MED MED PHS Skills Any CCT IEC IS HS S OUT to HP TECH D HS Health system Supply Demand Closed Active Note: ANY HS = any health system intervention; HF = health financing; FI to HS = Financial Incentive to health Services; HMIS = health information and management systems / monitoring and evaluation; P&L = procurement and supply chain logistics; STW = stewardship, regulations/policy/strategy reform; MED = drugs, vaccines and consumables/supply chain; MED TECH: medical technology; EQUIP = health equipment and labs health; INFRA = infrastructure or land acquisition; PHS = Public Health services (for example, Surveillance), SKILLS = training and skills to HS providers; ANY D = any demand-side intervention; CCT = financial incentives to users with health conditionalities); IEC = information campaign to HS users (households, patients, general population); IS = insurance schemes. Independent Evaluation Group | World Bank Group 21 Only 60 percent of World Bank–financed projects included demand-side interventions, and their presence is slightly decreasing over time. However, projects implemented in FCV situations see an equal presence of supply-side and demand-side interventions (see appendix D). Activities to overcome behavioral or demand-side constraints focused on information or education campaigns and financial incentives. Over time, the presence of information campaigns has decreased, while the use of financial incentives has increased. Examples of financial incentives include CCTs; subsidies to insurance premiums to encourage enrollment uptake; the provision of support for transportation costs from remote areas to district or state health facilities; gratuity of certain drugs and products, such as contraceptives and insecticide-treated bed nets; and the provision of a free package of maternal and child health services. World Bank–financed projects have also provided vouchers to poor women to access free maternal and infant health services and family planning services. World Bank–financed projects focus more on disease prevention than control or treatment. IFC investments focus primarily on treatment. Overall, about 71 percent of World Bank project financing includes prevention activities. Prevention activities are more common in projects in low-income countries and FCV situations, and in recent projects (81 percent in open projects compared with 66 percent among closed projects). Activities directed at disease control and treatment are identified in about half of the projects and appear stable over time. However, treatment activities have increased from 48 percent among closed projects to 59 percent among active projects. Most of IFC investments (84 percent) comprise disease treatment activities, but activities directed at disease prevention and control were identified in only 14 percent and 10 percent of IFC investments, respectively. The prevalence of treatment activities in IFC investments is observed over time, across countries and income levels, and in FCV situations. World Bank project financing supports mostly the primary care level and focuses primarily on maternal and child health and on nutrition. Eighty percent of World Bank–financed projects focus on primary care (the first point of contact with the health system), while only one-third of projects support secondary (or specialized) services. World Bank support to secondary care increased from 27 percent to 42 percent between closed and open projects. Support to the tertiary level and networks is less frequent (around 20 percent of projects). World Bank support to primary care is even more pronounced in low-income countries and FCV situations, where 87 percent of World Bank–financed projects focus on primary care. Regarding health focus, World Bank projects address primarily maternal and child health, and nutrition (46 percent), and communicable diseases (24 percent). The focus on maternal and child health and on nutrition increased between closed and open projects, while the focus on communicable diseases and general health has decreased. IFC’s support is divided more evenly among three levels of care, with slightly more emphasis on secondary care, and the health focus of IFC’s projects toward noncommunicable diseases is increasing. IFC often supports entire networks, which include all levels of care. This confirms the view of some authors that the private sector would usually provide only a limited set of services and would not offer comprehensive care, including preventive and promotive care (Morgan, Ensor, and Waters 2016). IFC concentrates on noncommunicable diseases (35 percent) and general health (50 percent). Consistent with the demographic and epidemiological changes, such as aging populations and 22 World Bank Group Support to Health Services | Chapter 2 the rise of deaths from noncommunicable diseases across developing countries, the IFC IS focus on noncommunicable diseases has increased from 27 percent of closed or matured projects to 45 percent of open projects. Alignment with Country Needs The World Bank project financing shows a positive correlation with both the overall burden of disease and the relative importance of disease burden in the client countries. The scatter plot between country-level World Bank project financing commitments and the disability-adjusted life year per capita indicates that countries with sicker population receive more project financing. ASA’s correlation is much weaker. The evaluation also compared a ranking based on the relative size of World Bank commitments assigned to the six health theme codes,2 with the relative importance of the corresponding disease or health condition measured using the disability-adjusted life years in each of country receiving World Bank support during the evaluation period.3 The analysis shows an overall positive correlation between the two rankings.4 The comparison for the specific health conditions also indicates most countries had good alignment for HIV/AIDS, malaria, child health, and tuberculosis.5 On the other hand, World Bank–financed projects tend to underprioritize other communicable diseases as well as noncommunicable diseases and injuries. Analysis of the alignment of IFC investments with health needs shows that IFC seems to be in the right places, but the alignment with countries’ health needs could be improved further. Ideally, IFC should be better placed to support clients in a country where the needs are large, but also where the business environment is good enough for private sector companies to deliver services with positive returns. For this purpose, IEG developed an index with indicators that reflect health needs and indicators of good business environment with the aim of determining the countries in which there is potential for IFC’s investments to meet these two criteria (see appendix A for methodology). Results show that 64 percent of IFC investments are in the quadrant with the most needs and good commercial attractiveness. However, IFC is active in only about one-third of countries having high needs and good business environment, which suggests that there is potential for IFC investments to expand beyond countries in its current portfolio (see appendix B). Overall, the World Bank portfolio in the six case study countries is aligned with country health needs. This evaluation compared country needs in national health strategies with the objectives of projects in the portfolio of the six countries and found that World Bank Group projects were mostly in line with country development needs. In countries with medium to high government capacity, 6 support has evolved as countries become more committed to universal access and increase their capacity to undertake systemic reform (Brazil and the Philippines). In Romania, however, the government strategy was met with an unsystematic World Bank Group response, mostly because of unstable political leadership, which made Romania unable to sustain and implement reforms. In countries with low government capacity, alignment centered on development needs, but shifted during emergency situations (Liberia and the Republic of Yemen). In Bangladesh, both projects and country health Independent Evaluation Group | World Bank Group 23 strategies centered on government priorities of improving health service delivery and advancing health outcomes. During the evaluation period, the World Bank stepped up investment in population and reproductive health in the countries with high maternal mortality ratios and high fertility rates. Reproductive health services are key to improving reproductive health outcomes through lower fertility rates, improved pregnancy outcomes, and fewer sexually transmitted infections. The 2009 IEG evaluation recommended boosting population and family planning to reduce high fertility. As a result, in 2010 the World Bank approved the Reproductive Health Action Plan 2010–15, which committed to prioritizing the World Bank’s support in the 57 countries with high maternal mortality ratios and high fertility rates (World Bank 2010). The IEG analysis shows that the allocation of project financing to population and reproductive health in World Bank–financed projects approved in FY10–16 versus FY05–09 in these 57 countries increased by 6.9 percentage points faster than the rest of the countries comprised in the World Bank health service portfolio.7 This finding is consistent with the commitment to increase support to reproductive health, as indicated in the 2010–15 health plan. However, based on the qualitative analysis of projects in the evaluation portfolio having population and reproductive health activities, it is not clear if the additional resources provided by the World Bank would be sufficient to resolve pressing issues (see appendix A). The nature and extent of challenges in FCV situations go well beyond the capacity of the World Bank Group, pointing to the need to coordinate with all development partners. However, this is not always achieved. The recent experience in the Republic of Yemen illustrates a groundbreaking engagement model that ensures continuity of basic health services through strong coordination with other multilateral organizations to combine lifesaving humanitarian aid with long-term development objectives. By contrast, the support provided in Liberia after the civil conflict (2007–13) was only loosely coordinated with other development partners (see appendix E). Health services are an important element for SGBV prevention and response, but few projects in the World Bank portfolio address this issue. Projects providing reproductive, maternal, and child health services can reach potential victims and survivors of SGBV through targeted outreach, information, education, and communication campaigns. Notwithstanding the expanding needs and the potential support that could derive from World Bank–financed projects, the IEG evaluation identified few projects supporting SGBV. Therefore, substantial room for expanding World Bank Group support to SGBV in FCV situations through health services exists (see appendix B). Project Outcome Ratings The performance of World Bank health services projects closed and evaluated during FY05-16 has improved markedly since 2005, but the average rating remains below the average for the overall World Bank portfolio. Over the entire evaluation period (FY05–16), 71 percent of project financing supporting health services was rated moderately satisfactory or better (MS+), which is slightly below the overall World Bank portfolio (73 percent).8 The performance of the health services portfolio 24 World Bank Group Support to Health Services | Chapter 2 lagged the overall World Bank portfolio by 11 percentage points in the period FY02–06 (World Bank 2009, 19). The gap in performance grew to about 20 percentage points in the period FY05–07 to FY07–09. The performance of the health portfolio improved markedly starting in FY10, and during FY10–16, the HNP portfolio outperformed the overall World Bank portfolio (see appendix B) with the performance rating higher by 3 percentage points than the World Bank Group average. World Bank–financed projects performance shows marked Regional differences. Project financing in the Middle East and North Africa, Europe and Central Asia, and Africa Regions perform better than the entire portfolio of health service projects. Projects implemented in the Latin America and the Caribbean, East Asia and Pacific, and South Asia Regions underperform compared with the evaluation portfolio. This picture has changed markedly from the 2009 IEG evaluation, which reported that the share of health operations in Africa with satisfactory outcomes was the lowest across Regions.9 A closer look at the set of World Bank project financing supporting health services closed and evaluated during FY05–16 reveals project- and country-level factors that correlate with project outcomes. Project outcome is found to correlate negatively with the number of project development objectives (PDOs; projects with three PDOs or more are less likely to be rated MS+ compared with those with one or two PDOs). Projects implemented in countries with a more favorable economic context (more resources allocated to health and higher per capita gross domestic product growth) were found more likely to be rated satisfactory. World Bank Group projects in the health sector are not immune to corruption. In fact, World Bank Group health sector projects appear to be more vulnerable to corruption than in other sectors. The Integrity Vice Presidency of the World Bank Group opened 191 cases in the HNP sector from FY07 to FY16, which represent around 18 percent of the total cases opened during this period (only the transport sector registered more cases). This is consistent with the general findings that the health sector is particularly vulnerable to corruption (see Savedoff 2006; Vian 2008). Opportunities for corruption are greater in situations where the government agent has monopoly power over clients; officials have a great deal of discretion, or autonomous authority to make decisions, without adequate control on that discretion; and there is not enough accountability for decisions or results (Klitgaard 1988). However, as indicated in IEG’s framework for the evaluation of service delivery and behavior change strategies, enhancing citizen participation and voice (for example, the active participation of stakeholders in the planning and provision of services) are effective strategies for increasing transparency and accountability and to control discretion and reduce monopoly power (Lewis 2006; Vian 2008). IFC’s performance in health services is well above the rest of IFC’s portfolio, but the performance has been declining over time. About 75 percent of IFC IS health projects are rated MS+, which is a significantly better than the overall IFC investment portfolio (57 percent). Success is associated mostly with IFC’s strong experience in supporting large networks and investing in repeat projects with the same sponsor (see appendix B). Projects evaluated during the FY13–16 period show a steeper decrease in performance in the outcomes of health sector projects (from 91 percent to Independent Evaluation Group | World Bank Group 25 65 percent) than the overall portfolio (from 63 percent to 55 percent). The evaluation shows that the decrease in performance of these projects is attributable to different factors, including competition with the public sector, uncertainty of public payment, sponsors with poor local knowledge, and poor governance and management quality. Finally, 64 percent of IFC AS projects are rated satisfactory or better, comparable with the overall IFC AS portfolio success rate of 58 percent. However, because of the limited number of IFC AS projects evaluated, the confidence interval is relatively large and the difference is not statistically significant. Achievement of Project Development Objectives The assessment of the specific PDOs allows for a better understanding of World Bank Group performance along the health services–related outcomes, and in turn, its contribution to universal health coverage. PDOs identified in World Bank–financed projects and in IFC investments and advisory services comprising the evaluation portfolio were grouped into homogeneous categories that map the evaluation’s intervention logic, such as (i) improve access to health services, (ii) improve quality of health services, (iii) improve equity, (iv) strengthen health systems, (v) improve health outcomes, and (v) enhance efficiency. IFC investment and advisory services can also seek objectives beyond the health sphere, such as (i) stimulate private sector development, (ii) promote job creation, and (iii) foster innovation (see figure 2.5). Improve access to and quality of health services are two of the most common objectives in World Bank Group projects approved in the FY05–16 period. World Bank project financing also shows a strong focus on strengthening health systems and improving health outcomes. Improving health systems is a PDO sought by IFC AS, but to a lesser degree than the World Bank. PDOs such as improve equity and enhance efficiency are not very common across World Bank Group interventions. World Bank PDO ratings given in the IEG validation of World Bank financing projects’ implementation completion and results were used to calculate the achievement of each PDOs category.10 The IFC evaluative evidence contained in project-level evaluation reports provides important evidence. However, it measures the project development outcomes based on four dimensions (project business success, environmental and social effects, economic and social sustainability, and private sector development), and no specific health services project objectives are rated. To assess the extent to which IFC projects achieved their intended PDOs, additional analyses were required, and a rating of achievement was assigned based on the available evidence provided. Improving Access to Health Services The World Bank shows substantial contributions to improving access to health care services, particularly in low-income countries. In fact, improving access appears as the top priority of World Bank–financed projects. It is present as a PDO in 54 percent of World Bank project financing in the evaluation portfolio (figure 2.3), and its presence has increased over time. Access was by far the project goal with the best achievement (70 percent of these PDOs are rated substantial or better [S+]). Lack of access is one of the early challenges on the path to improving health services in low- 26 World Bank Group Support to Health Services | Chapter 2 FIGURE 2.5 | Frequency of Project Development Objectives, by Institution World Bank Financing IFC Investments IFC Advisory 71% 73% 83% Improve access 54% 88% 88% 46% 73% 100% Improve quality 52% 27% 69% 55% Strengthen health systems 19% 37% 3% 51% Improve health 1% 29% 1% 38% 25% Improve equity 1% 8% 11% 45% Enhance efficiency 12% 11% 8% 38% Private sector development 30% 48% 50% 3% Promote job creation 56% 50% 6% Foster innovation 1% 10% Other 4% 1% 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Percentage of PDO rated about High or Substantial (5+) Percentage of projects by PDO type Independent Evaluation Group | World Bank Group Note: IFC = International Finance Corporation; PDO = Project Development Objective; WB = World Bank. 27 income countries. The focus on the access objective and its achievement are more prominent in low- income countries (60 and 86 percent, respectively) than in lower-middle- and upper-middle-income countries (40 and 60 percent, respectively). World Bank-financed projects that have adopted CCTs and PBF interventions have shown better results in improving access to health services than the rest of the portfolio. CCT provides demand- side incentives to stimulate the uptake of health services (especially the nutrition interventions in poor households). PBF provides financial incentives to health providers based on the quantity (and quality) of health services provided. Evaluated World Bank–financed projects with CCT interventions always achieved a substantial or better (S+) rating for PDOs aiming at improved access, and projects with PBF interventions received an S+ rating in 84 percent of PDOs aimed at improving access (see appendix D).11 The evidence on the effectiveness of CCTs on access is more conclusive than the evidence on the effectiveness of PBF. Evidence gap maps have been constructed to show visually the quantity and quality of the evidence, derived from systematic reviews of the literature, on the effectiveness of CCT and PBF interventions on the relevant health service outcomes (see appendix E). The evidence on the effectiveness of CCTs on access is based on 11 systematic reviews, the majority of which were of medium quality, while the evidence on the effectiveness of PBF on access is based on four systematic reviews, all of low quality.12 In part, the more limited evidence on PBF is related to PBF being introduced in World Bank projects more recently than CCTs. The underlying systematic reviews present evidence of a positive influence of CCTs on access, but results for PBFs are mixed. For example, CCTs appear to be effective in improving access to various health services, such as antenatal visits, institutional deliveries and deliveries attended by a health professional or skilled birth attendant. Most systematic reviews of PBFs assess the impact on institutional deliveries, but the results were generally mixed. Mixed results were also reported in the only high-quality systematic review conducted by Witter et al. (2012). Therefore, more research is needed in this area. IFC contributed to improving access to health services, but with limited evidence of systemic impact among the underserved population. IFC aims to improve access in 88 percent of its investment projects, and this priority has remained the same over time. Overall, 73 percent of IFC IS projects show evidence of improving access. Most of the evidence is related to enhancing the availability of health services (for example, number of beds added, number of medicines distributed) followed by limited evidence related to use of services (for example, number of patients treated). Although some projects still seek to assess the use of health services, these indicators are not adjusted for the health needs of the target population. Because IFC focuses on large markets and, within them, networks, it could potentially achieve systemic impact among the underserved population. However, the board reports and the evaluative evidence reviewed did not offer evidence to assess affordability or indicate the main users of the facilities. Therefore, even if evidence that IFC IS improves availability and use of health services exists, it is not possible to determine if those reached were populations already covered or the underserved. This speaks to the weakness identified in the 2009 IEG health evaluation, which recommended that IFC enhance its results orientation through stronger M&E, including beneficiary-level information. 28 World Bank Group Support to Health Services | Chapter 2 IFC AS supporting PPPs show evidence of improving access and having systemic impact. IFC AS projects advise governments to structure PPPs, and the projects usually end at commercial closure (that is, before the project implementation phase). Although IFC’s PPP advisory projects have development objectives, their achievements cannot be measured until the PPP becomes operational. Consequently, IEG reviewed the post-completion reports of projects that benefitted from IFC AS and complemented them with country case studies. In Romania, the system of dialysis clinics that are privately operated and publicly financed by the National Health Insurance has significantly increased capacity. Because of the positive impact of the pilot, the Government of Romania decided to expand the model. To date, 90 percent of dialysis patients are treated privately. Similarly, in Brazil, an emergency greenfield hospital in a poor area catered to around 11 percent of the hospitalizations in public hospitals in the city of Salvador. Improving Quality of Health Services Quality, which can be divided into three distinct dimensions, is an essential attribute to ensure that health services are effective in improving health status. Quality is defined as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” (Lohr 1990). A growing body of empirical evidence shows that the quality of care varies significantly, that it is typically inadequate, and that it contributes to observed differences in health outcomes (Das, Hammer, and Leonard 2008). In addition, patients have difficulties in determining the quality of health services, and their decision to use health services—particularly in settings with low resources and poor administrative capacity— correlates with provider effort (Das, Holla, Mohpal, and Muralidharan 2016). The clinical literature highlights three distinct dimensions of the quality of health: (i) “structure” refers to the attributes of the settings in which health care occurs (that is, material resources such as facilities, equipment, and financing; human resources, such as the number and qualifications; and organizational structure); (ii) “process” relates to what is actually done in giving and receiving care (that is, the interactions between the provider and the patient); and (iii) “outcome” describes the effects of care on the health status of patients and populations (see Mainz 2003; Smith and Nguyen 2013). The World Bank Group’s focus on quality of health services has been low but improving. Over time, World Bank–financed projects have been focusing more on improving the quality of health services (18 percent of closed projects have this PDO versus 44 percent of active projects). The share of PDOs aiming at improving quality of health services rated S+ was only 46 percent, but projects with PBF interventions show better results (67 percent of PDOs rated S+) and stronger M&E frameworks. Measuring quality of health care is complex because it requires covering all three dimensions (structure, process, and outcome) with relevant indicators, which is rarely seen in the evaluation portfolio.13 Virtually all PBF programs include indicators based on the structural aspects of quality (Gergen et al. 2017). About half of PBF projects that aimed to improve quality of services included an indicator of quality process (for example, compliance with medical protocols); quality improvement was achieved in about 76 percent of projects closed and evaluated. The use of indicators for outcome aspects of quality—for example, patient satisfaction—was less frequent. However, recent PBF projects have started to use PDOs indicators linked to aggregate health outcome measures Independent Evaluation Group | World Bank Group 29 at the facility level (for example, tuberculosis treatment success rate, prevalence of high blood pressure under control). Overall, although the indicators used to monitor quality improvement in World Bank–supported projects have been improving, there is still room for improvement because M&E frameworks rarely comprise indicators that can capture all the three relevant quality dimensions (structure, process, and outcomes) and the links among them. The evidence on the effectiveness of PBF on the quality of services is mostly of low quality (only one high-quality systematic review was identified), and the results of the underlying systematic reviews are generally mixed. One of the key reasons for this result is that PBF is not a uniform intervention, but rather a range of approaches. Its effects depend on the interaction of several variables, including the design of the intervention (for example, who receives payments, the magnitude of the incentives, and the targets and how they are measured), the amount of additional funding, other ancillary components such as technical support, and contextual factors, including the organizational context in which it is implemented. Therefore, even if PBF is a promising approach and World Bank–financed projects containing PBF interventions perform better in improving quality of services, it is not possible to draw clear conclusions; and more research in this area, including how PBF works and how to maximize its impact, is needed (Witter et al. 2012). IFC projects contribute to enhancing the quality of health services, although the emphasis on quality is declining over time and it relates, almost exclusively, to structural aspects. About 69 percent of IFC investment projects aim to improve the quality of health services, and the proportion of projects containing this objective has decreased from 75 percent among the closed portfolio to about 61 percent of the active portfolio. Overall, 73 percent of IFC IS with quality as an objective has improved the quality of health services. By contrast, 27 percent of IFC IS with quality as an objective have not achieved their results. Investing in institutions that provide good quality of health care is a prerequisite for all IFC investments in the sector. However, when IFC projects aim specifically at improving quality, they monitor structural metrics of quality almost exclusively. Among the 20 PDOs of 17 projects, 15 PDOs were related to structural quality (10 were related to licensing and accreditation standards required for health facilities and health providers, and five were related to trained staff), and five PDOs did not have any indicator. However, other service quality measures, such as responsiveness of staff measured by patient satisfaction or process quality related to the content of the visit, were not measured. This finding is consistent with the service delivery framework analysis of IFC health investments, which found that service monitoring was rarely included in projects. This gap in measurement represents a missed opportunity. Some evidence suggests that service quality in terms of patient satisfaction tends to be better in the private sector (Morgan, Ensor, and Waters 2016). Improving Equity in the Use and Financing of Health Services World Bank Group projects rarely present explicit equity objectives (8 percent of World Bank project financing, 11 percent of IFC IS, and 1 percent of IFC AS). However, there seems to be an implicit equity focus in a larger number of World Bank Group projects. For example, the majority of World Bank investment project financing (64 percent) identifies specific disadvantaged population groups as the intended beneficiaries, and the focus on disadvantaged population groups has increased over 30 World Bank Group Support to Health Services | Chapter 2 time (70 percent in active projects versus 59 percent in closed projects). The focus on disadvantaged population groups captures both the concept of universal health coverage (everyone, everywhere can access quality health services when needed without being forced into poverty) and the shared prosperity goal (expand service delivery for the poor or the bottom 40 percent). However, even if World Bank–supported projects often prioritize disadvantaged groups—and this is also evident from the increasing focus on maternal and child care—the distributional impacts are rarely monitored and evaluated. This finding confirms the statement in the management action record of the 2009 IEG evaluation that the World Bank substantially increased its focus on the poor during 2010–13. It also confirms the finding of the IEG 2014 health financing evaluation that the M&E of World Bank Group projects needs to be strengthened by “monitoring distributional indicators, including on access and outcomes, consistent with benchmarking and tracking progress toward Universal Health Care coverage” (World Bank 2014a, xv). IFC’s support to vulnerable populations through investments has yet to yield satisfactory results. The evaluation of IFC projects identified only four operations with an explicit equity objective. All but one were rated unsatisfactory, mostly because of an inadequate level of government compensation or incentives for the provision of services, lack of enforcement and accountability, and poor governance. The private sector can contribute to universal health coverage by providing quality services that are affordable for the poor. IFC is working toward this goal by investing in clients with a strong focus on corporate social responsibility and a commitment to serving low-income patients as well as by piloting investments that target the base of the economic pyramid. Strengthening Health Systems The presence of explicit PDOs aimed at strengthening the health system has been decreasing over time. About 37 percent of World Bank project financing approved during the evaluation period had a specific PDO aimed at strengthening the health system, but their presence decreased from 42 percent among closed projects to 27 percent among open projects. It is noteworthy that projects supporting pandemic preparedness and control have shown, over time, increased presence of PDOs aimed at strengthening health system functions, which is consistent with the need to improve health system capacity to respond to pandemic outbreaks (see appendix D). PDOs aimed to strengthen the health system are rare in IFC investments (3 percent) because they usually support individual health services providers, but they are more common among IFC AS (19 percent). Health systems–strengthening projects are more effective when the scope of the objectives is well defined. Overall, about 55 percent of all health systems objectives in World Bank–financed projects were rated S+ during the evaluation period. World Bank projects supporting CCTs performed better on this PDO type (67 percent are rated S+). The in-depth analysis reveals that CCT projects define the scope of the health systems–strengthening objective better and focus in areas where the World Bank has accumulated significant technical experience (for example, strengthening of management information systems with the aim of integration with the health sector systems). This lesson—better defining the scope of the objective—is relevant for all World Bank Group projects aiming at health system strengthening. For IFC, the evaluative evidence is too limited to assess performance on this Independent Evaluation Group | World Bank Group 31 objective. The evaluation portfolio comprises only three evaluated IFC projects (one investment and two advisory services) that aimed to strengthen health systems. Improving Health Outcomes About 29 percent of World Bank-financed projects have PDOs aiming explicitly at improving health outcomes, and their presence has declined over time (from 35 percent of closed projects to 16 percent of open projects). This decrease is more pronounced in projects in low-income countries (from 40 percent to 11 percent) and, even more, in projects comprising pandemic interventions (from 92 percent to 25 percent). Only 1 percent of IFC investments and advisory services comprise PDOs aiming at improving health outcomes. The decline is a deliberate effort of the HNP GP to focus on PDO outcomes that are measurable within the lifetime of the project. About half the PDOs achieved the desired health improvement. It should be noted that indicators used in health outcome PDOs (mortality rates, for example) may not be sufficiently sensitive to detect changes within the project duration and may be subject to attribution challenges, and their declined use may therefore be understandable. However, this should not discourage the quest for M&E health outcome improvement. As already indicated, the clinical literature provides technically robust indicators for outcome aspects of quality that are overall related to final health outcomes, and some have been adopted in World Bank–supported projects (for example, indicators that measure tuberculosis treatment success rate, prevalence of high blood pressure under control). Therefore, health outcomes of World Bank Group operations can be monitored and evaluated with appropriate and contextualized indicators of outcome. World Bank–supported projects comprising CCT and PBF interventions were more successful than the rest of the evaluation portfolio in achieving health outcome PDOs (80 percent and 67 percent of PDOs were rated S+, respectively). The evidence gap maps in appendix E show one high-quality, six medium-quality, and two low-quality systematic reviews assessing the impact of CCTs on nutritional status and health outcomes assessed by anthropometric measurements and self-reported episodes of illness, respectively. The evidence base on the effectiveness of PBF interventions to improve health outcomes is of lower quality (one high quality and two low-quality systematic reviews). Selected Delivery Mechanisms for improving Health Services World Bank Group Support to Public-Private Interaction This evaluation identified only a few World Bank–financed projects aiming to better integrate the private sector into national health systems. This specific aspect of strengthening health system functions was found in only 7 percent of World Bank–financed projects (46 of 619 projects approved during FY05–16). Although the targets used to monitor the success of the integration were achieved in 70 percent of the cases, they were often output-level indicators, such as developing a health sector strategy, that involved the private sector, and developing guidelines, policies, or regulations relevant to private providers (for example, licensing and accreditation of health facilities or individual health workers). In addition, there is a joint World Bank Group initiative, HIA, highly relevant to public- 32 World Bank Group Support to Health Services | Chapter 2 private sector partnership. HIA is a joint World Bank and IFC initiative that provides support to client countries and private operators in Africa to better integrate public-private interactions. HIA has been supporting governments to strengthen regulations, processes of public and private health facilities, and accreditations of private health facilities. However, so far, examples of successful integration have been limited to Ghana and Kenya (see box 2.2). Limited collaboration between the World Bank and IFC continues to be a bottleneck in the provision of integrated solutions relating to effective private sector participation in health. The 2009 IEG evaluation echoed the 2007 HNP strategies and identified the need for “leveraging World Bank sector dialogue on health regulatory frameworks to engage new private actors, and more systematically coordinate with the World Bank’s policy interventions regarding private sector participation in health” (World Bank 2016a, 2). Country case studies and portfolio review show similar findings. The appointment of a private sector global lead in HNP in 2016 is a positive development, but limited resources and expertise represent a constraint to stepping up integrated solutions in the field. The inadequate integration of IFC investments within public financing reduces their potential to expand coverage among the poor and their contribution to universal health coverage. The poor have little capacity to pay for health services, thus the integration of private provision with public financing is—usually—necessary to make health services affordable, and thus accessible to them (Hammer, Aiyar, and Samji 2007). Most IFC investments (48 projects, or 53 percent of the portfolio) went to companies seeking both private and public financing for their health services. Three IFC investments were to PPPs (located in Turkey), the other 45 to private operators. However, the IEG review found that hospitals and specialty chains supported by IFC investments did not manage to attract public financing and continued to rely primarily on out-of-pocket payments (see appendix D). The main reasons identified were the limitation of public resources (and related low pricing of the services) and inadequate regulation. However, as markets mature and public sector financing improves, private providers tend to integrate more with public financing (for example, in Brazil and Turkey). Health PPPs supported by IFC AS face challenges resulting from poor public-private integration. This was evident in IFC-supported PPPs in Bahia (Brazil), India, Lesotho, Mexico, and Romania. In Lesotho and Bahia, for example, the primary care facilities were not ready when the PPPs started to operate. Therefore, the referral system was inadequate, resulting in an overflow of demand and unexpected fiscal pressures on government. Other health systems challenges included retention of health care professionals (Lesotho, Mexico, and Romania), delays in government payment (India, Lesotho), inadequate calculation of government contribution (Bahia), delays in matching human resources availability with infrastructure (Mexico), limited accountability function (Romania), and government capacity to manage PPPs (Lesotho). These results were evident from IEG’s review of the post- completion reports and evidence gathered during field visits. IFC AS shows insufficient information to assess the equity, efficiency, sustainability, and fiscal burden of the PPPs. Although there is an established set of minimum requirements for post-completion reports, the methodologies and the framework for measuring long-term results vary significantly. This Independent Evaluation Group | World Bank Group 33 makes it difficult to compare results (appendix D). A recent IEG health PPP report confirms these findings: “Self-assessment reports (prepared two years after completion) indicate positive effects of PPPs in some areas, for example, access and quality; but evidence is still limited. The World Bank Group [M&E] system for health PPPs is inadequate and needs to be improved to better track results” (World Bank 2016a). Use of Incentives This evaluation shows that the World Bank Group increased capacity to generate evidence on the effectiveness of financial incentives, such as CCTs and PBFs. Most World Bank–financed CCT projects included in the evaluation portfolio indicated the intent for conducting an impact evaluation (65 percent). Most World Bank–financed impact evaluations of CCTs measured the effects on health outcomes (70 percent) and access to health services (50 percent), aspects of CCTs that were already well studied. However, few have assessed their distributional impact with respect to access, use, and financing of health services, which may represent a missed opportunity. About one-third of World Bank–supported projects with PBF interventions planned for an impact evaluation. This is a significant improvement from the situation identified in the 2009, when IEG recommended to “boost investments in and incentives for evaluations” (World Bank 2009, xxii). The HRITF has been instrumental in stepping up the generation of evidence around RBF and PBF through impact evaluation and its dissemination, but more research is needed, and the use of the evidence generated in country-level support remains a work in progress (see box 2.1). PBF interventions are more complex and less studied than CCTs. The effects of PBF interventions depend on the interaction of several variables (for example, the design of the intervention, the amount of additional funding, ancillary components such as technical support, and contextual factors, including the organizational context in which it is implemented). Thus, more research is needed to better understand how PBF works and how to maximize its impact. The use of financial incentives to health providers needs to be integrated within the health system to ensure long-term sustainability and avoid distortions. The 2014 health financing evaluation noted that the limited integration of RBF programs, including PBF, with other health financing functions (for example, purchasing) and the broader public financing context generates sustainability risks and potential distortions (World Bank 2014). The substantial improvement recorded in the management action records shows management’s efforts in this area. This evaluation also noted the opportunity to improve the integration of PBF monitoring and verification systems within the overall national health information systems to improve the long-term sustainability of PBF pilots (see appendix E). World Bank Support for Pandemic Preparedness and Control The results of World Bank support for pandemic preparedness and control under the Global Program on Avian Influenza Control and Human Pandemic Preparedness and Response (GPAI), are mixed. The World Bank–managed GPAI was approved in 2006. During 2006–13, it financed 83 operations (across 63 countries) that addressed avian influenza, zoonotic diseases, or pandemic preparedness 34 World Bank Group Support to Health Services | Chapter 2 Box 2.1 | The Health Results Innovation Trust Fund The Health Results Innovation Trust Fund (HRITF) is a multi-donor trust fund established in 2007 with support from Norway and the United Kingdom, aimed to support design and implementation of results-based financing (RBF) approaches to improve service delivery in maternal, neonatal, and child health. HRITF has fully achieved two of its objectives: supporting design, implementation, monitoring, and evaluation of RBF mechanisms; and attracting additional financing to the health sector through leveraging. As of 2016, the HRITF financed 34 RBF impact evaluations—28 accompany country pilot grants and six are stand-alone impact evaluations. It committed $385.6 million for 35 RBF programs in 29 countries, linked to $2.0 billion in financing from IDA. HRITF is also progressing in its objective to develop and disseminate the evidence base for implementing successful RBF mechanisms. Learning events like the Annual Results and Impact Evaluation Workshop provide opportunities for RBF stakeholders to share results and knowledge, discuss implementation experiences, and learn from peers and technical experts. HRITF also participates in global events, such as the Global Symposium on Health Systems Research in November 2016, where it shared early evidence from impact evaluations. HRITF’s RBF Bulletin is another avenue for sharing knowledge with audiences around the world. HRITF is also progressing in its objective of building country institutional capacity to scale up and sustain the RBF mechanisms, according to national health strategies and systems. Several HRITF-supported projects include cost-effectiveness analysis as part of their impact evaluation, and exploring lower-cost options for the implementation of certain components of RBF programs to improve financial and institutional sustainability prospects. A few countries (for example, Burundi and Cameroon) have moved to a nationwide expansion of RBF, and others (for example, the Lao People’s Democratic Republic, Tanzania, and Zambia) adopted RBF principles and tools to inform new World Bank Group–supported projects. The Global Financing Facility (GFF) has committed to support sustaining and scaling up the RBF pilots if countries choose to do so (Cameroon, for example). However, it is still uncertain how the transition from HRITF will work in non-GFF countries. In those countries, the World Bank has a critical role to play to ensure a smoother exit or transition from the pilots. Independent Evaluation Group | World Bank Group 35 and control.14 In general, the human health components had slower implementation than the animal health components (except in cases where the human health component was integrated with an existing health project). Human health components were affected by a waning in momentum that followed the drop in the number of confirmed cases of avian influenza in 2007–08. The World Bank and partners (Food and Agriculture Organization of the UN and the World Organisation for Animal Health) did not manage to sustain the zoonotic disease risk management and pandemic preparedness efforts. Even if the 2007 health strategy recognized the need to strengthen health systems to fight pandemics (World Bank 2007b, 33), the World Bank failed to mainstream pandemic preparedness and control agendas into its operations. Since 2010, only two new World Bank–financed projects addressing avian influenza or zoonotic diseases (in Nepal and Vietnam) were identified (World Bank 2014b, 19), even if various subtypes of avian and swine influenza that are transmissible from animals to humans continue to be reported. The World Bank Group showed responsiveness and was a key member of the global coalition that fought the Ebola virus outbreak in West Africa during 2014–15. Following the preparation of a plan to contain the Ebola outbreak by a global coalition led by WHO, the World Bank approved the Ebola Emergency Response Project in September 2014—28 days after WHO declared the epidemic a public health emergency of international concern.15 The World Bank Group sent a senior public health specialist to WHO to assist in the coordination of the technical and financial efforts of these two institutions. The World Bank Group mobilized $1.62 billion ($1.17 billion from IDA and at least $450 million from IFC) to support Ebola response and recovery efforts in the three West African countries hardest hit by Ebola: Guinea, Liberia, and Sierra Leone.16 A top priority for the global coalition was to build the necessary health workforce quickly to detect Ebola cases, treat them, and contain the outbreaks. The World Bank Group also supported restoring basic health services, helped poor households with cash transfers, provided farmers with seeding to plant their fields, and supported foreign investors to come back into the Ebola affected countries. Experience from three other countries in Africa shows that stronger health systems in the three most affected countries could have helped avoid the spread of Ebola. The Ebola virus outbreak affected not only Guinea, Liberia, and Sierra Leone, but also Mali, Nigeria, and Senegal. However, the latter countries could mount a successful and rapid response before the virus outbreak spiraled out of control. The key elements for success in controlling the epidemic were (i) fast and thorough tracing of all potential contacts; (ii) ongoing monitoring of these contacts; and (iii) rapid isolation of potentially infectious contacts (Fasina et al. 2014). At the time the outbreak began, the capacity of the health systems in Guinea,17 Liberia, and Sierra Leone was limited (for example, health services understaffed, equipment in short supply, and limited capacities for laboratory diagnosis, clinical management, and surveillance; Briand et al. 2014; Kieny et al. 2014). Conversely, Mali, Nigeria, and Senegal had their own high-quality laboratories, facilitating the rapid detection or discarding of cases.18 Contact tracing was rigorous, and most identified contacts were monitored in isolation. Local staff and existing infrastructures were used in innovative ways. For example, Mali used medical students with training in epidemiology to increase staff numbers for contact tracing. All three countries established emergency operations centers and recognized the critical importance of public information 36 World Bank Group Support to Health Services | Chapter 2 campaigns that encouraged community cooperation (WHO 2015b). In Nigeria, for example, the Polio Emergency Operations Centre and its vast experience and resources operated as the national Incident Command System and served as a springboard for Nigeria’s Ebola response. Synergies and Complementarities Internal Synergies The potential for tapping the complementary roles between the private and public sectors (and, therefore, between IFC and the World Bank) exists but has not been sufficiently exploited. The portfolio review of projects suggests that only 10 World Bank Group evaluated projects (4 percent) envisioned some type of World Bank Group collaboration or complementarities among projects. The sample of six country case studies also found limited instances of complementarities between institutions. For example, in Liberia, IFC AS is working jointly with the World Bank to establish a National Diagnostic Center PPP. In the Republic of Yemen, good complementarity was envisioned between IFC’s investment in two hospitals and the World Bank–supported project financed by the Global Partnership on Output-Based Aid to support maternal health and childbirths in the low- income population. In Romania, World Bank and IFC support were not aligned initially, but they converged over time. In Brazil and the Philippines, the World Bank and IFC had little coordination. This finding is supported by a recent IEG study showing that complementarities between the World Bank and IFC are rare. In only a few cases have countries received timely private sector development support for health or specific support toward health PPPs (World Bank 2016). The World Bank Group’s cascade approach is relevant for interventions in the health sector. The approach first seeks to “mobilize commercial finance, enabled by upstream reforms where necessary to address market failures and other constraints to private sector investment at the country and sector level. Where risks remain high, the priority will be to apply guarantees and risk-sharing instruments. If private solutions are not possible through sector reform and risk mitigation, then official and public resources would be applied” (World Bank 2017e, 6). The building blocks enshrined in this approach are not entirely new. World Bank Group collaboration and joint multilateral development banks’ efforts to leverage the private sector investment have been sought before (World Bank 2017e). The current approach, however, aims to use the full capabilities of the World Bank Group. Although infrastructure and energy are often mentioned as highly relevant for the implementation of the cascade approach, the health sector is a candidate for the approach. Experiences from the HIA initiative and from Turkey speak to opportunities and challenges of implementing such an approach in the health sector (see box 2.2). Complementarities among World Bank GPs delivering support to health services in the countries selected for case studies are generally good. In Liberia, HNP led the delivery of multisector support to fight the Ebola outbreak, which comprised the provision of cash transfers, food, and other basic supplies for affected households,19 the distribution and storage of foundation seeds to reactivate the agricultural sector, and the distribution of textbooks and provision of hand pumps for public schools. In the Philippines, World Bank multisectoral work comprising both project financing (that is, the Independent Evaluation Group | World Bank Group 37 Box 2.2 |  Lessons Learned from Early Initiatives to Create Markets for Health The Health in Africa Initiative (HIA) was created in 2007 to improve access to and quality of health goods and services in Africa. The International Finance Corporation (IFC) through HIA mobilized investment vehicles, risk-sharing solutions, and technical assistance in the form of market studies and investment climate assessment to stimulate reforms and investment opportunities. The investment fund had some success. It reached small- and medium-sized enterprises and mobilized investments in a few countries. The risk-sharing facilities did not meet market conditions and were closed with little usage. The significant upstream support for market studies and country assessments that led to the identification of areas for reforms in about 10 African countries. However, subsequent engagements in the form of advisory services or investments were limited to few countries (for example, Ghana and Kenya). The main challenges are as follows: HIA involved the provision of support from both IFC and the World Bank. However, n  internal synergies were weak until the HIA team was transferred under the Health, Nutrition, and Population Practice Manager for the Africa Region. HIA governance was not well defined. The weak governance structure thwarted n  the reach of a shared strategy and generated mismatches among partners’ expectations. The weak coordination among partners, caused some major donors to retreat from the partnership, which raises questions about HIA’s future. Since 2003, IBRD supported the Government of Turkey in undertaking health sector reforms and strengthen its institutional capacity to implement the public–private partnership (PPP) program. The Multilateral Investment Guarantee Agency (MIGA) provided political risk insurance for six PPP projects, with a total value of $848 million.a IFC invested $163 million in senior debt and mobilized an additional $430 million from other lenders for three first-mover PPP projects. The European Bank for Reconstruction and Development (EBRD) has provided additional risk mitigation through a greenfield project bond. The combined MIGA-EBRD-IFC financing structure has enhanced the rating of the bond issued (rated by Moody Baa2, which is above Turkey’s sovereign rating) and attracted new investors in Turkey health PPP. To date, 10 health PPP projects have reached financial closure and are under implementation. Key Lessons Learned Strong government commitment and leadership is needed for success. n  Health PPP take years to materialize. Sustained engagement is key. n  38 World Bank Group Support to Health Services | Chapter 2 Box 2.2 |  Lessons Learned from Early Initiatives to Create Markets for Health (continued) Closer coordination and alignment within the World Bank Group is needed to n  realize the potential for market creation. Donor coordination requires a clear understanding of programs objectives and strategies. Approach and deployment of instruments must be tailored to country context. n  a. Only one of the six PPP projects was approved during the study period, and five were approved in 2017. Development Policy Loans to Foster More Inclusive Growth implemented during 2011–15) and ASAs (see the public expenditure review P122574 and Kaiser, Bredenkamp, and Iglesias 2016) supported policy measures that increased excises from alcohol and tobacco to finance the expansion of the subsidized health insurance program (PhilHealth). In Brazil, there was strong coordination between GPs (HNP, Education, Social Protection and Labor, and more recently, Water) that delivered six multisectoral projects at state and municipal levels with health service components. The collaboration between HNP and other sectors to support health services in the Republic of Yemen is recent. The Emergency Health and Nutrition Project complemented the ongoing interventions offered by the World Bank by adopting a multisectoral approach. A joint project with the Water GP aims to address the cholera epidemic in the Republic of Yemen. External Synergies The World Bank Group portfolio suggests that collaboration with other development partners is more common in low-income countries. The analysis of evaluated projects shows that World Bank Group seeks collaboration with development partners in 33 percent of project financing. Of these, 30 percent are in low-income countries and 22 percent in lower-middle-income countries. Cofinancing is present in only about 16 percent of the projects for both low-income and lower- middle-income countries. To assess the World Bank Group’s role in projects involving other development partners, the evaluation looked at the six case study countries to judge whether the World Bank Group has played a unique or complementary role in financing, coordination, or technical assistance and knowledge. Financing Role. In the three selected case study countries with low government capacity and a complex development partners’ network, the World Bank Group played an important financing role. In Liberia, the SNA identified two distinct financial flows, one through the government and the other directly to implementing nongovernmental organizations, mostly through the U.S. Agency for International Development (USAID). The World Bank Group is the largest contributor of total financial inflows through governments (about 25 percent or $55 million; see appendix G). The World Bank Independent Evaluation Group | World Bank Group 39 Group’s response to the Ebola virus outbreak provided the financial resources needed to control it at a critical time. In Bangladesh, for both sectorwide approaches (SWAPs), the World Bank was the largest single contributor among the development partners. World Bank–supported SWAPs sought to address critical weaknesses in public health services delivery that were inhibiting its quality and reach. Initiatives were implemented in time to meet most health-related MDGs. The World Bank’s leadership in the SWAP program encouraged participation and financing from other development partners. During the recent crisis in the Republic of Yemen, the World Bank canceled all undisbursed commitments in many sectors, but expanded its support to health services. The World Bank’s financing role is now critical because it remained one of the few development partners active in the country, along with WHO and the UN Children’s Fund (UNICEF). However, in the past few years in the Republic of Yemen, IFC has focused primarily on advisory support, but continued to monitor the investment portfolio (World Bank 2016). In three selected case study countries with high capacity, the World Bank has been asserting a more limited financing role, but the role intensifies when needs arise during crises. In the Philippines, the World Bank’s financial contribution to the country has been relatively small when compared with the entire health budget and tends to be sporadic. However, its financial support was important during 2013–16. In Romania, World Bank Group support to health services had a broader focus than that of other development partners. It was also larger in absolute terms and catalytic because it addressed a key government constraint, which was the limited capacity to absorb development funds from the European Commission. In Brazil, federal-level health projects were scaled back or canceled because the government preferred to proceed with its own resources or not proceed with some activities. The World Bank expanded support to health services at state and municipal levels, where demand for World Bank financial support was sustained. Coordination Role. The World Bank’s coordination role in countries with complex development partners’ network and low institutional capacity is important, but not always exploited to its full potential. The level and complexity of external aid and the capacity of the government to manage that aid represent two fundamental dimensions involving coordination. They represent key variables of the supply and demand dimensions of DAH, but more important, they define the need and motivation for managing partnerships and foster coordination (Leblanc and Beaulieu 2006, 33). In Liberia, coordination with other development partner was limited in the reconstruction period after the civil war, which produced duplication and fragmentation among donors’ activities. Collaboration with development partners was strengthened during the Ebola Emergency Response Projects, where about half of the resources were executed through UN agencies. The World Bank leadership promoted close coordination among partners in the post-Ebola recovery around the GFF and UHC2030 platform (formerly the International Health Partnership [IHP+]).20 However, there is still a need to improve communication among donors in the field. In Bangladesh, the World Bank administered a MDTF to support SWAPs implementation and provided fiduciary oversight on behalf of development partners (World Bank 2014d). In the Republic of Yemen, coordination before the crisis was at the project level, but there were no formal functioning mechanisms to engage in the 40 World Bank Group Support to Health Services | Chapter 2 country. During the 2016–17 crisis, a stronger collaboration with UNICEF and WHO was achieved because these agencies were also grant recipients and implementing entities. Although the World Bank’s coordination role in high-capacity countries is limited, there are opportunities for improvement. In Romania, the World Bank had close collaboration with the International Monetary Fund and the European Commission in defining the policy measures of the three development policy loans that supported health sector reforms. The European Investment Bank has been a partner of the World Bank Group for the entire period, cofinancing the World Bank investment project. More recently, however, the World Bank is financing the purchase of medical equipment for three new regional hospitals whose infrastructure financing is supported by the European Investment Bank, but coordination between the two entities is inadequate. Similarly, in Brazil, the World Bank and the Inter-American Development Bank deliver projects to improve health services in the same state or municipality but show limited coordination. In the Philippines, the World Bank collaboration with development partners appears more extensive. The World Bank, through the National Sector Support for the Health Reform Project that used government systems, allowed for coordination and cofinancing from other development partners (that is, the Asian Development Bank, the European Commission, and the German government). Technical Assistance and Knowledge. The World Bank Group is recognized as a leader in providing technical assistance and knowledge in select areas. Data from the 2014 Reform Efforts Survey (which asked a sample of 6,750 development policymakers and practitioners from 126 low- and middle-income countries to evaluate a set of development partner organizations with which they interacted directly) show that the World Bank’s policy advice in the health domain is valued, on average, more favorably than that provided by all the other development partners. However, the World Bank’s policy advice in the health domain was viewed less favorably than its policy advice provided in other policy domains, such as finance, credit and banking, public expenditure management, and macroeconomic management (see appendix H). World Bank policy advice in health is rated below that provided by Gavi, UNICEF, Global Fund, and the UN (WHO was not included in the sample), but better than all bilateral donors, the UN Development Programme, and regional banks. Country-level assessments confirmed this view. In Liberia, the institutional mapping and SNA indicated the World Bank Group is at the third layer when it comes to providing knowledge (see appendix G.). However, several actors recognized the World Bank’s expertise in human resource development and financial management. In Romania, only 10 percent of respondents to an IEG survey of primary health care professionals mentioned the World Bank as a technical leader in health, although 40 percent acknowledged the World Bank Group’s financing role. In low-capacity countries (Bangladesh, Liberia, and the Republic of Yemen), the technical assistance and knowledge services provided by the World Bank complemented the financing role. For example, the SWAP in Bangladesh envisioned some ambitious reforms that required World Bank technical assistance, but most were not implemented, and the program largely slipped into preserving the status quo, focusing on increasing financing and service delivery (World Bank 2014b). In Liberia, the World Bank’s technical assistance and knowledge products comprised human resource development analysis, fiduciary capacity assessments, and assessments of the socioeconomic consequences of the Ebola outbreak. Independent Evaluation Group | World Bank Group 41 Countries with higher capacity (Brazil, the Philippines, and Romania) had independently launched their health sector reform agendas. However, the World Bank Group has contributed to addressing specific, but important, constraints to health sector reform. In Brazil, the provision of technical assistance and knowledge services was critical to remaining engaged in the country. Analytic work produced by the World Bank set the stage for collaboration with other development partners.21 In the Philippines, the technical assistance provided by the World Bank was key to building the unified targeting system (Listahanan) that enhanced the performance of nationwide social programs—that is, the national CCT program Pantawid Pamilya, and the subsidized health insurance program PhilHealth (Acosta and Velarde 2015; Orbeta and Paqueo 2016; Bredenkamp et al. 2017). Similarly, in Romania, government officials and development partners acknowledged that the health sector functional review conducted by the World Bank was a valuable input to the National Health Strategic Plan, but other examples of technical assistance were rarely mentioned. IFC was instrumental in the development of PPPs that enabled leveraging private sector investments to improve emergency health services in the state of Bahia and dialysis services in Romania. 1 This is the total IFC funding balance. 2  he six theme codes were used in the analysis were Child Health (63), Other Communicable Diseases (64), HIV/AIDS T (88), Non-Communicable Diseases and Injury (89), Malaria (92), and Tuberculosis (93). 3 T  he disability-adjusted life year is a measure of overall disease burden, expressed as the number of years lost due to ill health, disability, or early death. It was developed in the 1990s as a way of comparing the overall health of different countries. 4  he Spearman’s correlation coefficient is 0.307. The Spearman’s correlation coefficient is a nonparametric measure T of rank correlation (statistical dependence between the ranking of two variables). It assesses how well the relationship between two variables can be described using a monotonic function. 5  he measure of the alignment for an individual health condition in each country is given by the difference in the T rankings provided by the global burden of disease study and the portfolio of World Bank Group support. A difference in the rankings of 0 or +-1 is considered an indicator of good alignment. A difference in the rankings of -2 or lower indicates under-prioritization by the World Bank; conversely, a value equal to 2 or better indicates that the condition is overprioritized by the World Bank. 6  Government capacity is measured by the World Bank Country Policy and Institutional Assessment of the health sector. 7  difference-in-differences estimator was used to compare the increase in the investment in population and A reproductive health in the 57 countries prioritized with the trends in the rest of the evaluation portfolio. 8 EG completes an independent review of all Implementation Completion and Results Reports and assigns its own I outcome rating, which is scored on a six-point scale (highly unsatisfactory, unsatisfactory, moderately unsatisfactory, moderately satisfactory, satisfactory, and highly satisfactory). The evaluation focused on a binary classification of projects (those that rated moderately satisfactory or better). 9 T  he 2009 Independent Evaluation Group evaluation considered that the complexity of the operation in the Africa Region was a key driver of the low performance. However, it is not possible to determine if the relative improvement in the share of health projects rated satisfactory is the exclusive result of project design more attuned country capacity. 42 World Bank Group Support to Health Services | Chapter 2 10 EG rates the efficacy of each project development objective (PDO; the PDO outcome rating) on a four-point scale: I high, substantial, modest, negligible. The evaluation focused on a binary classification of PDO rating: those that rated high or substantial (S+) and those that were not (that is, those rated modest or negligible). 11  ey performance indicators related to access were often associated with effective use of services, mostly with K maternal health and vaccination services, whereas other dimensions of access such as availability and affordability were less frequently used in monitoring and evaluation project frameworks. 12  he quality of the systematic review was determined applying the Supporting the Use of Research Evidence T (SURE) Collaboration checklist adapted by 3ie. Download the checklist at http://www.3ieimpact.org/media/filer_ public/2012/05/07/quality_appraisal_checklist_srdatabase.pdf. 13 P  DOs aiming at improving the quality of health services have 1.6 related indicators on average, which is an insufficient number to cover all the relevant dimensions. 14  f these 83 operations, 62 stand-alone projects focused primarily on avian influenza, other zoonoses, or pandemic O preparedness and control, and of those 62 avian influenza projects, 36 had project costs of at least $2 million. The 83 operations included $2,607 million in World Bank commitments to projects with a total cost of $7,978 million, but this includes several larger health or agriculture sector projects with only modest contributions to avian influenza or pandemic preparedness (World Bank 2014b). 15  he Ebola Emergency Response Project (P152359) committed $105 million to fight the outbreak. An additional T financing for $285 million was approved two months later in November 2014. 16 T  he initial $518 million commitment comprised $390 million from the global IDA Crisis Response Window; $110 million from IDA allocated to Guinea, Liberia and Sierra Leone; and $18 million reallocated from existing health projects. 17  ven if Guinea had established an emergency operations center and recognized the critical importance of public E information campaigns, it did not manage to set up a contact tracing system quickly enough to avoid the spread of the Ebola virus. 18 t is worth noting that in Nigeria, the World Bank Group supported the Partnership for Polio Eradication between April I 2003 and March 2011 through four projects with a cumulative commitment of $190.4 million. 19  his experience has been catalyzed by the Liberia Social Safety Nets Project approved in April 2016 aiming at T improving the country’s overall safety net. 20  ountries, international agencies, and bilateral donors joining UHC2030 make collective and individual commitments C to adhere to agreed-on aid effectiveness principles in the health sector by supporting country and government-led national health plans in a well-coordinated way. Dr. Bernice Dahn, Minister of Health and Social Welfare of Liberia, signed the UHC2030 Global Compact on April 13, 2016 at the World Bank headquarters in Washington, D.C., in the presence of the Director-General of the World Health Organization and the Senior Director for HNP GP. 21 S  takeholders recognized landmark studies on noncommunicable diseases, hospital performance efficiency, the aging population, the 20-year assessment of the unified health system, and the cancer care study, which was groundbreaking in using the “narrow” cancer issue to understand broader health system challenges. Independent Evaluation Group | World Bank Group 43 3 highlights World Bank 1 The World Bank Group plays multiple and distinct Group Support roles in global partnership programs. The most to Global common are as founding partner, governing Partnership partner, implementing partner, and trustee. Programs 2 The World Bank Group plays a central role in online interactions among global development actors, and it could potentially spread information effectively through the online network of relevant actors. 3 Although the World Bank Group’s more recent global engagements in health seem to be selected more carefully, the overall global partnership programs portfolio includes partnerships that appear to have lost their relevance or have overlapping mandates. x 44 T H E U R G E N CY of reaching the SDGs and universal health coverage by 2030 requires all actors of the global aid architecture to join forces and share resources, expertise, and knowledge. The World Bank Group with its global reach and stature is well positioned to foster collaboration among global players. Furthermore, in certain areas, health demonstrates global public goods characteristics: in the case of communicable diseases, for example, no single country can mount a response sufficient to protect the health of its population. Through GPPs in health—also referred to in the literature as multistakeholder programs—the World Bank Group aims to mobilize global donors around agreed health objectives, help channel funds and knowledge to client countries and leverage donors’ resources with its own, and address externalities as, for example, in the case of pandemics. GPPs are one of the key instruments the World Bank Group uses to tackle global health issues requiring urgent collective action at the global level. The Global Health Landscape SNA is used to visualize how organizations operating in the global health landscape interact online. In SNA terminology, the World Wide Web domains of each organization are the ‘‘nodes,” and the hyperlinks and co-citations among them are the “links.” Figure 3.1 depicts the co-citation network of organizations. The sizes of the nodes and links between nodes (or edges) are sizes relative to their “web score” value, which is the sum of co-citations. `The World Bank Group plays a central role in online interactions among global development actors, and it collaborates directly or through GPPs with almost all relevant actors. The first noticeable property of the network is the tendency of organizations to cluster with similar organizations. The World Bank has a central position in the network.1 IFC is less central to the network.2 The SNA was also used as a susceptible-infected model to simulate the capacity of a sample of development partners, such as the World Bank, WHO, UNICEF, and USAID, to spread information. The susceptible-infected model has been used successfully to track the spread of information in social networks (Kwak et al. 2010). The World Bank can reach almost the entire network, but WHO reaches saturation faster. The World Bank outperforms UNICEF and USAID, which manage to reach only a small portion of the network (see appendix G). Health GPPs have contributed to important achievements and enabled progress toward MDGs, but not all have lived up to their promise. Gavi and the Global Fund, for example, have played a major role in advancing public health science and in scaling up and strengthening evidence-based public health efforts in developing countries (Sachs and Schmidt-Traub 2017). The Stop TB Partnership gets credit for designing the Global Plans to Stop TB, developing innovative approaches to case Independent Evaluation Group | World Bank Group 45 FIGURE 3.1 The Network of Worldwide Web Co-Citation of Aid Organizations | Note: Each node in the graph is an international aid organization, whereas the color of a node reflects its type (red = bilateral donors that are members of the OECD’s Development Assistance Committee (DAC); blue =United Nations agencies; orange = other multilateral agencies; green = bilateral donors that are non-DAC members; purple = global partnership programs; yellow = international nongovernmental organizations; brown = private foundations). IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; WB = World Bank; WHO = World Health Organization; UNICEF = United Nations Children’s Fund; USAID = U.S. Agency for International Development. detection through TB REACH; and increasing the supply of tuberculosis commodities (Cambridge Economic Policy Associates 2015). IHP+’s assessment tools are used in several countries as a common framework for assessing the quality of national or disease-specific strategies, although the program did not have significant impact on improving donors’ coordination and use of national health strategies (Ofosu, Enemark, and Sonderstrup 2016). Moreover, it has been argued that GPPs may contribute to aid fragmentation (Walt and Buse 2000), and shortcomings have been identified regarding GPPs’ effectiveness, governance, strategic focus, and links to country programs (Bezanson and Isenman 2012; Buse and Tanaka 2011; Wescott and Wessal 2015). World Bank Group sector and corporate strategies call for increased selectivity in GPP engagements. The 2007 HNP strategy recognized GPPs’ opportunities, but also their limitations, and it called for increasing selectivity, improving strategic engagement, and reaching agreement with global partners on a collaborative division of labor for the benefit of client countries. The 2013 strategy committed to (i) deepening the World Bank Group’s role in promoting partnerships, (ii) ensuring strategic alignment of partnerships with the twin goals, and (iii) making provisions for partnerships to be adequately resourced and managed (World Bank 2013). The closure of the Development Grant Facility in 2012, which provided World Bank Group grants to GPPs, along with the new corporate strategy created an opportunity to revisit partnership engagements. 46 World Bank Group Support to Health Services | Chapter 3 The number of GPPs with World Bank Group engagement decreased from 34 in 2007 to 25 in 2016.3 Since 2007, seven GPPs ceased activities, the World Bank Group disengaged from 16 (which were generally not central to its priorities or were already mature), maintained its engagement with 11, and joined 14 new partnerships. About half of these programs provide country-level investments and technical assistance (for example, Gavi, Global Fund, and Polio Buy-Down) and some finance research and development in disease-specific areas (for example, the International AIDS Vaccine Initiative). Others help generate knowledge and provide a platform for advocacy, for example, in health systems strengthening and in maternal and child health and nutrition (for example, Scaling Up Nutrition and the Partnership for Maternal, Newborn & Child Health). The World Bank Group is also involved in several network partnerships to strengthen aid coordination and harmonize development partner practices at the country level; for example, the IHP+ and the coalition for Harnessing Non- State Sector for Better Health for the Poor. The World Bank Group’s more recent global engagements in health seem aligned with its sector and corporate strategies. During 2014–16, the World Bank Group engaged in seven new GPPs. The largest initiative is the GFF, which represents the financial platform of the UN’s Global Strategy for Women’s, Children’s, and Adolescents’ Health 2016–2030 (UN 2015). The World Bank mobilized international support to fight the Ebola outbreak in West Africa through the Ebola MDTF. In addition, it established the Pandemic Emergency Financing Facility, an innovative insurance mechanism that provides a surge of funds to enable a rapid and effective response to a large-scale disease outbreak. Another innovative initiative is the Tobacco Control program, which supports multisectoral World Bank teams to engage in country-level policy dialogue aiming at increasing taxes on tobacco products. Tobacco taxation is one of the most cost-effective measures to improve population health while also generating substantial domestic revenue (see Shibuya, Ciecierski, and Guindon 2003). The World Bank also engaged in global initiatives aimed at better measurement and knowledge sharing in service delivery, such as the Primary Health Care Performance Initiative and the Service Delivery Initiative. The evaluation shows that there is still room for better selectivity and alignment in the overall portfolio of GPPs. Some of the programs in the World Bank Group’s GPP portfolio have overlapping mandates and objectives because of changes in the global health landscape. However, it is rare that an existing GPP closes or merges when new, larger programs with similar objectives are created, even if they have common stakeholders. Some GPPs may also have become less relevant to the World Bank Group because of shifts in the corporate and sector priorities. For example, Stop TB Partnership’s mission, which had achieved notable results in the past, has some overlaps with that of the Global Fund. The WHO-housed Special Programme of Research, Development, and Research Training in Human Reproduction was highly relevant in early 1990s as one of the few engaged partnerships in reproductive health research in low-income countries. However, it is worth exploring to what extent the research that the program conducts on a global scale to improve sexual and reproductive health informs the World Bank Group’s support in this area. The Medicines for Malaria Venture and International AIDS Vaccine Initiative followed the 1997 HNP strategy, which envisioned that the World Bank would cooperate in the production of health-related goods, including malaria, HIV, and tuberculosis drugs. However, the 2007 HNP strategy has shifted the focus to strengthening health systems. Independent Evaluation Group | World Bank Group 47 Role and Effectiveness of World Bank Group Engagement in GPPs The World Bank Group plays different roles in GPPs—founding and governance partner, program host, trustee of donor funds, development partner at country and global levels, and provider of complementary technical assistance and investments. The World Bank Group has become a founding partner of many important programs supporting global and national public goods, often in collaboration with specialized UN agencies, such as WHO and UNICEF (see appendix C). The two largest Financial Intermediary Funds supporting health services—the Global Fund and Gavi—were established in the early 2000s with strong support from the World Bank Group. The World Bank remains an effective trustee for both programs (World Bank 2012, 2014c). The World Bank also helped set up and manage two innovative financial vehicles—the International Finance Facility for Immunization and the Advanced Market Commitments—which give Gavi significant and predictable resource flows for immunization. The World Bank continues today as the financial manager for the International Finance Facility for Immunization. Jointly with WHO, the World Bank has been key in transforming the Partnership for Maternal, Newborn & Child Health into an umbrella entity that improves aid coordination. The World Bank has also collaborated with WHO to transform the IHP+ (now UHC2030), into a multistakeholder platform to promote global- and country-level collaboration on health systems strengthening. The intensity of the World Bank’s participation in the governance of some GPPs has changed to reflect the evolving health architecture and landscape. The World Bank is represented in the governing body (at board or committee levels) of 22 of the 25 health GPPs.4 Participation in governance allows it to reach a wide range of stakeholders. Often being constituency based, the governing bodies of the partnership programs are inclusive by nature. They include not only traditional donors and client governments, but also other multilateral banks, UN entities, and nonstate actors such as civil society organizations, the private sector, private foundations, and various beneficiary and interest groups representing a broad range of constituencies. In recent years, the World Bank’s participation has weakened in some formerly Development Grant Facility–funded programs, as the interviews with the World Bank representatives in these programs indicated. This is partly because these programs provide little additionality, and partly because of reduced funding to participate in their board meetings. However, the World Bank continues to stay involved in their governance, often at the request of other partners and donors. The World Bank Group provides the secretariat and hosts about half of the GPPs with which it engages. The remaining are either housed in specialized UN agencies, such as WHO, UNICEF, and the UN Population Fund, or are set up as independent entities. The in-house GPPs, along with the managed MDTFs, reflect the World Bank’s comparative strengths—innovative financing and results (Polio Buy-Down and HRITF) and data for evidence-based decision making in health services (Service Delivery Initiative and Strategic Impact Evaluation Fund)—as well as strong complementarities with the World Bank Group’s own resources (for example, HIA, HRITF, and GFF). The World Bank Group’s implementation role in GPPs that are located outside the World Bank Group is limited; this is an untapped opportunity that could further enhance the benefits from such partnerships. Aside from direct cofinancing with development partners, the World Bank also has 48 World Bank Group Support to Health Services | Chapter 3 an implementation role in some external GPPs that provide country-level investments and technical assistance. These programs include the Power of Nutrition, the Sahel Women’s Empowerment and Demographic Dividend, the Joint UN Programme on HIV/AIDS, and Gavi. The new implementing partner arrangement with Gavi since 2015 is another example of the World Bank using its operational role for leveraging resources and aligning priorities. Along this line, the expansion of its operational role in major GPPs that are not housed in the World Bank Group is another venue to enhance the benefits from the partnerships by leveraging resources, sharing solutions, and aligning donor priorities at the country level. The country-level collaboration with the two largest Financial Intermediary Funds—Gavi and the Global Fund—has improved, but untapped opportunities remain. Previous assessment had found that despite close engagement at the governance level and through various global platforms, collaboration at the country level was not systematic. More clarity on the division of labor and expectations would have helped the World Bank’s country teams collaborate more effectively with the Global Fund and Gavi (Macro International 2007; World Bank 2011, 2012, 2014c). The Health Systems Funding Platform, established in 2009 to harmonize World Bank, Gavi, and the Global Fund support mechanisms and to align their results frameworks and M&E plans, fell short of its expectation because of excessive focus on governance aspects (Brown, Sen, and Decoster 2013; England 2009). Since 2015, the country-level partnership with Gavi has been revamped. Gavi contributes to an MDTF that supports analytical work and policy dialogue in nine countries where Gavi financing is ending. Country-level collaboration with the Global Fund is now improving as well. One such recent example is the cofinancing of a World Bank health project by the Global Fund and Gavi in the Democratic Republic of Congo. The Global Fund’s new 2017–22 strategy’s focus on sustainability and transition is expected to provide more venues for collaboration with the World Bank to support the countries transitioning out of Global Fund support. 1 The World Bank scores the highest “betweenness centrality” value among all organizations considered in the social  network analysis, which indicates the fraction of paths in the network that would get longer—or be disconnected—if the node were removed from the network. 2  It is worth noting that the Multilateral Investment Guarantee Agency’s centrality does not appear related to the number of its operation in the health sector, which is just one project so far. 3 The 2007 Health, Nutrition, and Population strategy (World Bank 2007b, 179) lists 33 global health partnerships,  initiatives, and programs. If we also consider the Polio Buy-Down Program, in which the World Bank has participated since its inception in 2002, the total number is 34. 4  In general, the higher the level of representation on the governing body, the more importance is given to the program. In nine of these programs’ boards, the World Bank is represented at the highest level—vice president (Scaling p Nutrition Movement), senior director (Gavi, International Health Partnership, and the Partnership for Maternal, Newborn & Child Health, Middle East and North Africa chief economist (Service Delivery Initiative), director or senior adviser level (TDR; United Nations Programme on HIV/AIDS; The Global Fund to Fight AIDS, Tuberculosis and Malaria, Global Financing Facility). In the rest of the programs, lead or senior health specialists represent the World Bank. Independent Evaluation Group | World Bank Group 49 4 The Drivers of Universal Health Coverage O U TC O M E R AT I N G S of World Bank health projects have improved over time, as did the quality of their M&E framework. However, the evaluation identifies opportunities to improve Conclusions and specific aspects of World Bank Group–financed health services Recommendations projects, particularly those related to quality and equity outcomes. Recent methodological changes introduced in World Bank project financing, such as the explicit representation of the theory of change in the project appraisal document and the introduction of PDO-level efficacy ratings in the implementation completion report, represent opportunities for improving the M&E of World Bank–financed projects supporting health services. IFC projects are also hindered by structural limitations in the design of their M&E frameworks, which do not track health services–specific outcomes. In this regard, the recent IFC efforts to develop a more comprehensive impact framework to measure and articulate the social and public impact of its investments could also improve the M&E framework of IFC projects supporting health services. Access. The World Bank Group showed substantial contributions to improving access to health care services. The World Bank health portfolio focuses strongly on primary care, control and prevention activities, and maternal and child health care services—consistent with the priorities of less- developed countries. The IFC investments portfolio concentrates more on the provision of secondary care health services and pharmaceuticals in lower-middle- and upper-middle-income countries. The use of financial incentives, such as CCT and PBF, has been effective in the short term for simple, distinct, well- defined behavioral change on the demand and supply side of x 50 access to health services. The evaluation emphasizes the need to continue to put strong emphasis on a balanced approach that comprises relevant demand-side, system-wide, and supply-side interventions and to foster beneficiary participation to improve accountability, fight corruption, and ensure that services benefit the poor. Quality. World Bank Group–financed project objectives show greater emphasis over time on improving the quality of health services, but limited capacity to monitor all the relevant aspects of the desired quality improvement. IFC quality improvement objectives and related indicators focus on the narrow aspect of structural quality. World Bank project financing is only rarely able to monitor all the relevant dimensions—structure, process, and outcomes—and the underlying links or theory of change. Some World Bank–financed projects, such as those adopting PBF, present stronger M&E frameworks for quality improvement. The World Bank has produced analytical works and is engaged in global initiatives aiming at improving the M&E of quality of health services (see the Primary Health Care Performance Initiative; Das, Hammer, and Leonard 2008; Smith and Nguyen 2013). This indicates the opportunity to improve the M&E framework of World Bank project financing and IFC investments seeking to improve quality of health services. Equity. Most World Bank projects identify the specific population groups with coverage gaps who are expected to benefit from interventions (often the poor). However, even when the beneficiaries are identified, the projects’ M&E rarely measure improvements in relative terms (that is, comparing beneficiaries with nonbeneficiaries), thus the distributional impacts (as well as the contribution to universal health coverage and shared prosperity) are seldom measured. IFC focuses on large markets and networks, which suggests the potential for systemic impact, but also in this case, the distributional impact of IFC projects is rarely specified in project interventions. Health Systems Strengthening. The presence of explicit PDOs aimed at strengthening the health system has been decreasing over time, even if health systems–strengthening activities are identified in about 90 percent of World Bank project financing. IFC rarely aims at health systems strengthening through its investments, but it does through its advisory services. In general, health systems– strengthening objectives are more likely to be achieved if the scope is well defined, and it is an area where the World Bank has accumulated significant experience. Health Outcomes. The presence of explicit PDOs aiming at improving health outcomes has declined over time among World Bank–financed projects. The decline is a deliberate effort of HNP GP to focus on PDO outcomes that are measurable and attributable to the project’s specific interventions. Only half of the PDOs are rated S+. The limited success is partially the result of using indicators (for example, mortality rate) that are not sufficiently sensitive and that are subject to attribution challenges. The literature on clinical indicators for outcome aspects of health care quality improvement offers examples of indicators that are sufficiently sensitive and attributable to the intervention, and thus strengthen the M&E framework of World Bank Group–supported projects. It should be noted that the wider use of strong and contextualized clinical indicators on the structural, process, and outcome aspects of quality might not require explicit health outcomes PDOs. Independent Evaluation Group | World Bank Group 51 Country-Level Support to Health Services Relevance, Synergies, Public-Private Interactions, Selected Delivery Mechanisms, and Response to Pandemics The mix of World Bank Group support to health services is generally aligned with the specific country needs. The analysis of the evaluation portfolio shows that various factors, including the development level, FCV situations, and specific needs and priorities, shape World Bank Group support. For example, World Bank financial support to low-income countries is more substantial as a share of total health expenditure than in lower-middle- and upper-middle-income countries. However, the World Bank Group has delivered substantial IBRD support to lower-middle- and upper-middle- income countries during crises as well as an increasing number of World Bank ASA and most IFC investments. The World Bank Group is usually recognized as a leader in the provision of technical assistance and knowledge in selected technical areas. This is the result of triangulating evaluative evidence derived from different methods. The 2014 Reform Efforts Survey suggests that, on average, the World Bank’s policy advice in the health domain is valued more favorably than that provided by other development partners (see appendix H). Case studies in selected countries confirm that the World Bank Group is perceived as the lead agency in the provision of technical assistance and knowledge in selected technical areas. For example, the World Bank has supported the use of incentives (that is, CCT and PBF) and health PPPs in several countries. In addition, the World Bank is usually able to articulate correctly the support to health services delivered by different GPs. The evaluation identifies missed opportunities in integrating World Bank and IFC support to health services. The World Bank Group’s strategy to better assist governments with integrating public and private health sectors within their broader health care systems was articulated a decade ago and restated in 2015 (World Bank 2007b, 2015). However, to date, follow-through has not matched intent. The evaluation found, from the selected countries, missed opportunities in Brazil and in Romania, where the PPPs went ahead with little coordination with the World Bank. A recent IEG analysis found many opportunities for better synergies between IFC and the World Bank in supporting health PPPs (World Bank 2016a). The experience of HIA shows that coordination between the World Bank’s upstream policy support and IFC support to small and medium health service providers remained weak until all HIA activities were transferred to the same management. The World Bank’s support to articulate private service provision and public financing is still limited, and IFC investee companies face challenges in integrating with public financing to improve access for the underserved. The main reasons are limited availability of public resources and capabilities; underdeveloped private markets for health services, including difficulties in making true price comparison between the public and private sectors; and inadequate regulation, including enforcement. The World Bank Group has taken steps to address this through the creation of a new unit to lead the implementation of the private health sector roadmap, but challenges remain due to the limited expertise and resources to support country teams and governments. The cascade approach to mobilize finance for development offers an opportunity to enhance public-private synergies. 52 World Bank Group Support to Health Services | Chapter 4 The evaluation identified opportunities for improving coordination between the World Bank Group and other development partners. Coordinating is more important in countries with complex development partner networks and low capacity. In Bangladesh, the World Bank’s leading role in the health SWAP has encouraged participation and financing from other development partners. In Liberia, World Bank support to health services was not well coordinated with other development partners in the 2007–13 period, but collaboration with development partners improved during the fight against the Ebola virus and in the subsequent period. However, the evaluation still identifies opportunities for enhancing communication among partners in the field. Similarly, World Bank support to health services in the Republic of Yemen during the recent crisis has evolved toward a much closer collaboration between the World Bank and UN specialized agencies. Missed opportunities were found in the high-capacity countries of Brazil and Romania, where donors work in the same space but with little coordination. Finally, country-level collaboration between the World Bank and with the two largest Financial Intermediary Funds—Gavi and the Global Fund—has improved, but opportunities for even closer collaboration remain. The World Bank Group performance in pandemic preparedness and response has improved through successive pandemic outbreaks, but World Bank Group support to pandemic risk management, mitigation, and preparedness is not fully mainstreamed into operations.1 Under the GPAI, the World Bank supported pandemic preparedness and response efforts in 63 countries during 2006–13, but it failed to sustain efforts (World Bank 2014b). As a central member of the global coalition that fought the Ebola virus outbreak in West Africa, the World Bank quickly mobilized the financial resources required to fight the spread of infection, restore basic health services, and reactivate the economy. A key lesson emerged: capable health systems are necessary to mount a successful response. They require adequately staffed health services, a supply of essential personal protective equipment, capacities for laboratory diagnosis, clinical management, and surveillance for quick diagnosis and rapid contact tracing. The Pandemic Emergency Financing Facility, approved in May 2016, may accelerate the release of funds to respond to future outbreaks. However, preparedness is the first line of defense. Country health systems’ pandemic risk management capacity should be strengthened. The commitment made under IDA18 to support about 25 IDA countries in developing pandemic preparedness plans and frameworks for health emergency preparedness, response, and recovery represents an opportunity to leverage World Bank experience in pandemic preparedness and response with IBRD financing (World Bank 2017a, 48). World Bank project financing adopting PBF mechanisms performs better than the World Bank portfolio in improving access, quality, and health outcomes. The World Bank contributed to expanding the knowledge and use of incentives to encourage health providers to improve access to and quality of health services, mainly through the HRITF. The sustainability and scale-up of the pilots were successful in some instances, but are still a challenge in many countries. Efforts should be directed at continuing to generate more evidence on PBF and at ensuring good practices are scaled up nationally. This would also help offset the use of alternative PBF programs across projects supported by various development partners. Independent Evaluation Group | World Bank Group 53 World Bank projects comprising CCT outperform the evaluation portfolio in improving access, health outcomes, and strengthening health systems. The World Bank’s impact evaluations of CCT programs primarily measure effects on access and health outcomes, which are already well studied. However, few have assessed the distributional impact of CCTs with respect to access, use, and financing of health services. This may represent a missed opportunity. The World Bank Group’s Role in GPPs World Bank Group participation in GPPs has contributed to aligning partner objectives toward health MDGs and SDGs and leveraging resources, but there are opportunities for improvement. The World Bank Group has often participated in GPPs at the request of other partners who value its convening capacity to align partners with shared objectives (usually MDGs and SDGs), its strong ability to manage and execute trust funds, and country presence. Although engagements in GPPs have become more selective and aligned with sector and corporate strategies over the evaluation period, the additionality of some partnerships remains weak and some mandates overlap. The absence of a strategy that defines World Bank Group priorities in its global-level engagement with health GPPs does not allow the assessment of the value-added of each partnership engagement and the worth the World Bank Group brings to them. A strategic review could define clear selectivity criteria for current and future engagements, delineate the division of labor among partners, clarify expectations, and ensure adequate resourcing for representation and participation in the GPPs’ governance. Recommendations Recommendation 1. Improve measurement of the quality of health services and the distributional effects of health services projects. The M&E framework of World Bank Group projects should include (i) appropriate indicators of the relevant dimensions of health service quality— structure, process, and outcomes, and (ii) the measurement of improvements of beneficiaries relative to nonbeneficiaries. Recommendation 2. Strengthen World Bank and IFC synergy to support public-private interactions in client countries to contribute to SDG3 and universal health coverage. (i) For the World Bank, strengthen the planning, regulatory, and accountability arrangements for public-private interactions working with IFC (ii) For IFC, crowd-in public financing for privately delivered services working with the World Bank. The World Bank Group’s newly launched cascade approach, aimed at mobilizing finance for development by focusing on upstream reforms where necessary to address market failures and other constraints to private sector investment, can be applied to achieve greater synergies between the public and private sector. Recommendation 3. To develop sustainable capacity to address pandemics, systematically integrate, in World Bank Group–financed projects and ASA, awareness and preparedness plans and governance frameworks for pandemic control with the client country’s own health 54 World Bank Group Support to Health Services | Chapter 4 system. Building on the commitment made under IDA18 to support health emergency preparedness, response, and recovery, the management of the World Bank Group institutions could seek to ensure that World Bank’s project financing and ASA are not one-off responses outside the client country’s health system. Recommendation 4. Enhance the strategic alignment and selectivity of World Bank Group engagement in ongoing and future GPPs. A strategic review should apply clear selectivity criteria that reflect the World Bank Group’s comparative advantage and the broader global development agenda. It can inform the selectivity and relevance of ongoing and future GPPs, and a more effective use of resources needed for engaging in partnerships. 1 It is worth mentioning, as a first step in this direction, the institutionalization of the Contingent Emergency Response  Components as part of Health, Nutrition, and Population World Bank investment project financing. Independent Evaluation Group | World Bank Group 55 references Acosta, P. A., and R. Velarde. 2015. “Sa Pantawid, Malapit nang Makatawid!” (With Pantawid, we are closer to getting out of poverty).” Philippine Social Protection Note No. 8 (October 2015), World Bank, Washington, DC. Bezanson, Keith, and Paul Isenman. 2012. “Governance of New Global Partnerships: Challenges, Weaknesses, and Lessons.” CGD Policy Paper 014, Center for Global Development, Washington, DC. Bredenkamp, C., J. Capuno, A. Kraft, L. Paco, S. Quimbo, and C. A. Tan Jr. 2017. “Expansion of Health Insur- ance in the Philippines: Evidence from Panel Data.” Health, Nutrition, and Population (HNP) Discussion Paper, World Bank, Washington, DC. Briand, S., E. Bertherat, P. Cox, P. Formenty, M. P. Kieny, J. K. Myhre, C. Roth, N. Shindo, and C. Dye. 2014. “The International Ebola Emergency.” New England Journal of Medicine 371 (13): 1180–1183. Brown, S. S., K. Sen, and K. Decoster. 2013. “The Health Systems Funding Platform and World Bank Legacy: The Gap Between Rhetoric and Reality.” Globalization and Health 9 (1): 9. Buse, K., and S. Tanaka. 2011. “Global Public–Private Health Partnerships: Lessons Learned from Ten Years of Experience and Evaluation.” International Dental Journal 61 (s2): 2–10. Cambridge Economic Policy Associates. 2015. Stop TB Partnership Evaluation: External Report. Geneva: Stop TB Partnership. Das, J., J. Hammer, and K. Leonard. 2008. “The Quality of Medical Advice in Low-Income Countries.” Journal of Economic Perspectives 22 (2): 93–114. Das, J., A. Holla, A. Mohpal, and K. Muralidharan. 2016. “Quality and Accountability in Health Care Delivery: Audit-Study Evidence from Primary Care in India.” American Economic Review, 106 (12): 3765–3799. England, R. 2009. “The GAVI, Global Fund, and World Bank Joint Funding Platform.” The Lancet 374 (9701): 1595–1596. Evans, D. B., J. Hsu, and T. Boerma. 2013. “Universal Health Coverage and Universal Access.” Bulletin of the World Health Organization 91 (8): 546–546A. Fair, M. 2008. “From Population Lending to HNP Results: The Evolution of the World Bank’s Strategies in Health, Nutrition and Population.” Independent Evaluation Group Working Paper 2008/3, World Bank, Washington, DC. Fasina, F. O., A. Shittu, D. Lazarus, O. Tomori, L. Simonsen, C. Viboud, and G. Chowell. 2014. “Transmission Dy- namics and Control of Ebola Virus Disease Outbreak in Nigeria, July to September 2014.” Euro Surveillance 19 (40): 20920. Filmer, D., J. S. Hammer, and L. H. Pritchett. 2002. “Weak Links in the Chain II: A Prescription for Health Policy in Poor Countries.” World Bank Research Observer, 17 (1): 47–66. Flanagan, A. E., and J. C. Tanner. 2016. “A Framework for Evaluating Behavior Change in International Develop- ment Operations.” Independent Evaluation Group Working Paper 2016/2, World Bank, Washington, DC. Gergen, J., E. Josephson, M. Coe, S. Ski, S. Madhavan, and S. Bauhoffe. 2017. “Quality of Care in Perfor- mance-Based Financing: How It Is Incorporated in 32 Programs Across 28 Countries.” Global Health: Science and Practice 5 (1): 90–107. Haar, R. J., and L. S. Rubenstein. 2012. “Health in Postconflict and Fragile States.” Special Report 301, US Insti- tute of Peace, Washington, DC. 56 World Bank Group Support to Health Services | References Hammer, J., Y. Aiyar, S. and Samji. 2007. “Understanding Government Failure in Public Health Services.” Eco- nomic and Political Weekly 42 (40): 4049–4057. Kaiser, K., C. Bredenkamp, and R. Iglesias. 2016. Sin Tax Reform in the Philippines: Transforming Public Finance, Health, and Governance for More Inclusive Development. Washington, DC: World Bank. Kieny, M. P., D. B. Evans, G. Schmets, and S. Kadandale. 2014. “Health-System Resilience: Reflections on the Ebola Crisis in Western Africa.” Bulletin of the World Health Organization 92 (12): 850. Klitgaard, R., 1988. Controlling Corruption. Berkeley: University of California Press. Kutzin, J. 2008. “Health Financing Policy: A Guide for Decision Makers.” Health Financing Policy Paper 2008/1, WHO Regional Office for Europe, Copenhagen, Denmark. Kutzin, J., and S. P. Sparkes. 2016. “Health Systems Strengthening, Universal Health Coverage, Health Security and Resilience.” Bulletin of the World Health Organization, 94 (1): 2. Kwak, H., C. Lee, H. Park, and S. Moon. 2010. “What Is Twitter: A Social Network or a News Media?” In Pro- ceedings of the 19th International Conference on World Wide Web, edited by M. Rappa, P. Jones, J. Freire, and S. Chakrabarti, 591–600. New York: Association for Computing Machinery. Leblanc, R. N., and P. Beaulieu. 2006. “Evaluation of Coordination and Complementarity of European Assistance to Local Development: with Reference to the 3C Principles of the Maastricht Treaty.” Joint Evaluation 2006:7, the European Commission, Brussels. Lewis, M. 2006. “Governance and Corruption in Public Health Care Systems.” CGD Working Paper 78, Center for Global Development, Washington, DC. Lohr, K. N., ed. 1990. Medicare: A Strategy for Quality Assurance, Volumes I and II. Washington, DC: National Academies Press. Mackintosh, M., A. Channon, A. Karan, S. Selvaraj, E. Cavagnero, and H. Zhao. 2016. “What Is the Private Sec- tor? Understanding Private Provision in the Health Systems of Low-Income and Middle-Income Countries.” The Lancet 388 (10044): 596–605. Macro International. 2007. Evaluation of the Organizational Effectiveness and Efficiency of the Global Fund to Fight AIDS, Tuberculosis, and Malaria: Results from Study Area 1 of the Five-Year Evaluation. Geneva: The Global Fund. Mainz, J., 2003. “Defining and Classifying Clinical Indicators for Quality Improvement.” International Journal for Quality in Health Care 15 (6): 523–530. McPake, B., and K. Hanson. 2016. “Managing the Public-Private Mix to Achieve Universal Health Coverage.” The Lancet 388 (10044): 622–630. Miake-Lye, I. M., S. Hempel, R. Shanman, and P. G. Shekelle. 2016. “What Is an Evidence Map? A Systematic Review of Published Evidence Maps and Their Definitions, Methods, and Products.” Systematic Reviews 5 (1): 28. Morgan, R., T. Ensor, and H. Waters. 2016. “Performance of Private Sector Health Care: Implications for Universal Health Coverage.” The Lancet 388 (10044): 606–612. Murray, C. J., and J. Frenk. 2000. “A Framework for Assessing the Performance of Health Systems.” Bulletin of the World Health Organization 78 (6): 717–731. Independent Evaluation Group | World Bank Group 57 Musgrove, P. 1996. “Public and Private Roles in Health: Theory and Financing Patterns.” World Bank Discussion Paper WDP339, World Bank, Washington, DC. ———. 2010. Rewards for Good Performance or Results: A Short Glossary. Washington, DC: World Bank. Ofosu A., U. Enemark, and E. Sonderstrup. 2016. Rapid Independent Review of the IHP+. United Kingdom and Denmark: International Health Partnership. Orbeta Jr., A. C., and V. B. Paqueo. 2016. “Pantawid Pamilya Pilipino Program: Boon or Bane?” Discussion Paper Serices No. 2016-56, Philippine Institute for Development Studies, Manila. Pita Barros, P., and L. Siciliani. 2011. “Public and Private Sector Interface.” In Handbook of Health Economics, vol. 2, edited by Mark V. Pauly, Thomas G. Mcguire and Pedro P. Barros, 927–1001. New York: Elsevier. Prahalad, C. K. 2006. The Fortune at the Bottom of the Pyramid: Eradicating Poverty through Profits. Upper Saddle River, NJ: Wharton School Publishing. Rosenthal, G., and W. Newbrander. 1996. “Public Policy and Private Sector Provision of Health Services.” Inter- national Journal of Health Planning and Management 11 (3): 203–216. Sachs, J. D., and G. Schmidt-Traub. 2017. “Global Fund Lessons for Sustainable Development Goals.” Science 356 (6333): 32–33. Savedoff, W. D. 2006. “The Causes of Corruption in the Health Sector: A Focus on Health Care Systems.” In Global Corruption Report 2006, edited by Jana Kotalik and Diana Rodriguez, 3–24. London: Transparency International. Shengelia, B., A. Tandon, O.B. Adams, and C. J. Murray. 2005. “Access, Use, Quality, And Effective Coverage: An Integrated Conceptual Framework and Measurement Strategy.” Social Science and Medicine 61 (1): 97–109. Shibuya, K., C. Ciecierski, E. Guindon, D. W. Bettcher, D.B. Evans, and C. J. Murray. 2003. “WHO Framework Convention on Tobacco Control: Development of an Evidence-Based Global Public Health Treaty.” British Medical Journal 327 (7407): 154. Smith O., and S.N. Nguyen. 2013. Getting Better: Improving Health System Results in Eastern Europe and Cen- tral Asia. Washington, DC: World Bank. Snilstveit, B., M. Vojtkova, A. Bhavsar, and M. Gaarder. 2013. “Evidence Gap Maps: A Tool for Promoting Evi- dence-Informed Policy and Prioritizing Future Research.” Policy Research Working Paper 6725, World Bank, Washington, DC. Tung, E., and S. Bennett. 2014. “Private Sector, For-Profit Health Providers in Low and Middle Income Countries: Can They Reach the Poor at Scale?” Globalization and Health 10 (1): 52. UN (United Nations). 2015. Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–2030. New York: United Nations. UNHCR (United Nations High Commissioner for Refugees). 2011. “Action against Sexual and Gender-Based Violence: An Updated Strategy.” UNHCR Division of International Protection. Geneva: UNHCR. Vian, T. 2008. “Review of Corruption in the Health Sector: Theory, Methods and Interventions.” Health Policy and Planning 23 (2): 83–94. Walt, G. and K. Buse. 2000. “Partnership and Fragmentation in International Health: Threat or Opportunity?” Tropical Medicine and International Health 5 (7): 467–471. 58 World Bank Group Support to Health Services | References Wescott, C. G., and A. Wessal. 2015. “How Effective are International Development Partnerships?” Paper pre- sented at the International Public Management Network conference, Budapest, Hungary, September 2014. WHO (World Health Organization). 2015a. World Health Statistics 2015. Geneva: World Health Organization ———. 2015b. “Successful Ebola Responses in Nigeria, Senegal, and Mali.” World Health Organization. http:// www.who.int/csr/disease/ebola/one-year-report/nigeria/en/. WHO (World Health Organization) and World Bank Group. 2015. Tracking Universal Health Coverage: First Global Monitoring Report. Geneva: WHO and World Bank Group. Witter, S., A. Fretheim, F. L. Kessy, and A. K. Lindahl. 2012. “Paying for Performance to Improve the Delivery of Health Interventions in Low-and Middle-Income Countries.” Cochrane Database of Systematic Reviews 2012 (2): CD007899. World Bank. 2003. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. ———. 2005. World Development Report 2006: Equity and Development. Washington, DC: World Bank. ———. 2006. “Bangladesh Fourth Population and Health Project and Health and Population Program Project.” Independent Evaluation Group. Project Performance Assessment Report 36564. Washington, DC: World Bank Group. ———. 2007a. Guidelines for Global and Regional Program Reviews (GRPRs). Independent Evaluation Group. Washington, DC: World Bank. ———. 2007b. Healthy Development: The World Bank Strategy for Health, Nutrition, and Population Results. Washington, DC: World Bank Group. ———. 2007c. “International Development Association and African Development Fund Joint Interim Strategy Note for the Republic of Liberia.” Report 39821. Washington, DC: World Bank and African Development Bank. ———. 2007d. Sourcebook for Evaluating Global and Regional Partnership Programs Indicative Principles and Standards. Independent Evaluation Group. Washington, DC: World Bank. ———. 2009. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition and Population. An Eval- uation of World Bank Group Support Since 1997. Independent Evaluation Group. Washington, DC: World Bank. ———. 2010. Better Health for Women and Families: The World Bank’s Reproductive Health Action Plan 2010– 2015. Washington, DC: World Bank. ———. 2011. The World Bank’s Involvement in Global and Regional Partnership Programs: An Independent As- sessment. Independent Evaluation Group. Washington, DC: World Bank. ———. 2012. The Global Fund to Fight AIDS Tuberculosis and Malaria, and the World Bank’s Engagement with the Global Fund. Independent Evaluation Group. Washington, DC: World Bank ———. 2013. World Bank Group Strategy. Washington, DC: World Bank Group. ———. 2014a. World Bank Support to Health Financing. Independent Evaluation Group. Washington, DC: World Bank Group. ———. 2014b. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006–13. Independent Evaluation Group. Washington, DC: World Bank Group. Independent Evaluation Group | World Bank Group 59 ———. 2014c. The World Bank’s Partnership with the GAVI Alliance. Independent Evaluation Group. Washington, DC: World Bank ———. 2014d. “People’s Republic of Bangladesh—Health, Nutrition, and Population Sector Program.” Indepen- dent Evaluation Group, Project Performance Assessment Report 88078. Washington, DC: World Bank Group. ———. 2015. Communications for Financial Intermediary Funds. World Bank Guidance Note DFi2.02-GUID.05. Washington, DC: World Bank ———. 2016a. Public-Private Partnerships in Health World Bank Group Engagement in Health PPPs. Learning Product, Independent Evaluation Group. Washington, DC: World Bank Group. ———. 2016b. Updates to the Priority Directions for the Health, Nutrition, and Population Global Practice 2016– 2020. Washington, DC: World Bank. ———. 2016c. Using Evaluation Evidence to Improve the Effectiveness of Partnership Programs. Independent Evaluation Group. Washington, DC: World Bank Group. ———. 2017a. Report from the Executive Directors of the International Development Association to the Board of Governors Additions to IDA Resources: Eighteenth Replenishment Towards 2030: Investing in Growth, Resilience and Opportunity. Washington, DC: World Bank. ———. 2017b. The Toll of War and Economic and Social Consequences of the Conflict in Syria. World Bank: Washington, DC. ———. 2017c. “Panama—Improving Basic Health, Equity, Opportunities among Indigenous and Poor Rural Com- munities.” Independent Evaluation Group, Project Performance Assessment Report 110617. Washington, DC: World Bank Group. ———. 2017d. Leveraging the Private Sector in IDA Countries: An Integrated World Bank Group Approach. Wash- ington, DC: World Bank. ———. 2017e. “Forward Look: A Vision for the World Bank Group in 2030—Progress and Challenges Report.” Washington, DC: World Bank Group. World Bank and IFC (International Finance Corporation). 2015. Technical Briefing to the Board: Joint World Bank Group Approach to Harnessing the Private Sector in Health. Washington, DC: World Bank Group. 60 World Bank Group Support to Health Services | References A PPE NDI X E S World Bank Group Support to Health Services Achievements and Challenges AN INDEPENDENT EVALUATION Appendix A. Methodological Approach Evaluation Questions The overarching question of the evaluation is, What are the roles and contributions of the World Bank Group in support of health services, and what can be done to enhance them? In turn, this inspired four specific questions that are answered by assessing contributions to health services at global, country, and project levels. ƒƒ What has been the nature, extent, and evolution of support to health services in the last 10 years? ƒƒ How relevant has World Bank Group support to health services been to the main health needs and priorities? ƒƒ To what extent has World Bank Group support effectively contributed to the achieve- ment of its goals? ƒƒ What has been the role of the World Bank Group in global and country levels partner- ships supporting health services? Overarching Principles Key principles of the evaluation design are the theory-based approach and the use of mixed meth- ods. First, the development of the overall intervention’s logic and of intervention-specific theo- ries of change allows identification of the desired improved outputs and outcomes. Second, the mixed-methods approach allows triangulation of results from a range of methods for data collection and analyses, which strengthens the overall robustness of the evaluation conclusions. Overview of Methodological Design Table A.1 provides a brief description of the key evaluation components. More details on most promi- nent components are provided in the following section. Figure A.1 depicts the overall methodology design of the evaluation showing how data collection methods feed into different methods of analysis articulating with each other to provide the necessary evidence to answer the four evaluation questions. 63 Table A.1. Evaluation Components: Data Collection Methods and Analyses Inter- Evaluation Global Country vention Component Description Level Level Level World Bank The evaluation’s intervention logic spells out the expected √ √ Group’s contribution of the World Bank Group to health service-related interventions outcomes through project financing, investments, and Advisory logic and Services and Analytics. Intervention-specific theory of changes theories of based are developed for the interventions selected for in depth change analyses (CCT, PBF, pandemics and public-private interactions) (see appendix E). Literature Structured reviews of the literatures (that is, academic literature, √ √ √ reviews World Bank documents) around relevant global and country-spe- cific health service issues. Portfolio (i) Portfolio review of project documents (project appraisal √ √ √ reviews and documents, Board reports, Implementation Completion and analyses Results Reports, Implementation Completion and Results (appendix B) Report Reviews, Expanded Project Supervision Reports, Project Completion Reports, and evaluation notes) of health World Bank financing, IFC investments and advisory services supporting health services approved during the evaluation period (that is, information and data extraction, analysis and assessment) (ii) Portfolio review of project documents of a random sample of World Bank financing and IFC investments supporting health services to apply IEG’s frameworks for the evaluation of service delivery and behavior change interventions. Analysis of (i) Use of secondary data from OECD’s Creditor Reporting √ trends and System to analyze the evolution of Development Assistance for statistical Health (DAH) over the evaluation period 2005–16. modeling (ii) Mapping of Health Needs using secondary data on Global (appendix B) Burden of Disease and portfolio data to identify correlations between country health needs and World Bank support to health services. (iii) Statistical analysis of factors associated with projects’ out- come ratings. (iv) Secondary analysis of the 2014 reform effort survey collected by the College of William and Mary’s Institute (see appendix H). Case study In-depth analysis of three delivery mechanisms (CCT, PBF and √ analyses public-private interactions) and World Bank support to pandem- of selected ic preparedness and control. The In-depth analyses involved a interventions review of the portfolio including the identification of drivers for (appendix E) success and failure when pertinent, literature reviews, reconstruc- tion of a theory of change, and evidence gap maps. Evidence Systematic collection of Systematic Review Studies to map out √ gap maps existing evidence of the effects of CCT and PBF on expected (appendix E) health outputs and outcomes pursued by World Bank Group projects. 64 World Bank Group Support to Health Services | Appendix A Table A.1, continued. Inter- Evaluation Global Country vention Component Description Level Level Level Case Study In-depth field-based (Liberia, Romania, and Bangladesh) and √ analyses desk-based (the Philippines, Brazil, and the Republic of Yemen) of selected assessments of the World Bank Group’s support to health ser- countries vices, which involved: review of health service project documents, (appendix F) interviews with stakeholders, review of relevant literature including national development plans and other development partners’ strategies. The Romania case study involved a standardized online survey administered to 1,500 health care providers. Social Social network analysis of health sector actors in Liberia to better √ √ network understand the role of the World Bank Group as a knowledge analyses leader and a source of financing vis-à-vis other organizations in (appendix G) the county. Social network analysis of aid organizations webometrics (in- formation extracted from their websites) to visualize the role of the World Bank Group in the global health landscape and online interactions among global development actors. Global Identification and assessment of World Bank Group engagement √ partnership in health GPPs, which involved the analysis of World Bank’s programs Trust Fund database, semistructured interviews with program (GPPs) managers and World Bank focal points, review of relevant GPP’s analysis and documents (websites, annual reports, etc.). case studies In-depth, desk- based analyses of two selected global part- nerships in health (HRITF and HIA), which applied IEG’s evalu- ation framework for assessing global and regional partnership programs. Figure A.1. Methodological Design: Evaluation Components and Their Relationships Independent Evaluation Group | World Bank Group 65 The World Bank Group’s Intervention Logic and Theories of Change The overall evaluation’s intervention logic is depicted to spells out the expected contribution of the World Bank Group through project financing, investments, advisory services and analytics to health services. Reconstruction of specific intervention-centric theory of changes based on the World Bank Group sector and corporate strategies. Specific interventions specific theories of change have been developed for the selected interventions (conditional cash transfer [CCT], performance-based financ- ing [PBF], pandemics, public-private interactions) using primarily the relevant literature. Literature Reviews Literature review methods were part of the building blocks for many of the evaluation components mentioned before (see table A.1). At the first stages of the evaluation, literature reviews helped the team in the understanding of the policy debate on health services at the global level, and served to inform the development of portfolio review protocols to code World Bank Group health projects. Following the selection of the intervention-type case studies, intervention-specific literature reviews were also conducted to (i) reconstruct a specific intervention-centric theory of change, and (ii) collect evidence from impact evaluations on the effectiveness of the World Bank Group’s support to CCT and PBF. Country-level case studies also reviewed World Bank Group country strategies, government nation- al development plans, and academia policy papers to assess the degree of alignment of the World Bank Group support for health services and the countries health needs. Portfolio Reviews and Analyses The portfolio review exercise involved a systematic review of relevant project documents. The iden- tification of the World Bank Group portfolio supporting health services used health-related Opera- tions Policy and Country Services sector and themes codes,1 text analytics searches on the World Bank Group operational portal, and a manual review to remove false positives projects. The portfolio selection criteria were applied to all World Bank projects financing, 2 IFC Investments (IFC IS), IFC Advisory Services (IFC AS) and World Bank Advisory Services and Analytics (ASA) approved between FY2005 and FY2016.3 In a first stage, all projects mapped to relevant sector and themes codes were selected. Then a targeted keyword search on projects’ objectives and components was conducted to remove false positives. IFC health projects relevant to the evaluation were those classified as health care, life science (pharmaceuticals) and other services directly linked to the health sector (for exam- ple, medical education, health-related). The evaluation portfolio resulting from this identification strate- gy is presented in table A.2 (the data from the evaluation portfolio comprise database 1 depicted in figure A.2). 66 World Bank Group Support to Health Services | Appendix A Table A.2. The Evaluation Portfolio, FY05–16 Commitments Approved Evaluated (as a percent Projects Projects Commitments of the entire Type of Support (no.) (no.) ($, millions) portfolio) GPPs and MDTFs 31 n.a. n.a. n.a. World Bank–financed proj- 619 259 22,756 5 ects: IBRD, IDA, TFs World Bank ASA (for example, 1,033 n.a. 262.9 7 ESW, technical assistance) IFC Investments 124 28 2,672 3 IFC Advisory Services 67 14 71.4 2 Note: ASA = Advisory Services and Analytics; ESW = economic and sector work; GPP = global partnership program; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IFC = International Finance Corporation; MDTF = multi-donor trust fund; n.a. = not applicable; TF = trust fund. The World Bank Group has deployed a wide range of instruments to support health services over the period 2005–16 through two World Bank Group institutions and spanning multiple sectors. IEG identified 1,846 projects with a total commitment in health of $25.7 billion as the primary focus of this evaluation, from which 619 correspond to World Bank investment and policy operations ($22.7 bil- lion), 127 are IFC investment projects ($2.6 billion), and 67 projects are IFC AS ($71.4 million). In addi- tion, the World Bank delivered 1,033 ASAs for a value of $26.9 (table A.1). The portfolio review was conducted on project documents of World Bank project financing opera- tions (project appraisal document, Implementation Completion and Results Reports, Implementation Completion and Results Report Reviews, Project Performance Assessment Reports), and IFC IS and IFC AS projects (Board reports, Expanded Project Supervision Reports, Project Completion Reports and Evaluation Notes) using a protocol specifically developed for the evaluation purpose. The proto- col collect information on project design features and results at completion, and developed a typol- ogy for project objectives (for example, access, quality, efficiency), type of interventions (for example, demand, supply, health system), activities, health focus area, beneficiary groups and targeting, results framework indicators, and efficacy ratings. All World Bank–financed projects, IFC investments and advisory services are evaluated through self-evaluation. For all World Bank–financed projects, within six months of completion, an Imple- mentation Completion and Results Report (ICR) is prepared by World Bank staff. The ICR rates the overall project success relative to its development objective (that is, the project outcome rating) and the success in achieving its specific project development objectives (PDOs). The World Bank’s Independent Evaluation Group (IEG) completes an independent review of all ICR using the evidence contained in the ICR. This ICR Review (ICRR) assigns its own rating, which can differ from the World Bank’s self-assessed ratings, and provides additional analysis. At ICR and ICRR project success relative to its development objective (that is, the project outcome rating) is scored on a 6-point scale (highly unsatisfactory, unsatisfactory, moderately unsatisfactory, moderately satisfactory, satisfacto- ry, and highly satisfactory). The analysis focus on a binary classification of projects (those that rated Independent Evaluation Group | World Bank Group 67 moderately satisfactory or better [MS+]. IEG also rates the efficacy of each PDO (PDO outcome rating) on a four-point scale: high, substantial, modest, and negligible. The evaluation focused on a binary classification of PDO rating: those that rated substantial or better (S+) and those that were not (that is, were rated modest or negligible). A sample of IFC projects (currently 40 percent of the IFC portfolio) is selected for evaluation to draw relative performance inferences about the outcome quality of the portfolio by main strategic groups (for example, country risk/income group and sector). The sampling approach is designed to avoid bias and to be representative of the portfolio. The sampling level sufficient to allow for statistical inference at 95 percent confidence level on a three-year rolling average basis. The information extracted from World Bank Group project documents consisted of database 3 (see figure A.2). Database 3 was analyzed using multivariate regression to identify empirically correlates of World Bank investment project financing (IPF) outcomes. The analysis uses the in-sample and out-of- sample predictive performance of empirical models relating project outcomes to project characteris- tics observed at project approval. The analysis focused on World Bank IPF loans, credits, and grants comprising the evaluation portfolio. Two additional portfolio reviews were conducted within the scope of the evaluation: ƒƒ Intervention-specific protocols were applied to documents of closed projects supporting CCT and PBF interventions, respectively (that is, ICR, ICRR, and PPAR if available) to gather additional information on potential factors of success or failure affecting project outcomes. ƒƒ The Service Delivery and Behavior Change frameworks were applied to a random sam- ple of 85 health projects (75 IPF and 10 IFC) to understand how health projects concep- tualize and operationalized behavior change and service delivery issues at project design and completion. 68 World Bank Group Support to Health Services | Appendix A Figure A.2. The Data Architecture of the Evaluation Analysis of Trends and Statistical Modeling IEG used several secondary sources of quantitative data to provide evidence complementing that obtained through more qualitative methods of analysis to ultimately answer the evaluation questions. IEG mapped global burden of disease (GBD) country-level data to evaluation portfolio data to identify correlations between country health needs and World Bank support to health service. The GBD database provided information on the disability-adjusted life year (DALY) for health conditions or risk factors that directly matched some of the World Bank’s theme codes (for example, human immu- nodeficiency virus [HIV]/acquired immune deficiency syndrome [AIDS], malaria, tuberculosis, and noncommunicable diseases and injuries GBD aggregates). Other GBD measures had to be aggregat- ed to match the World Bank theme codes. For instance, the burden of other communicable diseases was calculated as the difference between total DALY caused by all communicable diseases minus that attributed to HIV/AIDS, malaria and tuberculosis. The combination of the country-level GBD data and additional World Development Indicators and portfolio review data represent database 2 (see in figure A.2). A cluster analysis of the country-level portfolio was performed to identify the groups of client countries receiving comparable volume and breadth of support from the World Bank Group to health services. The evaluation team used data from OECD’s Creditor Reporting System (CRS) to analyze the evolu- tion of Development Assistance for Health (DAH) over the evaluation period 2005–16. CRS provides overseas development assistance data at project-level disaggregated in 12 health-related subsec- tors and in 5 population-related subsectors. The team compared the CRS data set with others also Independent Evaluation Group | World Bank Group 69 tracking DAH, and concluded that the former better addresses the problem of double counting and is relatively more reliable in terms of comparability between countries and across time (Grépin et al. 2011). OECD-CRS data and World Bank Group portfolio data (database 2) formed database 4 (see figure A.2). IEG used the secondary data from the 2014 reform effort survey collected by the College of William and Mary’s Institute for the Theory and Practice of International Relations in partnership with the National Opinion Research Center at the University of Chicago. Case Study Analyses of Selected Interventions The evaluation conducted four case studies of selected interventions (also referred with the term in- depth analysis of interventions) to shed light on the World Bank Group’s contribution to specific areas of engagement: financial incentives to users (CCT) and to providers (PBF), pandemic preparedness and control (pandemics), and public-private interactions (PPIs). The selection of these interventions was based on peer reviewers’ comments and suggestions provided during the consultation process for the finalization of the approach paper. Table A.3 pres- ents the list of all proposed topics, which the evaluation team carefully considered and analyzed to eliminate duplications, improve their definition, and identify their relevance with respect to the health evaluation portfolio. The selection of the four cases was shared with the evaluation consultative group through the “track-your-evaluation” Spark page for confirmation. Table A.3. Suggestions Received and Analysis of the Evaluation Team Suggestions Analysis of the Evaluation Team The Ebola virus disease outbreak in West Africa The broader definition of “pandemics response” was Response to Pandemics using the regular project preferred to encompass the different type of pandem- allocations and crisis response window ics (for example, avian flu, Ebola Virus, and so on) and the various response modality / window. In this way, it would be possible to depict the overall trend in World Bank Group support toward pandemics response beside the specific virus or disease. Health system as an employer The role of health systems as major employer was not suitable for inclusion as employment is not considered an essential element in the evaluation theory of change (see approach paper, page 9). Technical assistance and support to guide middle-in- The intervention is not considered geographically come countries away from hospital centric and special- balanced as it reflects priority and need of a specific ist focused systems that are dominant in the north region (that is, Europe and Central Asia), thus not selected for the in-depth analysis. 70 World Bank Group Support to Health Services | Appendix A Suggestions Analysis of the Evaluation Team The role of public-private partnerships (PPPs; which The team decided to consider specific examples of have typically been for single hospitals) public and private health sector interactions: It is important to look at models of engagement from Health PPP (with long-term contracts and risk sharing PPP laws to various types of concessions to privatiza- among public and private partners) tions to other private provision modalities Contracting packages of health services Pick topics that have received a lot of attention over the last couple of years. How to interact with the private sector would fall squarely in this bucket; The health service delivery landscape in client countries is often composed of multiple actors (public, private, NGO). Research shows that these actors can differ substantially in their efficiency, effectiveness, and equity effects. In deciding whether and how the World Bank Group supports these actors, along some margins, the World Bank Group may have choice, while along other margins, the World Bank Group may be constrained. Result-based financing (RBF) RBF, output-based approaches, PBF, CCT share key Use of output-based approaches element, such as the use of financial incentives to change behaviors. It was decided to cover: Performance-based financing (PBF) financial incentives to providers – PBF; and Conditional Cash Transfer (CCT) programs to support financial incentives to health services users – CCT uptake of health services Provision of regional public goods through the regional window Interventions aimed to reduce consumption of un- Specific interventions to reduce consumptions of healthy goods (for example, tobacco, sugar, salt) goods that expose to risk factors for chronic conditions How to deal with health risk factors related to chronic are tobacco (cardiovascular diseases, cancer, and so conditions, such is the case of taxation for tobacco on), sugar (that is, diabetes) and salt (cardiovascular and alcohol and even sugar-sweetened beverages and diseases). junk foods, changes in the environment to promote physical activity, and so on The identification of these interventions in the overall health portfolio was conducted through the revision of project components that were consistent with the following definitions: ƒƒ CCT: cash transfer to households, on the condition that those households make pre- specified investments in human capital, including the use of health services. ƒƒ PBF: financial incentives to health providers (either to health workers or to health facility) conditioned to the quantity and quality of the services provided. ƒƒ Supporting pandemics preparedness and control (pandemics): virus outbreak that trans- mits readily and can spread fast around the world. ƒƒ Public-Private Interactions (PPIs): the integration of private provision of health services and public financing within the health system. The analysis of World Bank Group’s sup- port for PPI-related activities covers: ƒƒ World Bank projects financing that help strengthen the government’s stewardship of the health system, and particularly of the private sector Independent Evaluation Group | World Bank Group 71 ƒƒ IFC AS support to governments in undertaking public-private partnerships (PPPs) that deliver health services4 ƒƒ IFC IS that supports private providers of publicly financed health services. Table A.3 presents the number of projects and commitments comprising the four interventions. Table A.3. Projects Comprising the Four Intervention Case Studies Projects Committment (no.) ($, millions) World Bank World Bank Project Project Intervention IFC AS IFC IS Financing IFC AS IFC IS Financing PPI 53 58 46 67 1,334 3,549 CCT 0 0 25 0 0 1,320 PBF 0 0 79 0 0 4,222 Pandemics 0 0 63 0 0 2,869 Total (number) 53 58 213 67 1,384 11,969 Total (as a % of 79 46 32 94 52 48 the institution) Note: Number of projects by selected interventions do not add up to total because some projects include more than one type (for example, PBF and CCT). AS = Advisory Services; CCT = conditional cash transfer; IFC = International Finance Corporation; IS = Investment Services; PBF = performance-based financing; PPI = public-private interaction. The case study analyses used a combination of quantitative and qualitative methods (portfolio re- view and analysis, theory of change, evidence gap maps, structured-literature review) and exploit- ed different sources of data (project documents, impact evaluations studies on the effectiveness of PBF programs). First, case study analyses involved the reconstruction of specific intervention-centric theory of chang- es that illustrates how desired changes in health outputs and outcomes are expected to occur as a result of the selected interventions, given assumptions and specific contextual factors. The design process was based on an iterative approach of reconstructing and recalibrating the main causal pathways and was informed by a revision of relevant literature. Second, they used the same portfolio review protocols to describe the evolution of the World Bank Group support for the intervention, the main characteristics of project design (objectives, activities, results frameworks), and achieved results. The theory of change and portfolio review elements were compared with shed light on whether project design consistently addressed the intermediate links and country context factors emphasized in the theory of change. Additional portfolio review was con- ducted to closed and evaluated projects to identify factors explaining project performance through content analysis. 72 World Bank Group Support to Health Services | Appendix A Third, the effectiveness analysis of the case studies combined IEG’s outcome and objective-specific ratings with impact evaluation findings on access, quality, efficiency, and health outcomes. Project design, expected behavior changes, and contextual factors affecting the success of these interven- tions were also identified from the review of the literature. Evidence Gap Map (EGM) also contributed to the effectiveness analysis. Evidence Gap Maps The evaluation used EGMs to identify knowledge gaps on the effects of selected interventions (CCT and PBF) on expected health outputs and outcomes commonly targeted by World Bank Group projects according to portfolio review evidence. EGMs are evidence collections that map out existing and ongoing systematic reviews or primary studies on a particular set of interventions in a framework of policy relevant interventions and outcomes (Snilstveit et al. 2013; Miake-Lye et al. 2016). The EGM used in this evaluation mapped out completed systematic reviews only. EGMs were based solely on systematic review studies available in low- and middle-income coun- tries. The search for systematic reviews was restricted to the years 2000 to the present and to those reviews written in English. Qualitative systematic reviews were excluded, and the EGM focused on systematic reviews of randomized control trials, interrupted time series, and controlled before and after designs. In certain cases, less rigorous designs were included (for example, before and after, cross-sectional time series). The following relevant databases were searched resulting in a total of 5,506 citations: ƒƒ Cochrane Library (Cochrane Database of Systematic Reviews) ƒƒ PDQ Evidence ƒƒ Ovid MEDLINE(R) In-Process and Other Non-Indexed Citations ƒƒ Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) ƒƒ Econlit, RePeC, and World Bank e-library (Via Ebsco Discovery) ƒƒ Health Systems Evidence (McMaster University) ƒƒ 3ie database of impact evaluations The screening identified 47 potentially relevant systematic reviews. Twenty-three were excluded for various reasons (for example, high-income countries, protocol only, the review was not available, the study was not a systematic review) and two were systematic reviews of systematic reviews. The EGM was based on 24 systematic reviews of varying quality. The quality of the systematic review was determined applying the Supporting the Use of Research Evidence (SURE) Collaboration checklist adapted by 3ie. Independent Evaluation Group | World Bank Group 73 Case Study Analysis of Selected Countries IEG conducted six case study analyses of the World Bank Group country-level health support to assess (i) the alignment between World Bank Group support to health services and countries health needs and priorities; (ii) the synergies and complementarities within the World Bank Group; and (iii) the role and contribution of the World Bank Group within the country-level partnership supporting health services. The case studies were purposely selected according to the following two-step approach: 1. A first-level sampling of 17 countries resulted from applying the following selection criteria: (i) countries that received World Bank Group support during the last 10 years; (ii) countries that had a high level of closed projects or a mature IFC portfolio with evaluat- ed projects; (iii) countries where the World Bank Group supported at least one in-depth intervention; and (iv) countries where the World Bank had high/low committed funding as share of total health expenditures. 2. Then, the final sample of six countries (Bangladesh, Brazil, Liberia, the Philippines, Ro- mania, and the Republic of Yemen) was selected purposefully from the first-level sample based on the following principles: (i) coverage of the high and low World Bank support compared with other development partners; (ii) coverage of income level and fragility; (iii) coverage of in-depth analysis of interventions; (iv) regional balance; and (v) balance between high and low capacity to manage development assistance. The final selection of case studies involved three field-based and three desk-based assessments of the World Bank Group’s support to health services (table A.4). Table A.4. Field- and Desk-Based Case Study Analyses World Bank Group Support as a % of Total Health Network Government Country Financing Income Region Fragility Complexity Capacity Liberia 1.3 Low AFR Yes High Low Philippines (desk- 0.3 Lower middle EAP No Low Medium to high based) Romania 0.8 Upper middle ECA No Moderate Medium to high Brazil (desk-based) 0.1 Upper middle LCR No Low Medium to high Republic of Yemen 0.7 Low to lower MNA Yes Moderate Low to medium (desk-based) middle Bangladesh 1.8 Lower middle i SAR No High Low Note: AFR = Africa; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MNA = Middle East and North Africa; SAR = South Asia. a. Relevance is measured by total World Bank Group support as a percentage of total health expenditures. b. Complexity measured by the number of donors according to DAH. c. Government capacity is measured using the World Bank country policy and institutional assessment rating for the health sector (question 9b). 74 World Bank Group Support to Health Services | Appendix A Each case study involved a review of the World Bank Group health portfolio in each country including World Bank Group country strategies, and a review of relevant literature including national develop- ment plans and other development partners’ strategies. Another qualitative method of data collection used was semistructured interviews with subject matter experts within the World Bank Group, and external stakeholders such as government authorities, other development donors, academics, and health providers. For desk-based case studies, interviews with local stakeholders were conducted by phone. The narrative of each case study was guided by a predefined template to ensure the consis- tency of data across cases and facilitate subsequent analyses and comparisons. To assess the role of the World Bank Group further during the last 10 years, IEG conducted a web- based questionnaire distributed to a network of 1,500 primary care physicians in Romania. Some of the main questions were the following: ƒƒ What comes in your mind when you think about the World Bank support to health services in Romania? Would you consider the World Bank Group as having primarily a financial role, as a provider of knowledge or technical leadership or convening power (coordination role with other multilaterals or development institutions)? ƒƒ When you think of the International Finance Corporation (IFC), you would define it as (a) an institution that offers investment to encourage private sector development; (b) an institution that offers advisory and financing services to encourage private sector devel- opment; (c) a part of World Bank Group; (d) an international financial institution, not part of World Bank Group; (e) simply don’t know. ƒƒ What types of health services do you know World Bank Group supported in Romania in the past 10 years? (a) Primary Health Care/Family Medicine; (b) Secondary Health Care/ Ambulatory; (c) Paraclinic Care/laboratory, radiology, and imaging; (d) Hospitals—mater- nal and neonatal care; (e) Hospitals – emergency units and rooms; (f) Hospitals—dialysis centers; (g) Hospitals (in general); (h) Long-term and palliative care; (i) Technical assis- tance (analysis, review, strategies); (j) simply don’t know. ƒƒ What role do you think the World Bank should focus on more in its support to health services in Romania, considering the World Bank’s main strength, the needs, and what other actors provide? (a) Financing; (b) Knowledge leadership; (c) Convening power; (d) simply don’t know. The final number of respondents was 211 (14 percent response rate), from which 64 percent were family doctors who have a private family practice in urban areas and 36 percent in rural areas. Most of them (93 percent) owned their own practice, while the rest worked as employees. The specializa- tion profile was 68 percent were senior doctors, 28 percent were specialists of family medicine, and 4 percent physicians. In Liberia, moreover, a standardized questionnaire was developed and administered face-to-face with representatives of relevant health organization to collected evidence on the World Bank Independent Evaluation Group | World Bank Group 75 Group’s leadership role in knowledge and financing which was the basis for a subsequent social network analysis (SNA). Social Network Analyses IEG conducted SNA to generate visual network maps that provide insights about the relationships among key organizations involved in supporting the health sector in Liberia, and how the World Bank Group positions itself in the health sector in Liberia in relation to other organizations. The identification of key organizations in the Liberia health sector was based on the review of aide memoires of recent World Bank Group missions to Liberia, and membership information of the health sector coordination committees and technical working groups provided by the Health, Nutrition, and Population Global Practice. The preliminary list of organizations was further refined through consultations with World Bank’s task team leader of health projects in Liberia, the World Bank’s Health Specialist based in Monrovia, and the Manager of the World Bank’s Project Management Unit at the Ministry of Health. A total of 62 organizations (four Governmental Institutions, 15 Multilaterals, 12 Bi-laterals, and 31 NGOs and Foundations) were identified through this process. Data collection involved the development of a standardized questionnaire administered face-to-face during meetings held in the Liberia with a response rate of 87 percent. To inquire about the World Bank’s knowledge leadership and financial flows respondents were asked about: ƒƒ Which organizations do you consider “knowledge leaders” in the Liberian health sector? ƒƒ Which organizations produce the most credible and useful knowledge for your work in health? ƒƒ Which organizations do you turn to when you need technical advice? ƒƒ Which organizations’ publications do you often read to gain new information? ƒƒ How much funding did your organization receive from / provide to the listed organiza- tions in the last year (2016, calendar or fiscal year) in $ million? The SNA software Cytoscape was used to calculate the network metrics and to draw the network maps (Smoot et al. 2010). SNA of Webometric was also used in the evaluation to analyze relationships among international aid organizations, including the World Bank Group, proxied by the organizations’ co-citation networks. Sources of data were the organizations’ websites from which data were collected by a standard web crawler and implemented in Python using the “scrapy package” (see appendix F for more details). 76 World Bank Group Support to Health Services | Appendix A Global Partnership Programs Analysis and Case Studies The global partnership program analysis aimed at answering the following question: “What has been the nature and extent of World Bank Group support to health services provided through its global and regional partnership programs in the last 10 years?” And “What has been the role of the World Bank Group in global and regional partnerships supporting health services?” In addition, IEG carried out case study analyses of two partnerships to assess the achievement of their development objectives. The Health Results Innovation Trust Fund (HRIFT) and Health in Africa Initiative (HIA) were selected because of their relative importance in term of resources channeled, rel- evance to World Bank Group strategies, and synergies with other components of the evaluation. The HRITF is the largest multi-donor trust fund program in health housed in the World Bank, while HIA is the first IFC-led comprehensive initiative in the health sector to enable private sector participation in African countries. The list of partnerships that support health services relevant to the evaluation was identified through the World Bank Group’s TFs database and targeted web, and finalized with the help of HNP GP Global engagement team. The methods used to collect and triangulate the evaluative evidence consisted of an analysis of portfolio of the World Bank Group’s health global partnership programs and MDTFs, semistructured interviews with program managers, and development partners, searches on partnerships external websites, review of periodic external evaluations. The in-depth case studies were analyzed by adapting IEG’s evaluation framework for assessing global and regional partnership programs (World Bank 2007a, 2007b). Ensuring Validity of Findings The evaluation team undertook several measures to ensure the validity of the evaluation’s find- ings, including consultations with World Bank Group staff, use of specific protocols and coding templates (to review portfolio documents, impact evaluations, systematic review studies, and collect country case evidence), and intercoder reliability and quality control measures to guar- antee a consistent approach to coding and analysis across evaluation components and across team members. The team also applied triangulation at multiple levels, first by crosschecking evidence sources within a given methodological component. For instance, within country case studies interview findings were compared across type of stakeholders (World Bank Group staff, government officials, academia, health experts, and other development partners). Second, the team applied triangulation across evaluation components—for example, cross-validating findings from country-level case studies with findings from portfolio analysis and literature reviews. The evaluation team also applied external validation mechanisms at various stages of the evaluation process. For example, the selection of intervention case studies was based on suggestions received Independent Evaluation Group | World Bank Group 77 during the consultation process for the finalization of the Approach Paper, and the final selection was shared with the evaluation consultative group thought “track-your-evaluation” spark-page for confir- mation. Four peer reviewers provided feedback at the approach paper stage and at the final report review stage. Limitations Limitations to the evaluation design fell broadly into two categories: limitations due to conscious choices about scope, and resources constraints. One scope-related limitation was the inherent trade-off between breadth and depth of analysis. This is apparent in country-level case studies since the institutional mapping of all the actors in the health sector required significant amount of time, and thus it was decided to pilot the first institutional mapping and a sector network analysis in one country only, Liberia. The other five country-level case studies used a more simplified approach centering the interviews on the top five development partners. Also, the selec- tion of field-based country-level case studies favored those countries where there was World Bank Group support for health interventions at the expense of those where there were no health-related projects. Although overall portfolio analysis exploited the breadth of the evaluable material, IEG acknowledges that the assessment of project effectiveness through outcomes ratings is challenges the internal validity of the evaluation findings. First, outcome ratings used in the portfolio analyses are based on incomplete samples of closed projects. Second, when available, outcome ratings tend to be a biased measure of the overall projects success. Third, the team recognizes that IFC IS, IFC AS and World Bank project financing define and monitor objectives differently, therefore direct comparison between interventions with regards to the ratings of project’ outcome and PDO’s efficacy should be considered with caution. The use of outcome ratings in intervention-type case studies offers additional challenges: (i) the complexity of health projects usually comprising multiple interventions turns project outcome and ob- jective ratings a rather imperfect measure of the effectiveness of the intervention itself, but the entire project activities; and (ii) few closed projects with available ratings cannot generate robust evidence about the World Bank Group contribution to health outputs and outcomes through the selected inter- ventions. Notwithstanding these shortcomings, the team has made efforts to provide more in-depth evidence on the effectiveness of the World Bank Group. First, by estimating indicators’ achievement rates by type of objectives for selected intervention-types (CCT and PBF), and second, by systemati- cally reviewing impact evaluation of PBF and CCT programs supported by the World Bank Group. There are a couple of limitations entailing the SNA. One refers to the construct validity of the data used to measure the variables of interest. In this study, the team tried to achieve a high level of con- struct validity of findings by collecting data based on a customized standardized questionnaire that was administered to all the selected organizations. In the case of webometrics, proper names and acronyms do not always uniquely identify an organi- zation, as some of them are also words can be used, without necessarily referring to the organization 78 World Bank Group Support to Health Services | Appendix A that is searched. To avoid the potential biased a procedure was used to exclude organizations using the citation information. Each organization can be cited in two ways: (i) in the text of a page. and (ii) by a hyperlink. Although the former citation is affected by this problem, the latter is not. Therefore, if an organization is disproportionally cited through mechanism (i), then the organization is likely to introduce noise in the networks and has been excluded from the graph. The econometric analysis of correlates of World Bank financed project outcomes is affected by po- tential endogeneity problems and measurement errors stemming from the rating system. Finally, external sources of data may impose limitations to the external validity of the survey due to sample selection (for example, 2014 Reform Survey). References Grépin, K.A., K. Leach-Kemon, M. Schneider, and D. Sridhar. 2011. “How to Do (or Not to Do). Tracking Data on Development Assistance for Health.” Health Policy and Planning 27 (6): 527–534. Miake-Lye, I. M., S. Hempel, R. Shanman, and P. G. Shekelle. 2016. “What Is an Evidence Map? A Sys- tematic Review of Published Evidence Maps and Their Definitions, Methods, and Products.” Systematic Reviews 5 (1): 28. Snilstveit, B., M. Vojtkova, A. Bhavsar, and M. Gaarder. 2013. “Evidence Gap Maps: A Tool for Promoting Evidence-Informed Policy and Prioritizing Future Research.” Policy Research Working Paper 6725, World Bank, Washington, DC. ———. 2007a. Sourcebook for Evaluating Global and Regional Partnership Programs Indicative Principles and Standards. Independent Evaluation Group. Washington, DC: World Bank. ———. 2007b. Guidelines for Global and Regional Program Reviews (GRPRs). Independent Evaluation Group. Washington, DC: World Bank. Smoot, M.E., Ono, K., Ruscheinski, J., Wang, P.L. and Ideker, T., 2010. Cytoscape 2.8: new features for data integration and network visualization. Bioinformatics 27 (3): 431–432. 1 Operations Policy and Country Services Sector Boards and Corresponding Sector and Theme Codes. Relevant sector codes were: compulsory health finance (BK), public administration—health (BQ), noncompulsory health finance (FB), and health (JA). Relevant theme codes were: child health (63), other communicable diseases (64), health system performance (67), nutrition and food security (68), population and reproductive health (69), other human development (70), HIV/AIDS (88), noncommunicable diseases and injury (89), Malaria (92), and Tuberculosis (93). 2 World Bank project financing comprise investment project financing, development policy financing, and Program-for- Results. 3 Advisory Services and Analytics include economic and sector work, impact evaluation, technical assistance, and reimbursable advisory services. 4 This excludes health financing public-private partnerships (PPPs), and studies and regional PPPs. Independent Evaluation Group | World Bank Group 79 Independent Evaluation Group | World Bank Group 79 Appendix B. Portfolio Analysis Portfolio Trends Country-level Cluster Analysis of the Portfolio The evaluation used cluster analysis to classify client countries according to the volume and breadth of the World Bank Group support to health services, specifically: (i) the total volume of support received in health sector during the FY05–16 (in log); (ii) the total number of projects (World Bank project financing, ASAs, and IFC investments and advisory services); (iii) the total number of instruments (type of World Bank project financing, ASAs and IFC investment); and (iv) the total number of global practices (GPs) delivering the support within the country. Table B.1 shows the main characteristics of the three clusters that comprised 39, 60, and 37 countries, respectively. Table B.1. Volume and Breadth of World Bank Group Support in the Three Clusters Average Commit- Average ment per Total number of Number Countries Country commitment projects per Projects of GP Instruments Cluster (no.) ($, millions) ($, millions) country (total no.) (no.) (no.) 1 39 3 117 3.3 129 0.9 2.0 2 60 61 3,660 7.6 456 1.4 3.7 3 37 558 20,646 20.1 744 2.2 5.2 In addition, World Development Indicators and global burden of disease indictors were used to determine the characteristics of the countries of the three clusters: the average gross domestic product (GDP) per capita of the period; and the average DALYs per capita of the period. Countries in cluster 1 have high GDP per capita, low burden of disease. The countries in cluster 2 and 3 have comparable GDP per capita and health condition. However, the two groups of countries present a remarkable difference in the performance of the World Bank project financing over the period: 75.38 percent of projects in countries receiving more support from the World Bank Group (cluster 3) are rated as satisfactory or better compared with 65 percent in cluster 2 (see figure B.1). World Bank Project Financing and Advisory Services and Analytics Between fiscal year 2005 and 2016, the World Bank approved 619 projects financing and supporting health services, with total original commitments of about $22,756 million. The distribution of World Bank project financing and the evolution between two subperiods 2005–10 and 2011–16 is also presented in table B.2. Africa is the region that received the highest number of World Bank project financing supporting health services and Latin American and the Caribbean the region receiving the largest share of commitments. Over time, the Africa and Europe and Central Asia regions have shown the highest increase in term of commitments. World Bank project financing focusing on fragile and conflict-affected countries represents 23 percent of the total number of approved projects and 11 percent of commitment volumes ($2,489 million) over the entire evaluation period. The share of 80 Figure B.1. Portfolio and Additional Characteristics of the Three Clusters Cluster DALYs per 1000 habitants GDP per capita 1 327 7,380 2 478 3,312 3 430 3,403 Note: DALY = disability-adjusted life year; GDP = gross domestic product. Table B.2. Portfolio of World Bank Project Financing: Health Services Original Original Commitment Projects Commitment Projects 2005–10 2011–16 2005–10 2011–16 2005–16 Region % of total % of total % of total % of total $, millions % of total no. % of total AFR 22 30 39 39 5,973 26 242 39 EAP 6 7 12 13 1,512 7 76 12 ECA 10 17 13 15 3,049 13 86 14 MENA 2 2 7 7 488 2 41 7 LAC 38 21 14 12 6,513 29 81 13 SA 22 16 11 8 4,286 19 60 10 Regional 1 7 4 7 934 4 33 5 projects Total 100 100 100 100 22,756 100 619 100 FCV situation 9 13 23 23 2,489 11 144 23 Note: AFR = Africa; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. Independent Evaluation Group | World Bank Group 81 World Bank commitments to FCV situation has increased between the 2005-10 and 2011-16 period from 9 to 13 percent. Table B.3. Portfolio of World Bank ASAs: Health Services Original Original Costs Projects Costs Projects 2005–10 2011–16 2005–10 2011–16 2005–16 Region % of total % of total % of total % of total $, millions % of total no. % of total AFR 9 18 13 16 38.0 14 154 15 EAP 10 8 11 10 22.0 8 108 10 ECA 4 3 7 9 8.9 3 84 8 MENA 7 2 10 6 9.9 4 84 8 LAC 2 4 5 8 8.4 3 66 6 SA 7 4 6 9 12.7 5 77 7 Regional 62 62 48 41 163.1 62 460 45 projects Total 100 100 100 100 263.0 100 1,033 100 FCV situation 5 4 7 8 10.9 4 79 8 Note: AFR = Africa; ASA = Advisory Services and Analytics; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. In the same period, the World Bank approved a total of 1,033 ASAs for a total cost of $262.9 million. Most ASAs supporting health services have a regional or global focus. The Africa and the East Asia and the Pacific Regions are the two main recipients of country-specific ASAs (see table B.3). ASAs in FCV countries represent about 4 percent of ASAs total value. IFC Investments and Advisory Services Between fiscal year 2005 and 2016, IFC approved 124 investment projects in the health sector, with total original commitments of about $2,673 million. These projects were mostly concentrated on South Asia (25 percent of the volume and 23 percent of the total number of approved projects), followed by East Asia and Pacific, and Europe and Central Asia. The bulk of IFC investment commitments ($1,516 million, 57 percent of total) was concentrated in four countries: India, receiving the lion’s share with 23 percent ($615 million); China at 17 percent ($458 million); Turkey at 12 percent ($315 million); and the Russian Federation with 5 percent ($128 million) (see table B.4). IFC’s investments have mostly been concentrated on low-income and lower-middle-income countries (LMICs) and upper-middle-income countries (UMICs) with 40 percent and 36 percent of total projects, and 35 percent and 51 percent of total original commitments, respectively. This pattern 82 World Bank Group Support to Health Services | Appendix B in health investments by country income level largely mirrors IFC’s overall investment portfolio. IFC’s focus on low-income is very limited and has decreased over time in favor of more UMICs. IFC approved 67 advisory services in the health sector between fiscal year 2005 and 2016, accounting for a total of $71 million. The majority of the projects were concentrated on Africa (31 percent in value and number of projects), followed closely by South Asia. India (18), Nigeria (5), Table B.4. Portfolio of IFC Investments: Health Services Original Original Committment Projects Commitment Projects 2005–10 2011–16 2005-10 2011–16 2005–16 Region % of total % of total % of total % of total $, millions % of total no. % of total AFR 11 2 8 11 156 6 12 10 EAP 17 20 17 18 509 19 22 18 ECA 16 20 17 16 486 18 21 17 MENA 8 9 11 3 235 9 9 7 LAC 11 17 11 16 379 14 17 14 SA 25 24 24 21 657 25 28 23 Regional 13 7 11 13 251 9 15 12 projects Total 100 100 100 100 2,673 100 124 100 FCV situation 1 0 3 2 9 0 3 2 Note: AFR = Africa; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. Table B.5. Portfolio of IFC Advisory Services: Health Services Net Value Projects Net Value Projects 2005–10 2011–16 2005-10 2011–16 2005–16 Region % of total % of total % of total % of total $, millions % of total no. % of total AFR 13 37 29 33 21.8 31 21 31 EAP 0 2 8 2 1.2 2 3 4 ECA 2 7 17 12 4.0 6 9 13 MENA 17 0 13 2 3.3 5 4 6 LAC 7 15 8 9 9.4 13 6 9 SA 5 38 13 40 20.5 29 20 30 Regional 56 1 13 2 11.2 16 4 6 projects Total 100 100 100 100 71.4 100 67 100 FCV situation 1 5 8 7 2.8 4 5 7 Note: AFR = Africa; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. Independent Evaluation Group | World Bank Group 83 Kenya (4) and Lesotho (4) were the main beneficiaries of IFC Advisory Services, attracting together around 46 percent of all IFC advisory service projects. The majority of IFC advisory assistance has been concentrated on LMICs with 57 percent followed by low-income countries (LICs) with 19 percent of all projects, Hospitals in LMICs and LICs accounted for 37 percent of all projects, of which 28 percent (7) were in Africa and 32 percent (9) were in South Asia. This distribution indicates a similar pattern to that of the overall health advisory services portfolio. Activities of World Bank Project Financing IEG’s portfolio review classified project interventions according to whether investments were addressed to support the supply of health care services, its demand, or the health system as a whole. The comparison of closed and active World Bank–financed projects shows that the evolution at project design show more emphasis in health financing, health information management systems and financial incentives to health service providers over time. Health system–strengthening activities have increased (from 85 percent to 93 percent) driven by an increase in health financing (from 34 percent to 55 percent) and health management information systems activities (from 55 percent to 74 percent). The increase is more pronounced in low-income countries and, to a lesser extent, in middle-income countries. In turn, demand-side interventions show a slight decrease (from 64 percent to 56 percent). Within this type there has been a decrease in awareness campaign activities directed toward health services users (from 50 percent to 47 percent) in favor of financial incentives directed toward health services users (from 10 percent to 17 percent; figure B.2). Figure B.2. Activities Supported by World Bank–Financed Projects Note: ANY HS = any health system intervention; HF = health financing; FI to HS = Financial Incentive to health Services; HMIS = health information and management systems / monitoring and evaluation; P&L = procurement and supply chain logistics; STW = stewardship, regulations/policy/strategy reform; MED = drugs, vaccines and consumables/supply chain; MED TECH: medical technology; EQUIP = health equipment and labs health; INFRA = infrastructure or land acquisition; PHS = Public Health services (for example, Surveillance), SKILLS = training and skills to HS providers; ANY D = any demand-side intervention; CCT = financial incentives to users with health conditionalities); IEC = information campaign to HS users (households, patients, general population); IS = insurance programs. 84 World Bank Group Support to Health Services | Appendix B Overall, the large majority of World Bank financing includes prevention activities (71 percent) and their presence is increasing (66 percent among closed projects, compared with 81 percent among active projects). Prevention activities are more common in projects in low-income countries and FCV situations. About half of World Bank Group projects contains activities directed to disease control and treatment. However, while disease control activities appear stable over time, treatment activities have increased from 48 percent among closed projects to 59 percent among active projects (see table B.6). Most IFC investments comprised disease treatment activities (84 percent). On the other hand, activities directed to diseases prevention were identified in only 14 percent of IFC investments, and activities directed to diseases control in 10 percent of IFC investment. The prevalence of activities directed to treatment among IFC investments was detected in all countries and income levels and in FCV situations among active projects (see table B.6). Bank Group Projects: Presence of Diseases Prevention, Table B.6. World Control, or Treatment Activities (percent) World Bank Project Financing IFC Investments Project Status or Country Income Prevention Control Treatment Prevention Control Treatment Active 81 48 59 16 16 82 Closed 66 47 48 13 6 86 Total 71 48 52 14 10 84 Upper middle income 61 41 41 20 9 93 Lower middle Income 73 46 50 7 9 76 Low income 74 50 60 25 8 100 FCV/Marginal 80 56 66 0 0 100 Note: FCV = fragility, conflict, and violence/ Focus of World Bank Project Financing and IFC Investments Table B.7 summarizes the type of diseases or health conditions prioritized by World Bank project financing into four broad categories: (i) maternal and child health, and nutrition; (ii) communicable diseases; (iii) noncommunicable diseases; and (iv) general health without indication of specific diseases or health conditions. Overall, maternal and child health, and nutrition are most commonly addressed by World Bank financing projects (46 percent for the entire portfolio), and their importance among open projects (57 percent), compared with closed (38 percent). Overall over time, World Bank projects have reduced their focus on communicable diseases (from 30 percent among closed projects to 15 percent among active projects) and general health (from 26 percent among closed projects to 16 percent among active projects). The focus on noncommunicable diseases, though limited overall, is expanding over time. The health focus of World Bank project financing broadly follow the disease priorities of the client countries. For example, projects in low-income, FVC situations, as well as those implemented in the Africa and South Asia Regions focus more on Independent Evaluation Group | World Bank Group 85 maternal-neonatal and nutritional diseases and conditions and communicable diseases than the entire portfolio. Conversely, projects in upper-middle income countries and in the Europe and Central Asia countries focus more on general health and noncommunicable diseases and conditions than the entire portfolio (see table B.7). Table B.7. World Bank Project Financing: Health Focus (percent) General Health (no Project Status, specific Maternal and Noncom- Country Income, or Communicable diseases or Child Health, municable Other / Region Diseases conditions) and Nutrition Diseases Missing Active 15 16 57 10 2 Closed 30 25 38 6 1 Total 24 21 46 8 1 Upper middle income 14 34 32 18 1 Lower middle Income 21 19 50 9 1 Low income 28 18 50 3 1 FCV/Marginal 28 16 54 1 0 AFR 25 18 55 2 0 EAP 21 38 36 5 0 ECA 16 41 20 23 0 MENA 18 26 41 8 3 LAC 16 17 51 14 2 SA 31 12 44 9 4 Regional projects 50 13 30 0 7 Note: AFR = Africa; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. IFC investments focus primarily on general health without indication of specific diseases or health conditions (56 percent for the entire portfolio) and noncommunicable diseases (35 percent). Conversely, few IFC investments focus on, maternal, neonatal and nutrition conditions and diseases (4 percent) and communicable diseases (6 percent). The primary focus of IFC investments on general health (without indication of specific diseases or health conditions) or noncommunicable diseases is observed in all Regions, in countries at all income levels and in FVC situations. The comparison between open and operationally mature IFC investment, shows that over time the focus on noncommunicable diseases is expanding (45 percent among open investment, compared with 27 percent among operationally mature projects) (see table B.8). Table B.9 presents the information about the level of care supported by World Bank Group projects. Overall, the large majority of World Bank financing (80 percent) support to the primary care level (that is, the first point of contact). Support to secondary (that is, specialized) care was 32 percent of projects, and support to this level of care is expanding over time. The tertiary level of care, comprising 86 World Bank Group Support to Health Services | Appendix B Table B.8. IFC Investments: Health Focus General Health (no specific Maternal and Noncom- Project Status, Coun- Communicable diseases or Child Health, municable Other / try Type, or Region Diseases conditions) and Nutrition Diseases Missing Active 4 42 3 45 6 Operationally mature 9 56 4 27 4 Total 6 50 4 35 5 Upper middle income 4 41 6 41 9 Lower middle Income 6 55 3 32 3 Low income 21 32 0 42 5 FCV/Marginal 0 50 0 50 0 AFR 12 47 0 35 6 EAP 14 50 0 36 0 ECA 4 50 7 29 11 MENA 0 50 0 42 8 LAC 0 38 6 47 9 SA 10 46 5 36 3 Regional projects 0 87 0 13 0 Note: AFR = Africa; ECA = Europe and Central Asia; FCV = fragility, conflict, and violence; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. specialized medical investigation and treatments) was supported in about 23 percent of projects. Finally, about 22 percent of World Bank projects provide support to the entire network (comprising primary care and referral to specialized providers). Support to primary care was more common in projects in low-income countries and FCV situations. The comparison between closed and open projects shows that support to secondary care is increasing significantly over time (from 27 to 42 percent) (see table B.9). Overall, IFC investments support more the secondary level of care (54 percent of projects) and the entire network of health service providers (55 percent). Support to the primary care level is identified in 35 percent of IFC investments, and support to the tertiary care in 38 percent of the cases. However, IFC investments support to the primary care level increases in low-income countries (50 percent) and FCV situations (67 percent of projects) (see table B.9).Table B.10 presents the percentage of World Bank investment project financing (IPF) that target specific population group. Overall, about 64 percent of IPF identify specific disadvantaged population groups that are targeted by project interventions. Active projects identify specific disadvantaged population groups more frequently than closed projects. The poor is the disadvantaged population group more often targeted in World Bank IPF, followed by gender or sex, and by indigenous populations and ethnic groups. (see table B.10). Independent Evaluation Group | World Bank Group 87 Table B.9. World Bank Group Projects: Health Care Setting Project Status World Bank Project Financing IFC Investments or Country Primary Secondary Tertiary Primary Secondary Tertiary Income Care Care Care Network Care Care Care Network Active 82 42 23 22 36 60 32 64 Closed / Opera- 79 27 23 22 35 49 42 48 tionally mature Total 80 32 23 22 35 54 38 55 Upper middle 73 24 20 31 34 61 52 70 income Lower middle 80 40 30 18 27 38 27 44 income Low income 86 29 19 19 50 58 25 33 FCV/Marginal 87 39 28 20 67 67 33 67 Note: FCV = fragility, conflict, and violence. Bank IPF: Percentage of Projects Targeting Specific Table B.10. World Population Groups Population Group Active Closed Total Project targeting specific groups 70 59 64 Poor (or at risk of poverty) 34 27 30 Gender or sex 18 11 14 Indigenous population and ethnic groups 9 9 9 Youth 4 5 5 Rural population 3 5 4 Displaced people and refugees 2 1 1 Elderly 1 0 1 Recipients of IFC Projects Most IFC investment support goes to hospital and clinics and pharmaceuticals, and there is high concentration in large markets. Hospitals and clinics account for 62 percent and 28 percent of commitments, respectively (61 and 21 percent of total projects). About 69 percent and 47 percent of IFC commitments to hospital and clinics and pharmaceuticals, respectively, went to India, China, and Turkey (65 percent of projects). India received the lion’s share with $246 million and $318 million in pharmaceuticals and hospitals, leading China and Turkey, both of which received $264 million and $455 million for each category, respectively. IFC’s emphasis on hospitals and clinics has slightly declined from 64 percent to 57 percent (figure B.3) with almost all Regions experiencing this decline except for East Asia and Pacific. 88 World Bank Group Support to Health Services | Appendix B Figure B.3. Share of IFC Investment Services Health Services Portfolio, FY05–16 a. Share of Investment Services b. Evolution over time IFC’s investments in the health sector show high incidence of repeat support to client groups and networks. Repeat client groups with whom IFC has undertaken two or more investments during the FY05–16 period account for about 62 percent. This is higher than IFC overall which account for approximately half of IFC projects (48 percent). Repeat clients feature on average larger commitment volumes compared with nonrepeats. They receive more IFC instruments (such as equity, loans, quasi-equity) than one-off clients do; however, equity investments account for a higher share of one-off investments. Consistent with its strategic intent support to networks form the majority of its IFC investments. Support to networks varies from supporting specialty chains or to a network of clinics that may support all three levels of care (primary, secondary, or tertiary level of care) or any combination of the three. IFC support to networks has increased from 60 to 90 percent over the second half of evaluation period. As with IFC Investment Services, hospitals and clinics account for most of IFC AS portfolio, but its share has decreased over time. Accounting for 55 percent of interventions and 45 percent ($32 million) of total commitments, followed by Health in Africa Initiative funds accounting for 14 percent of interventions and 25 percent ($18 million) of total commitments. When comparing closed and active projects, IFC AS’s emphasis on hospitals has substantially declined from 70 percent to 41 percent in favor of funds that increased from 3 percent to 22 percent (figure B.4, panel b). Most of the decline was due to decrease in Africa, East Asia and Pacific, and Latin America and the Caribbean. The increases in funds were a result of the Health in Africa, through which IFC intends to help governments improve the business environment for private sector investments. Independent Evaluation Group | World Bank Group 89 of International Finance Corporation Advisory Services to Figure B.4. Share Health Services Portfolio, FY05–16 a. Share of advisory services b. Evolution over time Portfolio Alignment to Country Health Sector Priorities Rank Correlations between World Bank Project Financing and Advisory Services and Analytics and Global Burden of Disease Rank correlations between national health priorities, proxied by the global burden of disease and the volume of the World Bank project financing support according to selected conditions (that is, child health, human immunodeficiency virus [HIV]/acquired immune deficiency syndrome [AIDS], malaria, tuberculosis, other communicable diseases, and noncommunicable diseases and injuries) suggest that the World Bank’s portfolio is aligned with the country health priorities. Figure B.5 depicts the Spearman’s correlation coefficients estimated for each country for the entire period. Overall Spearman’s coefficient is 0.307. A breakdown by subperiods suggest that World Bank projects financing and ASAs are more closely aligned to country health priorities in the most recent years, since correlation increased from 0.25 to 0.342, although the difference is not (statistically) significant (at 95 percent level). The World Bank’s support has improved its focus on countries’ health priorities mainly in Latin America and the Caribbean and Europe and Central Asia Regions (see table B.11). 90 World Bank Group Support to Health Services | Appendix B Figure B.5. World Map from Tableau of Spearman Correlation Table B.11. Spearman’s Correlation by Region, 2005–10 and 2011–16 Period  Region 2005–10 2011–16 Africa 0.256 0.311 East Asia and Pacific 0.214 0.234 Europe and Central Asia 0.352 0.590 Latin America and the Caribbean 0.188 0.445 Middle East and North Africa 0.320 0.286 South Asia 0.291 0.234 Total 0.255 0.342 [and 95 percent confidence interval] [0.1621–0.3281] [0.2386–0.4108] The analysis also looked at the ranking for each specific disease or condition. Table B.12 shows the percentage of countries for which World Bank project financing was underprioritized, well- aligned, or over-prioritized compared with its relative burden of disease, measured in DALYs, for selected health conditions. While significant degree of alignment is observed for malaria, tuberculosis, and child health, it appears that the World Bank’s support to other communicable diseases, and especially noncommunicable diseases and injuries, has received less attention compared with what GBD data suggest. For example, in about 82 percent of client countries, the World Bank tends to underprioritized these health conditions. On the other hand, the analysis shows that in about 40 percent of the countries the World Bank devotes relatively more resources toward HIV/AIDS than what the burden of disease rank would suggest. Independent Evaluation Group | World Bank Group 91 Table B.12. Alignment for the Six Health Conditions, 2005–16 Percentage of Countries Health Condition Underprioritized Aligned Overprioritized HIV/AIDS 15.7 45.1 39.2 Malaria 13.7 66.7 19.6 Child health 31.4 49.0 19.6 Tuberculosis 29.4 57.8 12.8 Other communica- 50.0 43.1 6.9 ble diseases NCD and injuries 82.3 17.7 0.0 Note: AIDS = acquired immune deficiency syndrome; HIV = human immunodeficiency virus; NCD = noncommunicable diseases. Relationship Between World Bank Group Support and DALY The relationship between World Bank country-level support to health services and the burden of disease, measured by the DALY per capita, is shown in figure B.6. The scatter plot between country- level World Bank project financing commitments and the DALY per capita (figure B.6, panel a) show a positive and robust relationship between the two variables (that is, positive trend line and a relatively large R-squared). This indicates that in general, countries with sicker population receive more from the World Bank in term of project financing commitments. On the other hand, the relationship between country-level World Bank ASAs costs and DALY per capita is almost not existent (the trend line is almost flat) and weaker (as indicated by a low R-squared). Figure B.6. Relationship between World Bank Country-Level Support and DALY a. World Bank project financing b. World Bank Advisory Services and Analytics Due to IFC’s focus on middle-income countries, it is not surprising that the relationship between IFC support to health services and the DALY per capita is negative and weak for both IFC investments and IFC Advisory Services, as shown in figure B.7. This indicates that in general, countries with sicker population receive less support from IFC. 92 World Bank Group Support to Health Services | Appendix B Figure B.7. Relationship between IFC Country-Level Support and DALY a. IFC investments b. IFC Advisory Services Note: DALY = disability-adjusted life year; IFC = International Finance Corporation. Alignment of IFC investments with Country Health Needs and Commercial Attractiveness IFC’s deployment of resources needs to take account of the fact that IFC has two, often competing objectives: (i) to invest to generate positive financial results; and (ii) to have developmental results. To assess the alignment of IFC’s deployment of investment capital, IEG has developed an index (see annex 1 for methodology) that capture the two dimensions to determine the countries in which there is potential and actual for IFC’s investments. Results show that IFC investments seem to be, for the most part, in the right places; but coverage is still low and its volumes highly concentrated few countries. Figures B.6 and B.7 show that 64 percent of IFC investments are located in countries that combine relatively high health needs and relatively good environment for commercial attractiveness (the North-East quadrant). However, there are still 22 countries (about 65 percent of countries in this quadrant) where IFC is not active. This suggests that there is potential for IFC investments to expand coverage. World Bank Support to Population and Reproductive Health, 2010–16 Reproductive health services are key to lower fertility rates, improve pregnancy outcomes, and reduce sexually transmitted infections. The reproductive action plan 2010–15, approved by the World Bank in 2010, committed to prioritizing the World Bank’s support in the 57 countries with high maternal mortality ratios and high total fertility rates and, within this group, in countries where these rates have remained high over extended periods (World Bank 2010). One of the challenges highlighted in the action plan was the decline in resources allocated to population and reproductive health in the World Bank’s health portfolio. From 1995 to 2007, the share of population and reproductive health in total commitment to health declined from 18 percent to Independent Evaluation Group | World Bank Group 93 Figure B.8. Health Need versus Commercial Attractiveness Source: Based on portfolio and World Development Indicators data. Note: Size refers to Population size. Orange = countries with IFC Investments (excluding Pharma); blue = countries with no IFC investments. 10 percent in 2007. A similar downward trend in population and reproductive health investment was observed globally in the overseas development assistance data (World Bank 2010). The World Bank responded and stepped up investment in Population and Reproductive Health (theme code 69) in the 57 countries with high maternal mortality and total fertility. To test whether the increase was different from the rest of the portfolio we used a difference in difference estimator to compare the allocation to Population and Reproductive Health in World Bank–financed projects approved in 2005–09, versus 2006–16 in the 57 countries with the rest of the World Bank portfolio. The results show that investment in the prioritized countries increased by 6.9 percent, compared with the other countries (see figure B.9). 94 World Bank Group Support to Health Services | Appendix B There is a need, however, to go beyond financing and address the root causes of poor population and reproductive outcomes. It is important to involve partners in reproductive health decisions and to invest in sexual health education at early age. In other words, the increased support to population and reproductive health services in countries that need it the most is welcome, but it also important to improve the quality and scope of the investment to achieve results. Bank Project Financing Allocated to Population and Figure B.9. World Reproductive Health, 2005–16 Note: Comparison of 57 countries with high maternal mortality rate and total fertility (treated) and the rest of the sample (control). World Bank Projects Financing Addressing Sexual and Gender-Based Violence in Situations of Fragility, Conflict, and Violence Prevalence of sexual- and gender-based violence (SGBV) is high and common across the globe and it is especially acute in situation of FCV. SGBV can be physical, emotional, psychological, or sexual in nature, and can take the form of a denial of resources or access to services (UNHCR 2011). Unaccompanied women and girls in FCV settings are at high risk of sexual harassment, rape, violence, and complications during pregnancy (WHO 2013). During the current refugee crisis, the increasing incidence of sexual violence, survival sex, and child marriages at Syrian refugee settlements is being reported with high prevalence of acute psychological trauma documented among girls (HRGJ, MADRE, and WILPF 2016). SGBV has severe health consequences. SGBV can have devastating impact on mental and physical health, which may include acute injuries, chronic pain, gastrointestinal illness, gynecological problems, depression, posttraumatic stress disorder, and substance abuse. Violence is also linked to the risks of partners contracting HIV and other sexually transmitted infections, as well as the risk of attempting and/or completing an abortion (WHO 2013). In addition to the physical and mental health implications, SGBV has high social and economic costs, because of associated stigma and Independent Evaluation Group | World Bank Group 95 discrimination, barriers to participate in social and economic activities, and other cultural and social prejudices. Children growing in households affected by SGBV show mental and physical health consequences, including higher infant mortality, lower vaccination rates, and lower birth weight. They face greater likelihood of experiencing physical abuse themselves, which increases their risk-taking behaviors in adolescence, including drinking, drug use, and early initiation of sex (World Bank 2015). The portfolio of the evaluation identified few integrated health and SGBV projects. The Great Lakes Emergency Women’s Health Project (P147489) is the first World Bank Group project in Africa offering counseling, legal aid, and economic opportunities to survivors of SGBV within the framework of MCH services. As part of the $107 million grant, Burundi, Democratic Republic of Congo, and Rwanda aim to promote gender equality, behavioral change and violence prevention; and help economically empower SGBV survivors while they are coping with SGBV related trauma and isolation. While boosting the access to antenatal care, skilled midwives, and modern family planning, the project uses the opportunities for early identification and referral to other support services of women at risk of violence, or recovering from SGBV. The Prevention and Mitigation of SGBV (P150651) project is a $4 million grant that complement the Great Lakes Emergency Women’s Health Project, supporting the government of Democratic Republic of Congo to address SGBV and gender inequality in eastern Democratic Republic of Congo. The additional financing of the Health System Strengthening (P157864) project aims to address gender violence in Democratic Republic of Congo with other partners in 11 provinces beyond eastern Democratic Republic of Congo, providing training and information sharing for health staff on recognition, treatment, counseling, and referral for victims of SGBV. Finally, the Uganda Strengthening Social Risk Management and Gender-Based Violence Prevention and Response (P160447) project approved on June 20, 2017 comprise a subcomponent to strengthen the health sector response to SGBV. Development Effectiveness of World Bank Group Projects Project Outcome Rating Figure B.10 shows the proportion of World Bank Group project financing, investments and advisory services rated moderately satisfactory or better (MS+). Within the evaluation’s portfolio, 71 percent of World Bank–financed projects are rated moderately satisfactory or better (MS+), which is only two percentage points below the score of the overall World Bank portfolio over the same period (73 percent). About 75 percent of IFC IS projects are rated MS+, which is a statistically significant improvement from the overall IFC portfolio of investment (57 percent). Finally, 64 percent of IFC AS projects are rated satisfactory or better, which compares with the overall IFC AS portfolio success rate of 58 percent. However, because of the limited number of IFC AS projects evaluated, the confidence interval is relatively large and the difference is not statistically significant (figure B.10). Since all World Bank–financed projects are evaluated, it is not required to construct statistical confidence interval around the mean value. On the other hand, since only a (statistically representative) sample of IFC projects is evaluated, confidence intervals are provided around the mean values of IFC projects (see appendix A). 96 World Bank Group Support to Health Services | Appendix B of World Bank and IFC Rated (Outcome Rating) Figure B.10. Percentage Moderately Satisfactory and Above, 2005–16 Source: IEG elaboration on World Bank Group portfolio. Note: IFC = International Finance Corporation; 90 percent confidence intervals for IFC. World Bank–Financed Projects Outcome Rating Figure B.11, panel a, illustrates the percentage of World Bank health projects with successful efficacy ratings (that is, rated moderately satisfactory or better [MS+]) by funding source (IBRD, IDA and trust funds). World Bank–financed projects financed by trust funds (TFs) achieve the highest ratings (74 percent), followed by IDA (72 percent) and IBRD with 69 percent. Figure B.10, panel b, shows that by region, Mena has the highest outcome ratings (88 percent) followed by Europe and Central Asia (77 percent) and Africa (73 percent) and Regional programs achieve the lowest ratings (40 percent). Figure B.11. World Bank Outcome Ratings, 2005–16 a. By funding source b. Regional differences Source: World Bank Group portfolio. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; TF = trust fund; WB = World Bank. Independent Evaluation Group | World Bank Group 97 Over time, the performance of health sector World Bank project financing has improved in absolute term and relative to the entire World Bank. The comparison between the HNP and the entire World Bank project financing portfolios showed an 11 percentage point gap for projects closed in FY02–06 (World Bank 2009, 19). The gap reached 26 points for projects closed in FY05–07, but started to decrease soon after, and in the most recent years the gap changed sign and HNP portfolio overperformed the entire World Bank portfolio from FY11–13 to FY13–15 (see figure B.12). IEG rating of M&E quality of health sector World Bank project financing has improved over the evaluation period, but opportunities for improvement are still present. IEG rating of the quality of M&E is based on an assessment of three main elements: (i) M&E design; (ii) M&E implementation; and (iii) M&E utilization. The overall quality of M&E rating is expressed on a 4-point scale (high, substantial, modest and negligible). Figure B.12 shows that in FY05–07 only 25 percent of HNP project financing had a S+ rating for the M&E, which was 13 percentage points below the entire M&E rating for the all World Bank project financing that closed in the same period. During the evaluation period, the gap between the IEG M&E quality rating for HNP projects and the entire World Bank portfolio closed progressively and turned positive in the more recent year. However, still among the majority of HNP projects that closed in FY14–16 had a modest or negligible rating for M&E quality, which indicates the existence of important margin for improvement. Bank Project Financing Outcome Ratings and M&E Quality Figure B.12. World (3-year moving average) Source: World Bank 2009 (from FY92–96 to FY02–06), World Bank, in press (from FY05–07 to FY14–16). Note: HNP = Health, Nutrition, and Population; M&E = monitoring and evaluation; RAP = Results and Performance of the World Bank Group; WB = World Bank. 98 World Bank Group Support to Health Services | Appendix B Factors Associated with Outcome Rating of World Bank Investment Projects Various studies have empirically identified correlates of World Bank financed project outcomes. The literature has identified factors at country and project levels, but it has also argued that unobserved project characteristics also drive World Bank financed project outcome ratings (see Denizer, Kaufmann, and Kraay 2013; Geli, Kraay, and Nobakht 2014; Raimondo 2016). Specifically, the literature suggests that World Bank financed project outcomes rating (the dependent variable in the regression model) is a function of the following: ƒƒ Project characteristics. Some project characteristics are observable at approval (for example, amount of commitments, number of project development objectives [PDOs]), but other can be measured only when the project is completed (for example, duration of the projects). ƒƒ Country-level factors. Also in this case, some country-level variables are time invariant (for example, the region where the project is implemented), but other are time-variant, thus can be observed only at project completion (for example, GDP growth). ƒƒ Unobservable factors human factor. Among these elements, the literature has stressed characteristics of the task team leader responsible for the project (for example, task team leader experience, the supervision effort) and of the evaluator (for example, some IEG evaluator can be “tough” other “easy”). The empirical analysis comprised the 224 World Bank’s IPF evaluated projects included in the evaluation portfolio. The analysis uses IEG rating provided in the ICRR. It will focus on a binary classification of projects: those that rated MS+ and those that did not (moderately unsatisfactory or below; MU−). The empirical analysis focuses on two aspects. First, it analyzes correlates of World Bank’s IPF comprising the evaluation portfolio. Second, it uses the in-sample and out-of-sample predictive performance of empirical models relating project outcomes to project characteristics observed at project approval (see Geli, Kraay, and Nobakht 2014). Projects characteristics that are found to correlate statistically with projects outcome of World Bank IPF include: (i) the level projects commitment—larger commitment is associated with higher probability of positive of a MS+ rating; (ii) the percentage of commitment attributed to health sector theme— projects that have only health sector themes are slightly less likely to be rated MS+; and (iii) the number of PDOs—projects with three or more PDOs (which can be interpreted as an indicator of project complexity) are 18.5 percent less likely to be rated MS+. The analysis included, among country-level factors, regional dummy variables, that suggests (as already observed in figure B.10, panel b) the existence of regional difference in World Bank outcome rating. The analysis suggests that projects implemented in countries with a positive outlook (that is, higher GDP growth and higher health spending as a percentage of GDP) are more likely to be rated MS+. In addition, the total amount of World Bank Group to health services over the evaluation period is found to correlate positively with MS+ World Bank financed project outcome ratings, but the number of GPs delivering the World Bank support to health services over the evaluation period correlates negatively with project outcome. Independent Evaluation Group | World Bank Group 99 The out-of-sample predictive performance for projects that are still active suggests that that the percentage of projects rated MS+ will about five percentage points higher than the rate observed among the evaluated projects (However, the difference is not be statistically significant at the 95 percent confidence interval). However, we need to be aware of potential limitations and biases of the results, which are affected by various forms of endogeneity. Firstly, the potential inverse causality linking the independent and depended variables. Secondly, we acknowledge that the presence of unobservable variables that we are not able to control for in the analysis can produce potential misspecification biases (see Denizer, Kaufmann, and Kraay 2013). Finally, it is well-known that a World Bank–financed project outcome rating is subject to potential measurement errors (see World Bank 2016) that can bias the results. Project Development Objectives: Distribution and Ratings The evaluation grouped PDOs of World Bank–financed projects, IFC investments and advisory services into homogeneous categories: (i) improve access to health services; (ii) improve quality of health services; (iii) strengthen health system functions; (iv) improve health outcomes; (v) enhance efficiency, (vi) stimulate private sector development; (vii) promote job creation (viii) improve equity, and (ix) foster innovation. Table B.12 shows the percentage of projects that contain at least one PDOs for each PDO category by type of product. Figure B.13 shows the three-year moving average of the percentage of World Bank project financing comprising each of the six PDOs categories. Improve access was the most common PDO, identified in 47 percent of World Bank project approved in the first three years of the evaluation period (FY05– 07), followed closely by strengthen health systems and improve health, identified in about 40 percent of approved projects. The other three PDO categories (improve quality, improve equity and enhance efficiency) were much less frequent. Over time, PDOs aiming at improve access and improve quality become more frequent; PDOs aiming at strengthen health systems and improve health outcomes become less frequent, and the other two PDOS, improve equity and enhance efficiency continued to be identified in a small share of projects. In the last three years of the evaluation period (FY14–16) improve access and improve quality were the two more common PDO categories (identified in 74 percent and 38 percent of project, respectively). All the other four categories were in the narrow range between 21 percent and 7 percent. IEG rates the efficacy of each PDO (PDO outcome rating) on a four-point scale: high, substantial, modest, and negligible. The evaluation focused on a binary classification of PDO rating: those that rated substantial or better (S+) and those that were not (that is, were rated modest or negligible). Table B.13 shows the percentage of PDOs that are rated substantial or better (S+) for each PDO category. The evaluation uses the IEG rating at ICRR of World Bank–financed projects’ PDOs. Concerning IFC investments and advisory services, even if objectives are set out at approval, the standard evaluation methodology does not shed sufficient light on their achievement. Therefore, IEG performed additional analysis to assess the extent to which IFC IS achieved the intended PDOs. 100 World Bank Group Support to Health Services | Appendix B Table B.12. World Bank Group Projects with PDO Type: World Bank and IFC (percentage) World Bank Project Financing (%) IFC IS (%) IFC AS (%) PDO Total Active Closed Total Active Op. Mat. Total Active Op. Mat. Improve access 54 67 47 88 92 86 88 86 90 Improve quality 27 44 18 69 61 75 52 54 50 Strengthen health 37 27 42 3 6 0 19 24 13 systems Improve health 29 16 35 1 2 0 1 0 3 Improve equity 8 8 8 11 6 14 1 3 0 Enhance efficiency 12 13 11 8 8 7 12 5 20 Private sector devel- 0 0 0 48 55 42 30 22 40 opment Promote job cre- 0 0 0 56 67 48 3 0 7 ation Foster Innovation 1 2 1 10 16 6 6 11 0 Other  4 2 5 1 2 0 0 0 0 Note: IFC = International Finance Corporation; Op. Mat. = operational mature. Figure B.13. PDO Categories in World Bank Project Financing (3-year moving average) Note: PDO = project development objective. Independent Evaluation Group | World Bank Group 101 Development Objectives Substantial or Better (S+), by Table B.13. Project Institution and Selected Interventions (percent) World Bank Project World PDO Category Financing CCT PBF Pandemics Bank STW IFC IS IFC IS PPI Improve access 71 100 84 n. a 64 73 73 Improve quality 46 n. a 67 n. a n. a 73 78 Strengthen health 55 67 50 60 45 n. a n. a system Improve health out- 51 80 67 52 50 n. a n. a comes Enhance efficiency 45 n. a 33 n. a 50 n. a n. a Stimulate private sector n. a n. a n. a n. a n. a 38 50 development Promote job creation n. a n. a n. a n. a 50 40 Improve equity 38 n. a n. a n. a 29 25 n. a Innovation 50 n. a n. a n. a n. a n. a n. a Other 50 n. a n. a n. a n. a n. a n. a PDOs rated MS+ 56 81 65 52 49 61 67 Projects’ outcomes 71 86 77 78 63 79 74 rated MS+ Outcome-objective 15 5 12 26 14 18 7 effectiveness gap Source: Elaboration on World Bank Group portfolio. Note: CCT = conditional cash transfer; IFC = International Finance Corporation; IS = Investment Services; MS+ = moderately successful and above; PBF = performance-based financing; PDO = project development objective; PPI = public-private iteration; STW = stewardship, regulations/policy/strategy reform; World Bank–PPI: World Bank–financed projects aiming to better integrate private sector actors in the national health system, which is a subset of projects seeking PPIs. World Bank–Financed Project Development Operations Efficacy Rating Table B.14 shows the total number of rated PDOs in each category, the average number of indicators used to monitor the PDO, and the percentage of indicators that have achieved the expected target. Access, health systems strengthening, and improved health are the PDO categories with more indicators per objective (between 2.6 and 2.5 indicators per PDO). On the other hand, quality, and equity and efficiency had about 1.6–1.7 indicators per objective. Table B.14. World Bank Project Financing Development Objectives by Type Indicators Achieving Targets PDOs Rated Indicators per PDO (Out of Those Rated) PDO Type (no.) (average no.) (percent) Access 161 2.5 39.2 Health system 118 2.6 36.7 Efficiency 22 1.7 33.8 102 World Bank Group Support to Health Services | Appendix B Table B.14, continued. Indicators Achieving Targets PDOs Rated Indicators per PDO (Out of Those Rated) PDO Type (no.) (average no.) (percent) Equity 22 1.7 42.9 Improved health 111 2.5 34.0 Innovation 3 1.9 50.0 Job creation 9 1.1 69.2 Private sector devel- 12 1.4 37.9 opment Quality 57 1.6 48.7 Other 12 1.4 64.6 Total 527 2.1 39.9 Note: PDO = project development objective. IFC Investments Development Outcome Rating IFC overall health investments portfolio show better development outcomes,1 than the rest of IFC portfolio but the gap has been shrinking. Figure B.14, panel a, shows the IFC investments development outcome rating is above the overall IFC portfolio during the entire evaluation period. However, success show a decrease in outcomes over time and the gap between Health services and the overall portfolio has shrunk from 30 percent to 10 percent in the FY13–16 period (from 91 percent to 55 percent). Figure B.14, panel b, shows the specific development outcomes that are considered in assessing the overall development outcome rating of an IFC investment. IFC health sector investments perform better than the overall IFC portfolio, primarily in terms of environmental and social effects, economic and social sustainability, and project business success. Figure B.14. International Finance Corporation Investment Outcomes a. Development outcome rating over time b. Specific development outcomes Factors Related to IFC Development Outcome Ratings Successful IFC projects are associated with “chains” of medical practice groups and repeat engagement in multiple projects. Projects in India, Turkey, and Mexico had sequences of repeated successful projects. IFC supported major hospital groups in South Africa and Sri Lanka, both of which are expanding to other Regions. The decrease in performance of more recent (FY13–16) Independent Evaluation Group | World Bank Group 103 evaluated projects is attributable to different factors among them, competition with public sector, uncertainty of public payment, sponsor with poor local knowledge, poor governance and management quality. ƒƒ Competing services by public sector: political pressure and competition when the public facility is replaced by a private player. Public facility remained and competed with the client (Bosnia and Herzegovina) ƒƒ Uncertainty of public payment: government payment not quick and reliable (Bosnia and Herzegovina) ƒƒ Poor governance and management quality: (i) bypass IFC recommendation of modest investment: big cost overrun, financially unviable (India); (ii) scope alterations contributed to below expectation of operational and profitability benchmark (India); and (iii) owner shifted to much more risker venture, ignored IFC covenants (India) ƒƒ Lack local market knowledge and experience: A Singaporean sponsor invested in the Chinese specialized dental care market: hard to perform in Chinese urban market, with labor issues constraining its business ƒƒ Distress investment failed to turn around: investor consortium injected capital to failing hospital group but restructuring was not taking place due to misalignment of interest between the investor consortium and the main sponsor (Turkey). Sustained engagement made successful projects replicated and expanded, with deployment of different instruments, trying to meet customer demand for quicker, less conditioned financing. The relationship between IFC and the hospital group in India began to develop in 2005, and subsequent investments in 2009 and 2012. In its 2016 investment, IFC is seen more of a partner mixing loan and equity. IFC also contributed to energy efficiency support to the hospital group. IFC anticipates to support the hospital group’s ambition in expanding into middle- and LICs (Rwanda, Tanzania, and Zambia), while nothing has materialized to date, due to challenges in developing bankable project in these markets. IFC was effective in introducing strategic alliances. In India, a private health care provider developing an integrated health care delivery network in the Northern region of India offering primary secondary and tertiary level of services was at risk due management failings. IFC encouraged a strategic investment by South Africa’s sponsor. An injection of new equity and management expertise, which is being introduced gradually, and has to an extent helped to stabilize the business performance. Subsequently, the company has also revamped its management team in an effort to improve operational performance. There have been positive developments, such as compliance with IFC performance standards and more recent progress in construction of new hospitals. The south- south links add to the scope for further interaction between the South Africa and Indian health sector participants and the broader dissemination of best practices from a more experienced private healthcare operator. 104 World Bank Group Support to Health Services | Appendix B IFC’s support to vulnerable populations through investments has yet to yield satisfactory results. The private sector can also contribute to universal health coverage providing quality services that are affordable to the poor. IFC has made some efforts in this direction investing in clients that are committed to serve low-income patients and creating funds that piloted explicit bottom-of-the- pyramid focus. The review of IFC projects identified only four operations with an explicit equity objective. All investments, but one achieved unsatisfactory results. The main reasons cited for these results was inadequate government compensation, lack of enforcement, lack of accountability, and poor targeting (see b1). This is consistent with the literature, that has showed very limited evidence to support the notion that large-scale, bottom-of-the-pyramid models offer good prospects for extending health services to the poor (Tung and Bennett 2014). International Finance Corporation’s Support to Vulnerable Box B.1. The Populations Independent Evaluation Group assessment of International Finance Corporation (IFC) projects: In India, IFC supported a hospital network, which could treat both very poor covered under the public payment system as well as private patients. The project had a specific poverty-focused component, by including an establishment of Reach hospitals in tier 2 and 3 cities and lower-income states in India. But it was less successful than planned in reaching the poor. The weak results were attributed to the lower rates provided by the State and central government insurance programs for treating patients below the poverty line, which compressed the margin of the Reach hospitals (World Bank 2017). An investment vehicle of the Health in Africa strategic initiative experimented in moving down market with a bottom-of-the-pyramid focus. The fund did not demonstrate that such a fund could succeed on a fully commercial basis. The impact at the bottom-of- the-pyramid was only partly verified because benchmarks cover the relatively well-off clients because the income threshold included the middle-income group. An eye care company in China that delivered direct benefits to lower-income groups with the help of government subsidies achieved increased access to private medical care by reaching down into lower-income groups and raising standards of treatment. However, results to the extent or magnitude of the support might be overstated because the number of patients treated by the company is significantly lower than expected. Also, the original estimate of revenue per patient (as a measure of cost to patients) has more than doubled to $126 per patient. This could suggest that without larger subsidies, the treatment costs would presumably exclude some of the lower- income groups from treatment. IFC Additionality There is no difference in the expected additionality between open and operationally matured IFC investments. Ex ante financial additionality has remained at 100 percent whereas ex ante nonfinancial Independent Evaluation Group | World Bank Group 105 additionality has been kept at 83 percent. Within the first type of additionality, there has been a major emphasis on financial structuring with additionality present in 96 percent and 85 percent of the times for closed and active projects respectively. Ex ante nonfinancial additionality has kept its emphasis on new or better standards, but to a lesser degree as shown in its decreased trend from 78 percent to 67 percent. Knowledge and innovation is playing a major role reflected in its increase from 50 percent to 69 percent. In contrast, effective public-private risk allocation is marginally present due to a decrease from 12 percent to 2 percent. whereas new or improved regulations are no longer present in active projects (table B.15). Table B.15. Expected Additionality in International Finance Corporation Investments Projects Closed Projects Active Projects Additionality Ex Ante (no.) (percentage) (percentage) Total 124 n.a. n.a. Financial additionality 124 100 100 Financial structuring 113 96 85 Financial funds mobilization 29 27 19 Financial market comfort 55 39 52 Nonfinancial additionality 103 83 83 New or better standards 75 78 67 Knowledge and innovation 60 50 69 New or improved regulation 3 9 0 Efficient risk allocation 8 12 2 Both financial and nonfinancial 103 83 83 Ante and Ex Post Additionality in International Finance Table B.16. Ex Corporation Investments Projects, Projects with Ex Projects, Projects with Addi- Additionality of Evaluated Ex Ante Ante Additionality Ex Post tionality Achieved Projects (no.) (percent) (no.) (percent) All projects 28 100 23 82 Financial additionality 25 89 23 92 Financial structuring 25 100 22 88 Financial funds mobilization 8 32 3 38 Financial market comfort 10 40 9 90 Nonfinancial additionality 24 86 15 63 New or better standards 16 67 12 75 Knowledge and innovation 12 50 8 67 New or improved regulation 2 8 0 0 Efficient risk allocation 3 13 3 100 Both financial and nonfinancial 21 75 15 71 106 World Bank Group Support to Health Services | Appendix B Overall, IFC Investments delivers on its promised additionality, particularly on financial additionality. Of 28 evaluated projects, IFC’s financial additionality was successful most of the times, achieving a 92 percent success rate. Within financial additionality, financial funds mobilization had a success rate of only 38 percent as a result of coordination challenges, changes in funds sources, and adjustments in project scope (table B.18). IFC’s nonfinancial additionality had a lower level of success than financial additionality, achieving a 63 percent success rate. This may reflect the fact that a substantial part of financial additionality materializes at the time of commitment while nonfinancial additionality hinges mostly on planning and deploying knowledge and support during the life of the project. Over time realization of additionality of repeat interventions seem to decrease. IEG reviewed 13 interventions with five repeat sponsors to see the extent to which the additionality is in some way maintained or evolved. Results show that that realization of incremental additionality seems to diminish over consecutive operations (box B.2) Box B.2. Analysis of Five Sponsors An earlier investment in Africa focused on financial additionality such as long-term financing in local currency, which was achieved through the very first Naira loan by the International Finance Corporation (IFC) in the country. In the subsequent investment, IFC’s maintain long-term investment profile, but adding more additionality ex post through giving confidence to other investors (which was achieved by facilitating additional investors), knowledge networking and improving E&S standards. However, evidence of incremental additionalities in the most recent deal is missing, as there were no clear improvements in E&S standards from earlier periods. In the Russian Federation, a company’s IFC additionality was clear in both financial and nonfinancial areas in the earlier project but in the most recent deal, IFC additionality on financial and nonfinancial areas were unclear- sponsor wanted IFC to be out when relationship deteriorated (but fixed subsequently). Corporate governance improvement did not realize at the end. In Brazil, a company’s long-term financing has been the main additionality in all three interventions, plus industry knowledge and mobilization. However, in the first deal, b-loan mobilization did not take place. The team is currently negotiating the terms and conditions of the facility with the company, and the main challenge has been to agree in a pricing satisfactory to all parties.” Mobilization additionality may be realizing but too early to tell. A company in Turkey earlier investment concentrated substantially on financial additionality such as long-term financing and financial funds mobilization while providing confidence to the company in their decision to purchase a stake in the company. However, the companies’ subsequent investment provided financial Independent Evaluation Group | World Bank Group 107 Box B.2, continued. additionality only on financial structuring and financial market comfort, placing a greater focus on nonfinancial additionality by improving standards of care at the company’s hospitals, bringing knowledge and innovations to operations, and allocating public and private sector risk. Despite these efforts, IFC did not affect directly the medical technology available at the hospitals nor did risk allocation could be attributed directly to IFC’s efforts. In India, additionality focused on providing financial additionality through financial structuring, except for the first one that also included market comfort, and nonfinancial additionality through knowledge and innovation. IFC played a major role by making investments in 2009 & 2010 to support the expansion of its medical infrastructure, purchasing cutting-edge medical equipment and expansion of the company’s REACH hospitals in underserved locations. In its recent investment in 2016 IFC is perceived to be more like a partner and the investment is a mix of both equity and debt. IFC Advisory Services Development Effectiveness The development effectiveness of IFC AS supporting health sector is only slightly below the overall IFC Advisory Services portfolio (57 versus 58 percent, table B.18). However, the impact achievement among health IFC AS is lower than the rest of IFC AS portfolio (8 versus 33 percent). Impact Achievements are low because at the time of project closure it is too early to capture any PPP impact Table B.18. IFC Advisory Development Effectiveness Health Services IFC Overall (percent) (no.) (percent) (no.) Development effectiveness 57 14 58 568 Efficiency 57 14 62 603 Strategic relevance 71 14 75 633 Output 86 14 83 633 Outcome 71 14 58 580 Impact 8 6 33 336 IFC role and contribution 86 14 79 633 Note: IFC = International Finance Corporation. All but two evaluated projects are PPPs for which success is measured in terms of bringing PPP projects to commercial closure. Success in bringing PPP’s to commercial closure is due to government commitment and public support to succeed. Unsuccessful projects were mostly due to lack of government, lack of government financial capacity to undertake PPPs. However, role and contribution is relatively high mostly given the impact potential that these PPPs have. For example, in Romania, IFC played a strong role in terms of project credibility and transparency—investors took 108 World Bank Group Support to Health Services | Appendix B part in the tenders primarily because IFC had prepared the project. In Mexico, it was the first PPP to go beyond building infrastructure and maintenance. It expanded private sector participation into equipment, supplies and some services incorporation environmental sustainability. In Brazil, IFC supported the first PPP in the health sector in a “frontier” and pioneering project. Analysis of Development Assistance for Health Data The evaluation used OECD-CRS data to explore the evolution of Development Assistance for Health (DAH) commitments and identify patterns of complementarity and substitution among the World Bank Group and other donors (that is, multilateral organizations, regional development banks, bilateral agencies, and foundations). A priori it is expected to observed a different pattern in the DAH trends for UMIC, LIC, and LMIC, associated with the capacity of UMICs to absorb more debt. Aid agencies are thus expected to experiment a more competitive market in UMIC countries than in LMIC countries, where aid is received in the form of grants without involving debt sustainability issues. Figure B.15 compared World Bank Group’s health portfolio data and CRS data sources. Commitment levels and trends based on both data sources are similar, especially for UMIC countries, suggesting that it would be safe to assume that aid commitments of the various agencies would not be not biased due to lack of the data. Therefore, health portfolio data is used for the World Bank Group and OECD-CRS data for the rest of donors. Figure B.15. Total International Finance Corporation and World Bank Support in Health a. Upper-middle-income countries b. Low-income and lower-middle-income countries Source: Portfolio and DAH-CRS data and elaboration on World Bank Group portfolio and OECD-CRS. Note: Two-year moving averages. DAH = Development Assistance for Health; IFC = International Finance Corporation. DAH provided to upper-middle-income countries show a clear spike in the years 2009–11 as a response to the global financial crisis, which is in large part driven largely by the surge in World Bank Group support to health services that reached $2.5 billion of commitment in the year 2010. After 2010, the donors that historically played the main role such as Bilateral and Multilateral agencies, and Independent Evaluation Group | World Bank Group 109 especially World Bank Group, decreased their weight within UMIC countries substituted by regional development banks, especially in Latin America and the Caribbean. A closer look at the World Bank Group compared with regional development banks shows that the World Bank’s loss of lending market shares has been partially compensated during 2015 by IFC lending (see figure B.15). Since 2013 there has been a constant increase in the level of aid committed by the World Bank Group. Aid provided to LMICs during the same period is much less volatile. The World Bank Group support to health services has been stable between $1 billion and $2 billion of commitment approved per year. However, commitments from bilateral donors and multilateral agencies are much larger, and there is sustained increase in the support provided by the private sector whose main donor is the Bill and Melinda Gates Foundation (figure B.16). Comparing LMIC and UMIC countries it appears to be some degree of substitution in World Bank Group support for health during the 2010–12 period, when there is a sharp increase in commitments in the former countries and a decrease in resources provided in the latter group of countries. Support in Health Toward Upper-Middle-Income Countries Figure B.16. Total Provided by Different Agencies a. Cumulative b. Absolute values Note: B&M = Bill and Melinda; dev. = development. 110 World Bank Group Support to Health Services | Appendix B Figure B.17. Total Support in Health Provided by Different Agencies a. Cumulative b. Absolute values Note: B&M = Bill and Melinda; dev. = development. References Denizer, C., D. Kaufmann, and A. Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Cor- relates of World Bank financed project Performance.” Journal of Development Economics 105: 288–302. Geli, P., A. Kraay, and H. Nobakht. 2014. “Predicting World Bank financed project Outcome Ratings.” Policy Research Working Paper 7726, World Bank, Washington, DC. HRGJ (Human Rights and Gender Justice), MADRE, and WILPF (Women’s International League for Peace and Freedom). 2016. Human Rights Violations Against Women and Girls in Syria. New York: HRGJ, WILPF) and MADRE. Raimondo, E. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank–Fi- nanced Project Performance?” Policy Research Working Paper 7726, World Bank, Washington, DC. Tung, E., and S. Bennett. 2014. “Private Sector, For-Profit Health Providers in Low- and Middle-Income Countries: Can They Reach the Poor at Scale?” Globalization and Health 10 (1): 52. UNHCR (United Nations High Commissioner for Refugees). 2011. Action against Sexual and Gender-Based Violence: An Updated Strategy. UNHCR Division of International Protection. Geneva: UNHCR. WHO (World Health Organization). 2013. Global and Regional Estimates of Violence Against Women: Prev- alence and Health Effects of Intimate Partner Violence and Non-Partner Sexual Violence. Geneva: World Health Organization. Independent Evaluation Group | World Bank Group 111 World Bank. 2010. Better Health for Women and Families: The World Bank’s Reproductive Health Action Plan 2010–2015. Washington, DC: World Bank. ———. 2015. Violence Against Women and Girls Resource Guide: Health Sector Brief. Washington, DC: World Bank ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Inde- pendent Evaluation Group. Washington, DC: World Bank. ———. 2018. Results and Performance of the World Bank Group 2017. Independent Evaluation Group. Washington, DC: World Bank. 1 Evaluation of private sector projects assesses the project’s development outcome based on four development dimensions, relative to what would have occurred without the project, and compared against established benchmarks as well as stated objectives. Evaluation selection is based on projects that reached “early operating maturity,” as defined by the Good Practice Standard. This usually happens about five years after the project’s approval, when IFC has received at least one set of audited annual financial statements that covers at least 12 months of operating revenues (MDB, ECG, WGPSE 2006, p.2). 112 World Bank Group Support to Health Services | Appendix B Appendix D. Analysis of Service Delivery and Behavior Change The frameworks of the Independent Evaluation Group (IEG) for the evaluation of service delivery and behavior change provide a comprehensive set of factors, issues, and variables that needs to be considered in the design of projects providing support to health services. The service delivery framework, illustrated in figure D.1, builds on 2004 World Development Report Making Services Work for the Poor (World Bank 2003). The framework identified accountability (across and between citizens, government, and providers) as the critical condition for services to benefit the poor. It is about service delivery, but more important, it is a framework for evaluating service delivery, rather than just operationalizing it. It looks upstream to contextual factors, including political economy consideration, budgeting and regulatory arrangements. It also covers more immediate inputs such as understanding of and knowledge about would-be service users, the adequacy of human resource inputs, and the service delivery model before looking downstream to service outputs and outcomes. Critically, the entire framework is overlaid with observation of the extent to which the views, needs and wants of would-be users, referred to as citizen beneficiaries, are considered (Caceres et al. 2016). The IEG framework for behavior change—also called CrI2SP that stands for Communication, Resources, Incentives and Information, Society, and Psychology—is presented in box D.1. The framework focuses on targeted, demand-side interventions aimed at inducing behavior change to support achievement of development outcomes. It is used to questions whether World Bank Group projects identified beneficiaries and whether diagnostic work was undertaken to learn what factors influence people’s current behaviors (for example, service use) and to understand barriers to achieving a project’s desired outcome. The framework categorizes interventions or activities that target individuals’ behavior (demand-side interventions) and analyzes the results framework to assess the design and implementation of behavior change interventions. Finally, IEG’s behavior change framework captures the reporting of outputs and outcomes associated with behavior change activities and behavior changes. Monitoring and evaluation can help provide feedback during implementation and provide lessons learned for future project design (Flanagan and Tanner 2016). The frameworks suggest that the identified factors need to be addressed in the design of the projects to ensure that the desired results are obtained. Therefore, regardless of the desired service delivery or behavior change, the World Bank’s project is expected be aware of the factors that need to be addressed. IEG acknowledges the improbability that any individual World Bank financed project would cover, or be responsible for, the entire range of issues set out in the frameworks. Ideally that understanding and awareness should be set out in project planning documentation to demonstrate the full extent of due diligence—although we acknowledge that its absence from project planning documents does not indicate nonexistence. 125 Figure D.1. Framework for Evaluating Service Delivery Source: Caceres et al. 2016. Box D.1. Framework for Evaluating Behavior Change Identify and Diagnose CrI2SP is designed to help evaluators systematically assess the degree to which projects identify beneficiaries’ behaviors and diagnose barriers to adopting a desired behavioral outcome. Understanding how contextual factors influence behaviors is a significant factor in project design. Diagnostic work is generally conducted prior to project design, although it is also common for projects to conduct diagnostic work during implementation. Examples of diagnostic work include discussion of relevant sector issues in the project appraisal document, analytical services and advisory work, incorporation of lessons learned from a prior project, a knowledge, attitudes, and practices study, beneficiary analysis, focus groups, surveys, or field visits. Design and Implement CrI2SP is designed to identify explicit behavior change objectives, and to design and implement behavior change interventions relative to barriers previously identified. The relevance of project design to observed behaviors is assessed based on the set of project activities as a whole. Activities directly targeting individuals’ behaviors 126 World Bank Group Support to Health Services | Appendix D Box D.1, continued. set of project activities as a whole. Activities directly targeting individuals’ behaviors are categorized by CRI2SP components: communication, resources, information and incentives, society, and psychology. Questions evaluators might ask include the following: ƒƒ How was information communicated to beneficiaries? ƒƒ Were resources provided? ƒƒ What incentives or information was provided to beneficiaries? ƒƒ How were societal dynamics used to shape peoples’ behaviors? Monitor and Evaluate CrI2SP enables a systematic assessment of the extent to which projects monitor and evaluate behavior change to ensure midcourse corrections are made when needed and project designs are improved. The evaluation protocol collects data on the indicators identified to measure behavior change interventions, including outputs and behavior change outcomes. The distinction between outputs and outcomes is critical for determining the efficacy of behavioral interventions. Outputs include indicators such as the number of radio broadcasts, community events or home counseling visits, whereas outcomes track changes in behaviors such as breastfeeding, handwashing, and public transit use. Source: Flanagan and Tanner 2016. How the Framework Was Used in the Evaluation The protocols developed for behavior change and service delivery were applied to a sample of World Bank and IFC projects to answer to the evaluation question “To what extent is information on behavior change and service delivery presented and operationalized in project appraisal documents (and completion reports)?”1 Information on the extent to which project documents conceptualize and provide information on key behavior change and service delivery concepts was extracted from a randomly selected sample of 75 World Bank IPF projects. Ten IFC (advisory and investment) operations were randomly selected and reviewed for service delivery.2 In addition, the World Bank’s Implementation Completion and Results Reports (ICRs) of 31 of 37 closed operations within the sample of were reviewed to determine whether (i) certain service delivery-related elements were implemented as planned, and (ii) if any findings and lessons specific to service delivery could be identified. Independent Evaluation Group | World Bank Group 127 How Is Service Delivery Described and Operationalized in Projects Supporting Health Services? Enabling Conditions In World Bank operations, policy development/regulatory and legislative frameworks were supported in 61 out of 73 projects (84 percent), and in all operations in, FCV countries (see table D.1). Examples included decentralization of health service provision, introduction of private sector participation, promotion of multisectoralism, improving equity and efficiency in health financing, and strengthening the regulatory environment for disease control. In FCV countries, the policy development focused on: financing reforms, decentralization of health services provision, and policy of free maternal and child health services. Only the World Bank’s emergency operations, Global Partnership on Output-Based Aid projects, or technical assistance projects did not include policy/regulatory support. By contrast, none of the IFC operations provided support for policy development, regulatory change, or legal changes because the scope of these operations was limited to capital investment. Table D.1. Frequency of Reference to Enabling Conditions in Sample of Health Projects Included in PAD Included in PAD World Bank Sample FCV subsample Included in IFC (n = 75) (n = 9) (n = 10) Enabling Condition (no. [percent]) (no. [percent]) (no. [percent]) Policy development / regulatory / 61 (84) 9 (100) 0 legal change Capacity development 75 (100) 9 (100) 2 (20) Note: FCV = fragile and conflict-affected situations; IFC = International Finance Corporation; PAD = project appraisal document. Capacity building was provided in nearly all World Bank operations (and subsample of FCV countries; 97 percent and 100 percent, respectively) appraisal documents reviewed. IFC operations provided capacity development for frontline service providers such as health care workers and nurses less frequently (20 percent) than World Bank operations. Recipients of capacity building in World Bank operations included central ministries and decentralized government entities in almost all projects (96 percent) and frontline workers (that is, doctors, nurses, community health or nutrition workers, or midwives; 71 percent and 78 percent in FCV countries). Capacity building typically focused on health sector management, health finance, surveillance, and provision of basic health care and treatment. Capacity building activities for frontline service providers were largely carried out as planned. Design of Operations and Service Inputs Project designs included supply-side interventions (for example, equipment; drugs and medical supplies; training of health workers; development of clinical protocols or accreditation standards to improve quantity and quality of health services) more often than demand-side interventions (health insurance programs; behavior change communications to reduce exposure to risks; conditional cash transfer [CCT] to increase beneficiary demand for health services), respectively 93 percent versus 128 World Bank Group Support to Health Services | Appendix D 66 percent. Supply-side and demand-side interventions were equally present in operation in FCV countries (eight of nine countries). All projects in FCV countries articulated the intention to target disadvantaged groups, but they were explicitly identified in the project development objective (PDO), project beneficiaries, or through key project interventions only in 42 projects (58 percent). The poor (19 projects) and women (18 projects) were most frequently identified. High-risk populations (that is, commercial sex workers, drug users) were explicitly targeted in seven projects. World Bank–financed projects are often built on quantitative data from health surveys (53 out of 73 projects, or 73 percent). On the other hand, needs analysis or social assessments are conducted less frequently (only 22 projects). Financial sustainability was primarily discussed in the context of the government’s overall fiscal space for health (for example, total health spending as a percentage of gross domestic product, level of public spending and out-of-pocket). On the contrary, plans for the maintenance of project investments were rarely presented. Cost recovery, in the form of direct user fees from beneficiaries, was only included in eight project documents (11 percent and none in FCV countries). Implementation Models Almost all projects, the health activities were managed by a governmental entity (that is, Ministry of Health or National AIDS Commission). Nongovernmental entities played a role in implementation in half of the operations (59 percent), but slightly less frequent (33 percent) in FCV countries: direct provision of health services, conduct outreach or public awareness campaigns, and pharmacy and insurance services. The role of the community in the implementation of the project (for example, provide community health workers, managing local health facilities, monitoring health services, identifying beneficiaries) was contemplated in 15 percent of projects, but more frequently in FCV countries (four of nine projects). Only 18 percent of the projects provided an explicit rationale why the implementation model and execution arrangement were chosen in the project (for example, lack of health service providers or the ongoing decentralization process). Most IFC projects (70 percent) provided a rationale for private sector involvement (for example, leverage the capacity of the private sector, limited capacity of public providers, the goal to develop a private health market). In all IFC project, health services were provided by private entities. Only one operation (10 percent) included an explicit agreement with the public sector to accept public health insurance funds. The role of the private sector is rarely viewed as an integral part of the health system, but rather is often described as to “fill the gaps” that the public sector cannot fill due to lack of funding of capacity to provide health services in sufficient quantity or quality. For example, filling a gap in specialized or high-end health services, large-scale capital financing for expanding health facilities, and management expertise to improve quality of services. Therefore, IFC operations rarely create partnerships and identifying synergies with other entities (either public or private), and more about taking advantage of existing market opportunities. Independent Evaluation Group | World Bank Group 129 Service Outputs Most World Bank operations (82 percent) strengthened monitoring and evaluation systems (that is, beyond project-specific monitoring). Half of World Bank–financed projects (slightly more, 56 percent, in FCV countries), but only one IFC operation included some form of an accountability (for example, facility report cards and third party verification processes) or feedback (for example, performance agreements, performance-based financing, or memorandums of understanding) mechanism. However, of the 38 projects that included an accountability or feedback mechanism, only 20 involved direct input from the beneficiaries (for example, user satisfaction surveys, community scorecards) (see table D.2). Table D.2. Frequency of Service Outputs in Sample of Health Projects Included in PAD Included in PAD World Bank Sample FCV subsample Included in IFC (n = 75) (n = 9) (n = 10) Service Output (no. [percent]) (no. [percent]) (no. [percent]) Inclusion of accountability or feedback 38 (52) 5 (56) 1 (10) mechanism Of which, involves citizen beneficiary input 20 (27) 4 (44) 0 Inclusion of service monitoring system 60 (82) 6 (67) 1 (10) Note: PAD = project appraisal document. Service Outcomes The results frameworks of almost all projects (68 out of 73 project, or 93 percent) included indicators to track service delivery outcomes (table D.3.) presents the frequency of beneficiary outcomes (that is, improvements in service delivery experienced by the beneficiary) and provider performance outcomes tracked in the projects. Less than half the projects tracked outcomes that were disaggregated. Thirty-one projects (42 percent) disaggregated outcomes by gender, and 11 projects (15 percent) disaggregated by poverty level. Five projects (7 percent) had outcomes disaggregated by high risk population group (see table D.3). IFC investments also planned to track beneficiary outcomes (80 percent): the number of patients treated at the health facility was the most common. Only two of projects planned to collect data disaggregated by poverty/income. Eight out of ten IFC projects that were reviewed planned to track beneficiary outcomes, most commonly the number of patients treated at the health facility (use). Table D.4. provides examples of relevant indicators contained in projects, and provides detail where there is disaggregation by beneficiary group. 130 World Bank Group Support to Health Services | Appendix D Table D.3. Frequency of Key Service Outcomes in Sample of Health projects Included in PAD Service Outcomes (no. [percent]) Beneficiaries Coverage or access 49 (67) Quality 23 (32) Affordability 6 (8) Satisfaction 16 (22) Use 21 (29) Provider Performance Qualitya 33 (45) Reliability/Timeliness 16 (22) Efficiency 7 (10) Note: PAD = project appraisal document. a. This captures indicators on quality of services provided (according to facility assessments, adherence to norms, and so on), but not quality of outcomes (as measured by health outcomes such as tuberculosis cure rate, malnutrition rates). Table D.4. Examples of Service Delivery Outcome Indicators Beneficiary General Disaggregated Coverage or Contraceptive prevalence rate Coverage of antenatal care access % population living within 5km radius of health % of poor covered by health insurance facility Quality TB cure rate Reduced prevalence of anemia among preg- Malaria prevalence rate nant women Affordability % households that identify financial barriers as Share of poor households spending 25% or main cause for not seeking health care more of nonfood consumption on health Satisfaction % patients satisfied with quality of care and % poor expressing satisfaction with health availability of services services Use Outpatient visits per capita Outpatient attendance among lowest income quintile Provider Perfor- mance Quality Adherence to treatment protocols for selected n.a. conditions % facilities without essential drugs stock-outs Reliability or Turnaround time for diagnosis less than one n.a. timeliness week Efficiency Decrease in hospital admissions n.a. Increased share of outpatient case load at community level compared with hospital Note: n.a. = not applicable. Independent Evaluation Group | World Bank Group 131 What Can Be Learned from the World Bank’s Completion Reports? The World Bank’s completion reports were reviewed to learn how contextual conditions, extent of beneficiary involvement, and the type of implementation model may have played a role in determining service delivery outcomes. Specific contextual conditions that negatively affected implementation were reported in six of the 31 ICRs. These included the lack of social appropriateness of a project intervention, political resistance to a policy change, and deterioration in security conditions that prevented health personnel from accessing target areas. An additional 9 ICRs referred to more general contextual conditions such as postconflict fragility, political transitions due to elections, or the global economic downturn. Instances where beneficiary assessment or beneficiary participation, or the lack thereof, affected project outcomes were mentioned in 11 of the 31 ICRs. Of these, six ICRs reported examples where the inclusion of beneficiaries’ perspectives improved project design such as increased relevance or appropriateness of the activity to meet real needs. The remaining five ICRs noted the lack of beneficiary participation that subsequently contributed to poorer outcomes, for example, lack of understanding of patient preference for provincial hospitals versus primary care clinics or lack of information on beneficiary willingness-to-pay levels. How Is Behavior Change Described and Operationalized in Projects Supporting Health Services? Targeted behavior change was identified in nearly two-thirds of the projects reviewed. A targeted behavior change was present included as part of the PDO in 20 projects, and behavior change activities were incorporated in project components or subcomponents in 44 cases.3 Table D.5 presents the most commonly identified interventions. of Behavior Change Interventions in Project Design Table D.5. Presence Elements (number) Category Projects PDO Component Subcomponent Output Outcome Promote healthy be- 26 9 11 25 14 16 haviors and practices for prevention and treatment Increase the use of 18 10 4 18 9 10 services Increase access to 26 1 0 0 0 0 services and improve service qualitya Total 70 20 15 43 23 26 Note: PDO = project development objective. a. The PDO for a project in Tanzania aimed to increase the use of health services. However, its components were only resource-based, focused on capacity building of local governments and financing for local service delivery (for example, for medical supplies and medicines). 132 World Bank Group Support to Health Services | Appendix D Analysis of the barriers to behavior change found the interventions design was relevant to the barriers identified (see figure D.3). However, the analysis of barriers was focused heavily on resources. Nearly all projects (69) identified resource constraints and included resource interventions.4 And about one-third of the projects (23) included only resource-based interventions. After resources, the most common barriers were information and incentives. Related interventions included health education, financial incentives, and subsidies for health care, as well as communications, such as social marketing to promote the purchase and use of condoms and media campaigns to increase vaccination. Figure D.2. Types of Barriers Identified and Interventions Designed Finally, about one-third of projects mentioned social issues as constraints (21) and interventions (26), but very few addressed psychological issues, with just five identifying them as barriers and three as interventions.5 Two projects identified psychological barriers but did not design interventions to address them. In Lesotho, an HIV project described how a large proportion of those infected are still in denial and that stigma associated with the disease still exists, but only included resources interventions, mainly focusing on institutional capacity building, along with support for civil society and private sector capacity development. Meanwhile, in China, a project aimed at improving access to and quality of care in targeted hospitals stated that “many Chinese have become persuaded to consider the location (highest level hospital) and quantum of care (volume of tests, drugs and injections received) as manifestations of ‘good’ care,” thereby leading to an oversupply of services and raising total health costs, making them unaffordable for many. However, this project, likewise, only includes resource- based interventions. Whereas interventions aimed at changing mental models (for example, beliefs that more care is good care) would have been helpful to incorporate, this project instead focused on institutional capacity building, support for hospital management and service delivery performance, training of rural health professionals, and construction of county/district facilities. Independent Evaluation Group | World Bank Group 133 Table D.6. Types of Behavior Change Interventions Used in Project Design Information and Category Communication Resources Incentives Social Psychological Promote healthy 25 26 24 18 3 behaviors and practices for prevention and treatment Increase the use of 14 17 17 8 0 services Increase access to 4 26 3 0 0 services and improve service quality Total 43 69 44 26 3 With regard to the interventions, project documents cite various types of information and communication activities, but provide little content on their design and implementation. Nearly all of the projects with behavioral interventions (39 of 44) included these an information component or subcomponent of activity.6 Although most projects do identify the desired behavioral changes and one-third specify the information or communication delivery mechanisms (for example, radio, TV, posters, films, workshops, mosque announcements), many others do not.7 In addition, half (22) of projects with behavioral interventions did not provide information on the specific target audiences for behavior change activities (for example, pregnant women and mothers, smallholder poultry farmers, sex workers, drug users, and people with disabilities) within the project’s larger target area, nor did they provide details on the messages to be delivered. For example: ƒƒ Community awareness and education campaigns to stimulate demand for quality health services ƒƒ Awareness campaigns to promote quality maternal care ƒƒ Development of an IEC program aimed at behavior change ƒƒ A behavior change communication strategy on IYCF practices, disease preventive mea- sures, home-based care, recognition of danger signs, and care seeking for sick children ƒƒ Public awareness campaigns for citizens about their rights, eligibility, and access to services ƒƒ An integrated communications strategy will elevate knowledge and promote behavior change to control the spread of the virus, prevent infection, foster timely reporting and support containment. ƒƒ A social communication strategy for purposes of informing the public with respect to, inter alia, the ongoing health promotion strategy and health reform process 134 World Bank Group Support to Health Services | Appendix D ƒƒ Information and communication activities to increase public understanding of avian influ- enza, promote safe behavior to reduce risks to communities, and promote responsible media reporting to avoid panic and misinformation ƒƒ Communication aimed at changing hygiene habits However, some projects did mention that they planned to conduct formative research to identify and tailor appropriate messages for different target groups. Although limited beneficiary-specific diagnostic work was conducted in the sample—15 projects conducted knowledge, attitudes, and practices (KAP) studies, action-research on behavioral messages and communications strategies, or other types of formative research, there are several good examples included in the sample. For instance, a TB control project in India conducted both baseline and endline KAP studies to provide a basis for the development of an IEC strategy, particularly on the most effective ways to access hard-to-reach groups, and to evaluate the effectiveness of behavior change interventions. Using the baseline KAP as a starting point, communications activities would be viewed as part of an iterative process, with a mid-term impact assessment to monitor progress of the media plan and to correct and refine the program as it evolves. Similarly, in Barbados, an human immunodeficiency virus [HIV]/acquired immune deficiency syndrome project conducted KAP research “to create and test multichannel, high-impact behavior change communication programs tailored to each of the key populations at higher risk, as well as to the general population,” and a nutrition project in Pakistan carried out formative research “to inform the development of a IYCF behavior change communications strategy, including selecting the tools and media mix.” Many of these information and communications interventions also included social aspects, and a few addressed psychological barriers. In the sample, 26 of the 39 behavioral projects with information and communications interventions used social approaches, such as messages tailored to address cultural and religious barriers, interpersonal communication such as counseling and peer communications in PLHIV (people living with HIV) groups, activities aimed at reducing stigma and discrimination, and the use of educational materials in social groups such as youth groups and community organizations. Additionally, three projects included psychological interventions— two made things easier by supporting food fortification efforts (for example, fortify salt with iodine, flour with iron and folic acid, and oil/ghee with vitamins A and D), and one made it easier to access voluntary testing and STD treatment. In addition, 26 projects included financial incentives. In total, 31 projects included financial incentives.8 Financial incentives included, for instance, subsidies for insurance premiums structured to encourage early enrollment through higher contributions in the first year (that is, contributions declined from 85 percent of the total premium cost during the first year to 45 percent in year five). Another approach used by projects in the sample includes CCTs, which aimed to ensure that poor families make regular visits to health providers, receive the maternal and infant health services package, and increase their consumption of nutritious foods. Cost savings activities, meanwhile, included the provision of support for transport costs from remote areas to district/state health facilities, gratuity of certain drugs Independent Evaluation Group | World Bank Group 135 and products such as contraceptives and insecticide treated bed nets, and the provision of a free package of maternal and child health services. In one project, for instance, vouchers were provided to poor women for free maternal and infant health services as well as family planning services, along with additional cash payments for transport and food costs for pregnant women traveling from remote areas to have facility-based deliveries. Finally, the monitoring and evaluation of this sample of health projects illustrates that it is possible to change many kinds of behaviors through targeted interventions, although some projects lacked sufficient outcome indicators, baselines, and targets to measure progress. Of the 44 projects with behavioral interventions, 36 included behavior-related output and/or outcome indicators. However, only 17 of these had behavioral outcome indicators with results reported in an ICRR.9 Of these, 12 had at least one outcome indicator with a baseline and target with information on whether or not the outcome met its target. For example, projects successfully increased health service use (for example, number of outpatient visits per capita, number of women who made at least four prenatal care visits), increased the use of modern contraceptives, increased the share of children who are exclusively breastfed until six months of age, increased condom use, immunized children for polio, improved farmers’ poultry handling practices, increased the number and share of children under five sleeping under insecticide treated bed nets, improved IYCF practices, increased the number of people with advanced HIV infection receiving treatment, and decreased the prevalence of smoking. Of the other five projects, four were missing baselines and/or targets and one project did not achieve its targets. For one of the projects that is missing a target, it only has one outcome indicator on children immunized, but its design includes awareness and education campaigns to promote family planning and use of contraceptives, use of maternal and neonatal services, and hygienic practices, but does not include any outcome indicators to track progress on these behavioral interventions. Finally, for the project that did not achieve its targets, not only did the percentage of men having sex with men reporting the use of condoms not meet its target (that is, increasing to 68 percent from a baseline of 63 percent, falling short of its target of 73 percent), but the percentage of both female sex workers and people using drugs reporting condom use ‘at last sex over the last 12 months’ decreased (for FSW from 90.1 percent to 65 percent, for PUD from 38 percent to 24 percent). Moreover, the percentage of sexually active people reporting having been tested at least once for HIV remained essentially the same, at 37 percent. 136 World Bank Group Support to Health Services | Appendix D Table D.7. Output and Outcome Indicators Behavior Change Output Outcome Promote healthy behaviors and People from at-risk groups who Target groups reporting use of practices for prevention and treat- correctly identify ways to prevent condoms ment sexual transmission of HIV and Children under 24 months benefit- reject misconceptions about trans- ing from improved IYCF practices mission Children immunized for polio Number of condoms distributed Women of reproductive age using Number of community health work- modern contraceptive methods ers trained to provide IYCF services Pregnant women and children un- Public information campaign carried der five sleeping under insecticide out treated bed nets Mothers participating in monthly Infants 0 to 6 months who were information and education sessions exclusively breastfed Informational products dissemi- Injecting drug users who adopted nated behaviors that reduce HIV trans- Pandemic communications mes- mission (for example, avoid sharing sages produced and delivered injecting equipment) Attitude of pregnant women toward Improved hygiene and sanitation the importance of eating three behaviors times per day, including at least one Adopt practices for the prevention animal-sourced food and control of avian influenza (poul- People showing evidence of high try producers, distributors, retail awareness of program messages vendors) Increase the use of services Pregnant women who receive a Utilization of outpatient / inpatient ‘Safe Motherhood voucher’ services in district hospitals Women receiving information on Percentage of facility-based deliv- human immunodeficiency virus / eries acquired immune deficiency syn- Contraceptive prevalence rate drome, during prenatal/postnatal or People age 15 and older who family planning visits received counseling and testing for Increase in state/district capacity to HIV, and received their test results plan and execute IEC Pregnant women who are HIV posi- Percent of the near-poor covered tive who receive a complete course by the health insurance program of antiretroviral prophylaxis Beneficiaries who know their rights Number of visits to health facilities to access services and their co-re- Children under two registered with sponsibilities complete immunization Number of municipal programs Children under two that are for social communication in health weighed according to requirements approved Key Findings of the Service Delivery and Behavior Change Analysis The design of the projects analyzed included supply-side (that is, service delivery aspects) more often than demand-side (behavior change aspects), respectively 93 versus 66 percent. Supply-side and demand-side interventions were equally present in operation in FCV countries (eight of nine countries). World Bank operations delivering health services usually recognize and operationalize need for upstream support such as policy development and capacity building for government agencies and service providers. Nongovernmental entities played a role in implementation in half of the operations Independent Evaluation Group | World Bank Group 137 (59 percent). The role of nongovernmental organization and community-based organizations in the operationalization of services delivery is more frequent in FCV. Although most projects do identify the desired behavioral changes and one-third specify the information/communication delivery mechanisms (for example, radio, TV, posters, films, workshops, and mosque announcements), many others do not (only one-third, or 13 of the 43 projects with communications interventions and 13 of 40 for projects with information interventions, specify the delivery mechanisms). In addition, half (22) of projects with behavioral interventions did not provide information on the specific target audiences for behavior change activities (for example, pregnant women and mothers, smallholder poultry farmers, sex workers, drug users, and people with disabilities) within the project’s larger target area, nor did they provide details on the messages to be delivered. The analysis of barriers to health services focused heavily on resources, followed by information and incentives. About one-third of projects mentioned social issues as constraints (21) and interventions (26), but very few addressed psychological issues, with just five identifying them as barriers and three as interventions. World Bank–financed projects appear to be built on sound data foundation, but there was insufficient attention to needs/preferences as expressed by beneficiaries. Within the subset of FCV countries, more attention was placed on targeting and conducting beneficiary assessment. This may be a missed opportunity to improve design and implementation, as several of the World Bank’s completion reports noted lack of participation from beneficiaries subsequently contributed to poorer outcomes. The issue of sustainability of health services is usually framed in term of fiscal space and fiscal sustainability. However, operational aspects related to maintenance of the service delivered are rarely addressed. The application of the IEG service delivery and behavior change frameworks found that the rationale provided for IFC IS projects in the health sector emphasized how they are reacting to existing market opportunities and included far less discussion about how the project contributed to health systems in which they occurred. Most project’ monitoring and evaluation frameworks tracks outcomes experienced by citizen beneficiaries when related to coverage, access and use. Quality of health services is monitored more frequently from the providers’ perspective compared with beneficiaries (45 and 32 percent, respectively). Satisfaction, reliability/timeliness, affordability, and efficiency are monitored more rarely. References Caceres, S. A., R. Yin, A. Tyrrell, J. H. Gaubatz, and S. Williams. 2016. “Introducing a Framework for Evaluating Services Delivery in Sector Evaluations: Urban Transport, Water and Sanitation, and Nutrition.” Independent Evaluation Group Working Paper 2016/3, World Bank, Washington, DC. 138 World Bank Group Support to Health Services | Appendix D Flanagan, A. E., and J. C. Tanner. 2016. “A Framework for Evaluating Behavior Change in International Development Operations.” Independent Evaluation Group Working Paper 2016/2, World Bank, Washington, DC. World Bank. 2003. “World Development Report 2004: Making Services Work for Poor People.” Washing- ton, DC: World Bank. 1 AAA, Policy Loans, or Additional Financing were not reviewed. The random sample drawn for this exercise was 75 operations out of core operations contained within the Health Services evaluation. 2 The criteria for the subset of IFC operations: investment operations, evaluated/nonevaluated, sector: hospital and specialized, which resulted in 42 operations- from which 10 were randomly selected. The operations in the sample took place in the Europe and Central Asia, Latin America and the Caribbean, East Asia and Pacific, Middle East and North Africa and South Asia Regions in the following countries: the Russian Federation, Mexico, China, Republic of Yemen, Arab Republic of Egypt, Bosnia and Herzegovina, India, and Tunisia. 3 Eighteen of the 43 projects with behavioral interventions have behavior change objectives (project development objectives); 2 additional projects have project development objectives aimed at increasing the use of services but no behavioral activities mentioned in their components or subcomponents. 4 One project did not identify resources as a barrier due to the limited context in the project appraisal document on the health sector, as it aimed to improve local service delivery generally (for example, health, education, water). This project identified communications and information/incentives as constraints, and interventions were resource- and communications-based. The other project identified enabling resources as a constraint, but only focused interventions on information/incentives and communications (for example, subsidies, marketing via radio). 5 These three projects were the only ones to design interventions covering all five types, although none of them identified all barriers. 6 Behavior change projects without information and/or communication activities include four projects focused on increased use of health services and one project on promoting healthy behaviors. Instead, these projects’ interventions focused on a combination of resources (for example, capacity building both for institutions and health workers, infrastructure, equipment, drugs, vaccines) and financial incentives and cost savings (for example, subsidies, vouchers, cash payments for transport). In addition, five projects that did not include behavioral interventions, and instead focused on access to services, did included information/communications interventions (for example, for general awareness and understanding about the project). 7 One-third, or 13 of the 43 projects with communications interventions, specified the delivery mechanisms; 13 of 40 for projects with information interventions. 8 Five projects developed both financial incentives and cost savings interventions. 9 Fifteen did not have ICRRs, and four reported only on outputs (for example, launch campaign, increase awareness and knowledge) Independent Evaluation Group | World Bank Group 139 Appendix C. Assessment of Global Partnerships Programs Partnerships are arrangement for collective action between legally autonomous organizations that typically involve dedicated funding and common objectives. Partnership programs are one of the key instruments through which the World Bank Group engages with other development partners at both country and global levels. The World Bank’s support to partnership programs dates back four decades. The first World Bank health partnership was the African Programme for Onchocerciasis Control created in fiscal year (FY)73 to fight onchocerciasis (also known as river blindness). This assessment covers relevant global partnership programs (GPPs)1 and multi-donor trust funds (MDTFs).2 The review assesses the nature and evolution of the World Bank Group’s GPPs and MDTFs, their relevance and alignment to the World Bank Group’s strategic and corporate priorities, the role the World Bank Group plays in these programs. The methods used to collect and triangulate the evaluative evidence consisted of a construction and analysis of portfolio of the World Bank Group’s health GPPs and MDTFs, semistructured interviews with program managers, and development partners, and in-depth analyses of two partnership programs—Health in Africa Initiative (HIA) and the Health Results Innovation Trust Fund (HRITF). The HRITF is selected as the largest multi-donor trust fund program in health housed in the World Bank, while the HIA program is the first IFC-led comprehensive initiative in the health sector to enable private sector participation in African countries. The in-depth case studies are analyzed by adapting IEG’s evaluation framework for assessing global and regional partnership programs (World Bank 2007a, 2007c). The Changed Global Health Landscape The global health landscape and so the role of multilaterals, have changed drastically in the last two decades due to number of factors: ƒƒ The MDGs led to proliferation of health actors and partnerships and the Sustainable Development Goals (SDGs) are likely to lead to their further proliferation, putting aid co- ordination and harmonization at front and center of development assistance in health. ƒƒ New nontraditional donors, especially the Bill and Melinda Gates Foundation, have gained importance, with a priority focus on targeted health issues, such as communica- ble diseases, rather than investing in broader health objectives and systems. For exam- ple, 17 of 25 World Bank Group GPPs receive funding support from the Bill and Melinda Gates Foundation. ƒƒ Two large intervention or disease-specific funds—the Global Fund and Gavi, the Vac- cine Alliance—have emerged, both established with strong support by the World Bank Group; they have very different operating models than the multilaterals. ƒƒ Significant increase in noncore resources that bilateral donors or private foundations provide to multilaterals, including the World Bank, has allowed to broaden the scope of support in health, but brought challenges of further fragmentation. The World Bank is the second largest recipient institution of such funds (OECD 2015). 113 Inside the World Bank Group, some internal factors continue to shape the way the World Bank Group partners: ƒƒ The closing of the World Bank’s own grant- making facility, the Development Grant Facility,3 in 2012 has left the HNP GP and the regional VPUs with a funding shortage and a lack of strategy to deal with their former partnership commitments. The largest share of the $2.25 billion Development Grant Facility funding in FY98–13 period, supported programs in HNP—15 percent ($337 million) (World Bank 2012). ƒƒ The restructuring of the World Bank Group in 2014 has brought the World Bank’s and IFC partnerships under one umbrella. However, the incentives and mechanisms of inter- nal coordination do not seem to be aligned well, leading to weak ownership of some of the IFC programs. Box C.1. Multilateral Institutions and Vertical Funds There is a growing debate in the literature about the impact of increased use of noncore resources by multilateral institutions and the growing role of vertical programs, such as the Global Fund and Gavi, in the global health landscape. These are often seen as parts of the same trend. For some, the sharp increase of noncore funds to multilaterals is coming from the donor’s desire to more closely engage and monitor the delivery of the programs. Thus, through earmarked contributions, bilateral donors gain more influence to shape the priorities of multilateral organizations and their budgets, bypassing “purely multilateral” governance whereby decisions are made by all members per collectively endorsed rules (OECD 2015; Clinton and Sridhar 2017). Some find this dovetailing of the vertical funds with an increase in noncore earmarked funding to multilateral organizations as worrisome. The sharp increase in earmarked funding to the WHO and World Bank has replicated features of the vertical funds. Strong reliance on noncore funding, which is largely earmarked, shapes incentives for these institutions and can lead to diversions in priorities. In other words, these additional funds, which come in the form of noncore contributions to multilaterals, like World Bank and WHO, can undermine both the capacity of international organizations to deliver much-needed cooperation and the delivery of collective action in global health (Sridhar and Woods 2013). The World Bank Group’s Engagement in Global Partnership Programs Trends The World Bank Group is currently involved in 25 global partnership programs (GPPs) and six MDTFs in health. IFC has been a founding member of three GPPs—HIA, HANSHEP and Global Health 114 World Bank Group Support to Health Services | Appendix C Investment Fund (co-investor). About half of these 25 programs provide country-level investments and technical assistance (for example, Gavi, the Vaccine Alliance, Global Fund, Polio Buy-Down, Global Financing Facility), some finance research and development in disease-specific areas (for example, the International AIDS Vaccine Initiative), while many also help generate knowledge, and provide platform for advocacy—for example in health systems strengthening, maternal and child health, and nutrition (Scaling Up Nutrition, PMNCH). There are also network partnerships aimed to strengthen aid coordination and harmonization of development partner practices at country level (IHP+, HANSHEP). These 25 partnerships have disbursed more than $35 billion for health services since 2010. World Bank Group’s current global partnership engagements can be grouped in three: ƒƒ Early programs, dating back to 1990s and early 2000s, primarily supported via Develop- ment Grant Facility: Medicines for Malaria Venture, International AIDS Vaccine Initiative, RBM, Stop TB Partnership, TDR, UNAIDS, and so on ƒƒ Major Financial Intermediary Funds, such as the Global Fund and Gavi, the Vaccine Alli- ance, external to the World Bank, in which the World Bank plays multiple roles ƒƒ New or revamped programs to target SDGs, such as the Global Financing Facility, PMNCH or IHP+, where the focus is not only on investments but also on improving aid coordination and knowledge sharing. The World Bank Group’s current partnerships are broadly in line with its sectoral strategies but there is room for better selectivity and alignment. The World Bank Group’s global engagements in health are expected to align to strategic priorities articulated in the World Bank’s 2007 HNP strategy and updated in 2016 (World Bank 2007b, 2016). The strategy called for increasing selectivity, improving strategic engagement, and reaching agreement with global partners. The 2013 World Bank Group strategy has made a compelling case for strategic partnerships, committing to (i) deepen World Bank Group role in promoting partnerships; (ii) ensure strategic alignment of partnerships with the twin goals; and (iii) make provisions for partnerships to be adequately resourced and managed. The number of GPPs has decreased from 34 in 2007 to 25 in 2016 (World Bank 2013).4 Since 2007, seven GPPs ceased activities, the World Bank Group disengaged from 16 that were generally not central to its priorities or were already mature, maintained its engagement in 10, and engaged in 14 new ones. Current GPPs, however, have mixed relevance to the World Bank’s sector and corporate priorities and the quality of the World Bank’s engagement varies. The partnership with some early GPPs has become notably weaker due to their diminished relevance and limited resources to engage with them. The closure of the Development Grant Facility has left the World Bank’s participation in number of partnership programs uncertain or made it notably weak. Only in few cases, such as the Roll Back Malaria, and Medicines for Malaria Venture—once the funding stopped, the World Bank automatically disengaged, since it was not a board member. In programs, like TDR, African Programme for Onchocerciasis Control, Stop TB, and HRP, while stopping to contribute financially, the World Bank is still involved in their governance, often at the request of other partners and donors. Independent Evaluation Group | World Bank Group 115 Some of the programs in the World Bank Group’s GPP portfolio have overlapping mandates and objectives because of changes in the global health landscape. However, it is rare that an existing GPP closes or merges when new, larger programs with similar objectives are created, even if they have common stakeholders. There are also GPPs that may have become less relevant to the World Bank Group due to shifts in the World Bank Group’s corporate and sector priorities.5 Stop TB Partnership’s mission, for example, which had achieved notable results through its Global Plans to Stop TB, has some overlaps with that of the Global Fund to Fight AIDS, Tuberculosis and Malaria. After the establishment of the Global Fund, the intensity of the World Bank’s engagement with Stop TB waned. Evolution of the World Bank Group’s Engagement in Earlier Table C.1. Recent Health Global Partnership Programs World Bank Withdrew Weakened Partnership Remained the Same since 2014/Program with the World Bank but Will Diminish as the Remained the Same or Closed Group Funding Halts Became Stronger RBM, MMV, GAIN, MTA, HPR, TDR, APOC, Stop IAVI, UNAIDS Global Fund, IHP+, Gavi, GFHR, HMN TB Partnership the Vaccine Alliance Note: APOC =African Programme for Onchocerciasis Control; IAVI = International AIDS Vaccine Initiative; MMV = Medicines for Malaria Venture; TDR = Special Programme for Research and Training in Tropical Diseases; UNAIDS = Joint United Nations Programme on HIV/ AIDS. The WHO-housed Special Programme of Research, Development, and Research Training in Human Reproduction (HRP) was highly relevant in early 90s as one of the few engaged in reproductive health research in low-income countries. The World Bank’s co-sponsorship to the program was instrumental in bringing the program to country level. However, it is worth exploring to what extent the research that the program conducts on a global scale to improve sexual and reproductive health informs the World Bank Group’s support in this area. What is the value-added of the program for the World Bank Group on the top of its participation in the Partnership for Maternal, Newborn & Child Health (PMNCH)—an umbrella partnership to improve health outcomes in reproductive, maternal, newborn, and child health in line with the UN Global Strategy for Women’s, Children’s and Adolescents.’ The World Bank’s longstanding collaboration with the International AIDS Vaccine Initiative (IAVI) that has an objective to accelerate development of effective acquired immune deficiency syndrome vaccines has been revived in 2010 through support from Japan to supervise the implementation of a project “to support IAVI in the development of a novel vaccine based on the Sendai virus Vector.” The World Bank provided excellent financial and technical oversight and the project’s achievement of outcome is rated highly satisfactory. However, is the World Bank well placed and has the technical capacity to engage in vaccine research and development or other partners are better placed to take up this role? Would the World Bank continue its partnership with IAVI after the donor-support is terminated? In the last few years, World Bank Group’s new global engagements in health are relatively fewer and reflect how the World Bank’s priorities adjust to tackle global challenges. During the 2012–16 period, the World Bank Group has engaged only in seven new GPPs and established four MDTFs. These new programs are to help strengthen country health systems, improve private sector participation, 116 World Bank Group Support to Health Services | Appendix C improve services in maternal, neonatal and child health (for example, the Global Financing Facility [GFF]), improve data measurement and collection in health service delivery (for example, Service Delivery Initiative), and respond to epidemics (Ebola MDTF and Pandemic Emergency Financing). World Bank Group’s Role and Effectiveness The World Bank Group plays different roles in GPPs—a founding and governance partner, a host, trustee of donor funds, and development partner at country and global levels. The World Bank’s coordination role has been critical at formative stages of number of health partnerships to mobilize collective action and funding around the objectives of those programs. Often in collaboration with WHO, the World Bank has been a founding partner helping lunch many important programs supporting global and national public goods, such as HRP, UNAIDS, Gavi, the Vaccine Alliance, Global Fund, Stop TB, and Joint Learning Network, for universal health care. The two largest FIFs in health—the Global Fund and Gavi, the Vaccine Alliance—were established with strong support by the World Bank in early 2000s, and continue to rely on the World Bank’s fiduciary capacity. The World Bank remains the trustee of donor funds supporting both programs. World Bank helped set up and manage two innovative financial vehicles (International Finance Facility for Immunization and Advanced Market Commitments) that provide Gavi, the Vaccine Alliance with significant and predictable resource flows for immunization. These two mechanisms have contributed 30 percent of Gavi, the Vaccine Alliance’s financial resources in the period of 2000–10. The intensity of the World Bank’s participation in the governance of some GPPs has changed in recent years. It is represented in the governing body (at board or committee levels) of 22 of 25 health GPPs.6 Participation in governance allows it to reach a wide range of stakeholders. Often being constituency based, the governing bodies of the partnership programs are inclusive by nature. They include not only traditional donors and client governments, but also other multilateral banks, UN entities, and nonstate actors such as civil society organizations, the private sector, private foundations, and various beneficiary and interest groups representing a broad range of constituencies. There is limited evidence on how well does the World Bank Group performs its role(s) on the boards of health partnerships. Prior IEG reviews found that the World Bank performed its role effectively on the boards of the Global Fund and Gavi as well as smaller programs, such as Stop TB, but its representation would have benefited from corporate guidelines (World Bank 2012, 2014c). In recent years, the World Bank’s participation has weakened in some formerly Development Grant Facility–funded programs as the interviews with the World Bank representatives in these programs indicated. This is partly because these programs provide little additionality and partly because of reduced funding to participate in their board meetings. However, the World Bank continues to stay involved in their governance often at the request of other partners and donors. No systematic selectivity exercise has been carried out to align the GPPs with the World Bank Groups’ current goals and priorities. The World Bank Group helped gear global initiatives toward SDGs, also aiming to improve coordination among partners. The World Bank lunched in 2015 the GFF, a key financing platform Independent Evaluation Group | World Bank Group 117 of the UN Global Strategy for Women’s Children’s and Adolescents’ Health 2016–30 (UN 2015). In collaboration with UN partners, the World Bank has been key in transforming the PMNCH into an umbrella partnership that improves aid coordination around maternal and childcare. Jointly with WHO the World Bank helped transform the IHP+ into a multistakeholder platform to promote collaborative working, in countries and globally, on health systems strengthening (now UHC2030). To support the Scaling Up Nutrition Movement (a global advocacy network, which is successful in advocacy but made little progress in mobilizing resources), the World Bank has started to develop costed plans for nutrition in number of countries aimed to bridge the gap between the advocacy and investments. The World Bank also has become an implementing partner for the Power of Nutrition partnership in 2015 together with UNICEF and the funding will be used to cofinance IDA projects. The World Bank can use its coordination role to further improve integration of nutrition in broader health investments particularly those directed to maternal and child health. The World Bank Group is also supporting a collective action to fight against global pandemics. Based on the lessons from the Ebola outbreak in West Africa, and building on the experience of coordinating global response to avian flu,7 the World Bank has put in place a new Pandemic Emergency Financing Facility in 2016, an innovative, fast-disbursing financing mechanism designed to create the first- ever insurance market for pandemic risk for low-income countries, to prevent the outbreaks from becoming pandemics. The pilot mechanism still has some challenges to overcome. For example, to what extent its insurance model, which has been informed by successful catastrophe risk insurance instruments that the World Bank helped to establish in middle-income countries, could become financially viable in IDA countries, where the risk that the member countries may not be able to pay their insurance premiums is higher. Aside from direct cofinancing with development partners, the World Bank also has an implementation role in some external GPPs that provide country-level investments and technical assistance. These programs include the Power of Nutrition, the Sahel Women’s Empowerment and Demographic Dividend, the Joint United Nations Programme on HIV/AIDS (UNAIDS), and Gavi. The Polio Buy-Down program, for example, mobilized significant resources to eradicate polio in Nigeria, and Pakistan since the early 2000s through a credit “buy-down” program. An improved program, with a sharper focus on results, attracts more development partners. The latest polio project in Pakistan (the Pakistan National Immunization Project), with an IDA credit of $50 million, attracted $94 million in cofinancing (Gavi, $84 million; USAID, $10 million), while the Bill and Melinda Gates Foundation contributes to a buy-down. The new implementing partner arrangement with Gavi since 2015 is another example where the World Bank uses its operational role for leveraging resources and aligning priorities. The expansion of its operational role can be another venue to enhance the benefits from the partnerships by leveraging resources, sharing solutions, and aligning donor priorities at country level. GPPs housed in the World Bank Group reflect its comparative strengths. Eighteen of 25 GPPs are either housed in UN specialized agencies, such as the WHO, UNICEF, and UNFPA or are independent organizations. For those seven GPPs housed in the World Bank Group, the World Bank provides secretariat services and manages the funds. These few in-house GPPs, along with 118 World Bank Group Support to Health Services | Appendix C the MDTFs, well reflect the World Bank’s comparative strengths, focusing on innovative financing and results (for example, Polio Buy-Down and Pandemic Emergency Financing), aid coordination (IHP+), improving data for evidence-based decision making in health services (Service Delivery Initiative and PHCPI), and enabling the private sector’s participation in the social health insurance market in Africa (HIA). Country-level collaboration with the two largest Financial Intermediary Funds—Gavi, the Vaccine Alliance and the Global Fund—has improved, but there are still untapped opportunities. Since 2015, the country-level partnership with Gavi, the Vaccine Alliance has revamped through that fund’s contribution to a MDTF that will support analytical work and policy dialogue in nine countries that transition Gavi, the Vaccine Alliance, in addition to cofinancing in immunization in Pakistan (see box C.2). However, more clarity on the division of labor and expectations would have helped the World Bank country teams collaborate effectively. The engagement with the Global Fund at the country level has been dynamic but uneven over the years (World Bank 2012). The World Bank helped to institutionalize Country Coordination Mechanisms of the Global Fund in some countries, provided complementary support to the sector. They also, for the first time, are cofinancing an investment project jointly with the World Bank Group (GFF project in the Democratic Republic of Congo), Gavi, the Vaccine Alliance and bilateral donors. The Global Fund’s new 2017–22 strategy’s focus on sustainability and transition will provide more venues for collaboration with the World Bank to support the countries transitioning from the Global Fund support. The effectiveness of the GPPs varies, and not all live up their promise. The available literature observed frequent shortcomings in the effectiveness of their governance, weak strategic focus and monitoring of results, weak links to country programs, tendency to fragment aid further by parallel reporting and inadequate support to building country capacity (Bezanson and Isenman 2012; Buse and Tanaka 2011; World Bank 2011). The evidence also points to important achievements by some of the health partnerships enabling the progress toward MDGs. Gavi, the Vaccine Alliance and the Global Fund, for instance, despite their shortcoming, are considered particularly successful in pursuing their objectives and enticing behavioral change (Sachs and Schmidt-Traub 2017). Stop TB Partnership gets credit for designing the Global Plans to Stop TB, developing innovative approaches to case detection through TB REACH, and increasing supply of TB commodities. (Cambridge Economic Policy Associates 2015) IHP+’s Joint Assessment of National Strategies tool has been used in 13 countries as a common framework for assessing the quality of a national or disease-specific strategies, although the program did not have significant impact on coordination and use of national health strategies (IHP+ 2016). The in-depth reviews of two World Bank Group- housed partnership programs shed light on the extent to which these programs are effective in achieving their intended results, leveraging resources, and contributing to sharing knowledge and solutions. They also highlight the important role the World Bank Group could assume to ensure better learning from these programs, improve the results, and seek collaboration in the field to ensure that the benefits can sustain. Independent Evaluation Group | World Bank Group 119 the Vaccine Alliance and the World Bank Group: An Evolving Box C.2. Gavi, Partnership The Independent Evaluation Group’s 2014 review of the World Bank’s partnership with the Gavi, the Vaccine Alliance found that the World Bank’s financial engagement on behalf of Gavi has been transformative (International Finance Facility for Immunization and the Advanced Market Commitments). However, stronger World Bank involvement, drawing on its strengths in sustainable funding for immunization, addressing inequities in access to immunization, investments in health systems strengthening, and donor coordination in health, could help achieve greater development results. While engaged in the governing board of Gavi, the World Bank’s collaboration at the country level (via World Bank-executed activities) was limited before 2015 due to differences in expectations and perceptions of roles. Gavi now is in a stage when many countries will graduate from Gavi support and World Bank’s help to improve the sustainability of national immunization programs could be critical. In 2015, the World Bank has joined Gavi’s new Partnership Engagement Framework, which defines more clearly the roles of its implementing partners, including the World Bank, WHO and UNICEF. The Partnership Engagement Framework also sets clear reporting arrangements, where the implementing partners will report to the Join Appraisal meetings for each country. The largest share of Gavi’s trust fund support finances World Bank–supervised technical assistance in support to sustainable financing of immunization in nine Gavi high priority countries. These activities are carried out as part of the World Bank’s broader focus on sustainable heath financing in these countries. Some work is also carried out in HSS and data. The renewed collaboration with Gavi is multifaceted, involving cofinancing, analytical work and policy dialogue in areas complementing each other. The World Bank will also integrate vaccines in medicines regulatory harmonization initiatives (through the World Bank–managed Global Medicines Regulatory Harmonization trust fund) in the West Africa region; will work on strengthening the link between vital statistics and immunization through the Global Financing Facility; and improve the effectiveness of polio eradication, through Gavi’s cofinancing of Polio Buy-Down project in Pakistan. HRITF achievements and challenges. HRITF is an MDTF established in 2007 to support design and implementation of results-based financing approaches to improve service delivery in maternal, neonatal and child health. HRITF has fully achieved two of its objectives: supporting design, implementation, and monitoring and evaluation of results-based financing (RBF) mechanisms; and attracting additional financing to the health sector through leveraging. HRITF is also progressing in its other two objectives: develop and disseminate the evidence base for implementing successful RBF 120 World Bank Group Support to Health Services | Appendix C mechanisms, and build country institutional capacity to scale up and sustain the RBF mechanisms with the national health strategy and system. The HRITF has been effective in supporting the World Bank’s design and implementation of RBF approaches in 32 countries to improve service delivery through the Partnership for Maternal, Newborn & Child Health. The program also leveraged about $2.4 billion in IDA resources through linking its RBF pilots with IDA processes and operations, and ensured that the pilots are aligned to country priorities in health. HRITF’s contribution to learning on how RBF mechanisms work in comparison with other service delivery mechanisms will be very valuable, once the large part of the impact evaluations is complete. One challenge the program faces is how to ensure that the lessons from the impact evaluations inform policy discussions on health service delivery. Another major challenge is the prospect of sustainability and the scaling up of the RBF pilots supported by HRITF and IDA. To ensure learning from design and implementation of RBF projects as well as from growing number of impact evaluations, the program organizes learning events, like the Annual Results and Impact Evaluation Workshop, which provides an opportunity for RBF stakeholders to share results and knowledge, discuss implementation experiences, and learn from peers and technical experts. The program also participates in global events, such as the Global Symposium on Health Systems Research in November 2016, where it shared early evidence from impact evaluations. Another frequent venue of knowledge sharing with audience around the world is HRITF’s RBF Bulletin. To improve financial and institutional sustainability prospects, several HRITF countries included cost-effectiveness analysis as part of their impact evaluation and explore lower-cost options for the implementation of certain components of RBF programs (for example, streamlining verification processes; changing the role of communities in the RBF program from direct involvement in verification to monitoring and supervision; and strengthening the strategic purchasing role of the MOH). A few countries, such as Burundi and Cameroon, have moved to a nationwide expansion of RBF. Other countries, such as the Lao People’s Democratic Republic, Tanzania, and Zambia, adopted RBF principles and tools developed by the pilots to inform new World Bank Group projects.8 In Benin, a virtual joint basket system is now used to manage RBF programs, with a cofinancing arrangement that has made it possible to scale-up RBF in 85 percent of the country’s districts (by mid-2017). The GFF has committed to helping sustain and scale up the results achieved through the pilots if countries choose to expand their RBF approach (for example, Cameroon). However, it is still uncertain how the transition will work in non-GFF countries. In non-GFF countries, the World Bank has a critical role to play though its policy dialogue with national governments and through proactive collaboration with development partners to ensure smoother exit or transition from the pilots. HIA achievements and challenges. HIA aims to enable African countries be ready for private sector participation in the health sector through investments (equity funds and debt facilities) and noninvestment activities (analytical studies and policy development). HIA support to investments have had mixed results and debt facility never materialized. IFC supported two equity funds to enhance Independent Evaluation Group | World Bank Group 121 access to long-term capital for health-related small and medium enterprises. The Equity Vehicle for HIA, was structured with a “bottom-of-the-pyramid” focus, show below-target delivery in “provision of health care” and partial achievement in increasing access to health care on the poor (Brad Herbert Associates 2012). The second fund, Investment Fund for Health in Africa, included an NGO that focused strongly on inclusiveness in health care in Africa. The fund was successful in sustaining their businesses and in reaching smaller -size companies with a social focus. Results prompted the follow an investment fund. IFC tried setting up risk-sharing facilities in Tanzania and Kenya with local financial companies, but projects never materialized, because of unattractive rates, diminished appetite of partner banks and government. Some noninvestment engagements have resulted in the identification of the several areas of reforms. HIA noninvestment activities also comprised market studies that received little traction. Some policy engagements included subsequent areas of support. For example, the launch of the National e-Health Strategy in Kenya, the review and operationalization of the 2003 Private Health Sector Policy in Ghana, as well as follow-up actions on setting up the Committee for Dialogue and Engagement with the Private Sector in Mali. Subsequently, the program’s investment climate work continued in Burkina Faso, Democratic Republic of Congo, Ghana, Kenya, Mali, Nigeria, Senegal, South Sudan, and Uganda. On the other hand, outside of Ghana, private health financing initiatives in Kenya and Nigeria had limited results. Noninvestment activities might have received more traction had they been integrated better with World Bank activities since the start of HIA. The program also lacks a governance mechanism—a forum where the program’s stakeholders could transparently discuss and make joint decisions regarding the strategy and progress. This has led to instances of mismatch between the donor expectations and the actual results in some cases and a withdrawal of support of a long-time donor, raising questions about the sustainability of HIA. Conclusions There is a need to improve the selectivity and alignment of its GPPs and MDTFs in health, to align those better to its strategic and corporate priorities in the context of changing global health landscape. The new programs initiated over the last four years clearly fit the bill. Some of the GPPs and MDTFs initiated earlier also still provide clear additionality to the World Bank’s support in improving health services and complement the World Bank’s own work, while others may have lost their relevance. The World Bank’s collaboration with the two largest FIFs in health—Gavi, the Vaccine Alliance and the Global Fund—remains strong globally but can be further strengthened at the country level. While the collaboration with both programs, particularly with Gavi, has deepened at the country level through cofinancing and analytical work, it is still not systematic and, there are still untapped opportunities. The assessments of two in-house programs highlight that the World Bank Group can play a critical role in helping these programs improve their results, and have better uptake of knowledge and forge partnerships in the field to ensure that the programs’ benefits can sustain. 122 World Bank Group Support to Health Services | Appendix C Strategic collaboration with new partners in the global health landscape will help the World Bank Group to remain focused on its mission, as a multilateral institution. New donors/partners, such as the Gates Foundation and Gavi, the Vaccine Alliance, bring greater opportunities but also challenges. A more strategic approach to collaboration with these partners would help avoid the fragmentation and multiple reporting, and risk of entering restrictive contractual relations which can divert the World Bank from its strategic priorities and create reputational risks. Recent dynamics in the collaboration with Gavi, the Vaccine Alliance demonstrate that such an approach can result in more effective engagement for the client countries’ benefit. References Brad Herbert Associates. 2012. Health in Africa Mid-Term Evaluation: Final Report. Washington, DC: World Bank. Bezanson, Keith, and Paul Isenman. 2012. “Governance of New Global Partnerships: Challenges, Weak- nesses, and Lessons.” CGD Policy Paper 014, Center for Global Development, Washington DC. Buse, K., and S. Tanaka. 2011. “Global Public/Private Health Partnerships: Lessons Learned from Ten Years of Experience and Evaluation.” International Dental Journal 61 (s2): 2–10. Cambridge Economic Policy Associates. 2015. Stop TB Partnership Evaluation: External Report. Geneva: Stop TB Partnership. Clinton, C., and D. Sridhar. 2017. “Who Pays for Cooperation in Global Health? A Comparative Analysis of WHO, the World Bank, the Global Fund to Fight HIV/AIDS, Tuberculosis and Malaria, and Gavi, the Vaccine Alliance.” The Lancet 390 (10091): 324–332. OECD (Organisation for Economic Co-operation and Development). 2015. Multilateral Aid 2015: Better Partnerships for a Post-2015 World. Paris: OECD. Sachs, J. D., and G. Schmidt-Traub. 2017. “Global Fund Lessons for Sustainable Development Goals.” Science 356 (6333): 32–33. Sridhar, D., and N. Woods. 2013. “Trojan Multilateralism: Global Cooperation in Health.” Global Policy 4 (4): 325–335. UN (United Nations). 2015. Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–2030. New York: United Nations World Bank. 2007a. Guidelines for Global and Regional Program Reviews (GRPRs). Independent Evalua- tion Group. Washington, DC: World Bank. ———. 2007b. Healthy Development: The World Bank Strategy for Health, Nutrition, and Population Re- sults. Washington, DC: The World Bank. ———. 2007c. Sourcebook for Evaluating Global and Regional Partnership Programs Indicative Principles and Standards. Independent Evaluation Group. Washington, DC: World Bank. Independent Evaluation Group | World Bank Group 123 ———. 2011. Trust Funds evaluation. IEG. Washington, DC: The World Bank Group ———. 2011. The World Bank’s Involvement in Global and Regional Partnership Programs: An Independent Assessment. Washington, DC: World Bank. ———. 2011. The Global Fund to Fight AIDS Tuberculosis and Malaria, and the World Bank’s Engagement with the Global Fund. Independent Evaluation Group. Washington, DC: World Bank. ———. 2012. Refining the Approach to Below-the-Line Grant Making: A Consolidated Report on the World Bank’s Grant-Making Facilities. Washington, DC: World Bank. ———. 2013. World Bank Group Strategy. Washington, DC: World Bank. ———. 2016. Updates to the Priority Directions for the Health, Nutrition, and Population Global Practice 2016–20. Washington, DC: The World Bank Group 1 Global partnership programs are programmatic partnerships in which (i) the partners dedicate resources toward achieving agreed-on objectives over time; (ii) partners conduct activities that are global, regional, or multicountry in scope; and (iii) partners establish a new organization with shared governance and management unit to deliver these activities. 2 Multi-donor trust funds are (i) programmatic in nature, and (ii) conducting activities that are global, regional, or simply multicountry in scope, but (iii) do not have a governing body, and the program manager reports only to his or her line manager and ultimately to the board of the host organization. 3 Established in 1998, the Development Grant Facility focused only on global and regional partnerships and aimed to provide short-term funding “to encourage innovation, catalyze new partnerships, and broaden the World Bank’s services by convening and building coalitions, and provide financial support to external entities.” 4 These are the global health partnerships, initiatives, and programs listed in the 2007 Health, Nutrition, and Population strategy (World Bank 2007b, 179), plus the Polio Buy-Down Program, in which the World Bank has participated since its inception in 2002. 5 For example, the World Bank’s 1997 HNP Sector Strategy envisioned that the World Bank would cooperate with the product pipeline for health-related goods, including malaria, HIV, and tuberculosis drugs. In line with that, the World Bank formed partnerships with some global research and development programs, such as Medicines for Malaria Venture, International AIDS Vaccine Initiative, and the like. The 2007 HNP Sector Strategy, meanwhile, has shifted the focus on strengthening health systems. 6 In general, the higher the level of representation on the governing body, the more importance is given to the program. In nine of these programs’ boards, the World Bank is represented at the highest level—vice president (Scaling Up Nutrition Movement), senior director (Gavi, International Health Partnership, and the Partnership for Maternal, Newborn & Child Health, Middle East and North Africa chief economist (SDI), director or senior adviser level (TDR; United Nations Programme on HIV/AIDS; The Global Fund to Fight AIDS, Tuberculosis and Malaria, Global Financing Facility). In the rest of the programs, lead or senior health specialists represent the World Bank. 7 Through the World Bank executed Global Program for Avian Influenza Control and Human Pandemic Preparedness and Response, 200613, development partners ($4 billion and 35 donors) provided a coordinated response to the avian and pandemic influenzas. 8 HRITF, Achieving Results for Women’s and Children’s Health: Progress Report 2015. 124 World Bank Group Support to Health Services | Appendix C Appendix E. Case Study Analysis of Selected Interventions Conditional Cash Transfers The use of cash transfers has permeated the development agenda since the mid-1990s, creating a revolutionary shift in how the challenges of poverty reduction and human capital growth are approached. Conditional cash transfers (CCTs) are programs that transfer cash, generally to poor households, on the condition that those households make prespecified investments in human capital. Health and nutrition conditions generally require periodic checkups, growth monitoring, and vaccinations for children less than 5 years of age; prenatal checkups for the expecting mother and attendance at periodic health information talks (Fiszbein and Schady 2009). The first CCT was the Brazilian Bolas Escola program launched in 1995. The first CCT projects that received World Bank financial support were the Human Capital Protection Project (Familias en Acción) in Colombia (P069964) followed by Jamaica’s Social Safety Net Project (P067774) and Turkey’s Social Risk Mitigation Project (P074408) that were approved by the World Bank Board of Directors in the year 2001 (World Bank 2011, 86). World Bank Group strategies evolved to better address the multidimensional nature of health challenges and thus began incorporation of CCTs into the World Bank Group approach. The 2007 HPN strategy asserts that CCT projects offer the opportunity for Health, Nutrition, and Population (HNP); Social Protection and Labor (SPL); Education; and Poverty Reduction and Economic Management (now the Poverty GP) to work jointly for improving health and nutritional outcomes among poor mothers and children (World Bank 2007, 126). The 2012–20 SPL strategy considers CCTs as an effective way to provide basic income support to poor families while strengthening children’s health, education, and nutrition—a cornerstone for breaking the intergenerational cycle of poverty (World Bank 2012, 26). CCTs play a primary role in scaling up evidence-based knowledge surrounding SPL with regards to the generation and sharing of knowledge. The Theory of Change CCT programs are designed to stimulate demand for human capital investment, particularly in areas of health. CCTs that increase demand for health services transfer cash to households conditional on the use of those health services. The cash transfer results in an income effect, which determines an increase in demand for superior goods, and a price effect, which reduces the price of the goods conditioned on (World Bank 2014a, 26). CCT programs also provide information and educational packages that are expected to influence perceptions and behavior, thereby enhancing the use of health services. A critical factor in program design is the designation of the cash recipient, as his or her preference determines how the additional cash is used. CCT design often specifies the mother as recipient due to the higher propensity for utilizing the cash transfers for purposes that benefit human capital accumulation. Based on this design and a previous adaptation of the CCT causal chain (Gaarder, Glassman, and Todd 2010), the causal pathway is represented by enrollment into the program, increased demand for health services and health education, increased use of health services, and improved health (see figure E.1) 140 Figure E.1. Conditional Cash Transfer Theory of Change Several assumptions need to be met to achieve the desired outcomes (Gaarder, Glassman and Todd 2010, 12–13): (i) the beneficiaries of the CCT programs are currently underutilizing the health services that are conditioned; (ii) the existing supply of services is sufficient to accommodate the increasing demand; (iii) the beneficiaries of the CCT programs are aware of the program and correctly informed about eligibility, rules and health benefits; (iv) The cash transfers received are used to finance health services and improved food consumption, rather than for goods that could be detrimental for health (for example, tobacco and alcohol); (v) the amount of the transfer is of a sufficient amount to incentivize compliance with the required conditionalities; and (vi) the design features of the CCT program (that is, enrollment mechanisms, verification of conditionalities and cash transfer management) are credible to bring the desired behavioral changes. The review identified 26 World Bank financed projects comprising CCT interventions in the evaluation portfolio. The 26 projects flagged for analysis are comprised of IBRD/IDA commitments for a total of $4,483 million. Most these projects were allocated to lower-middle-income countries (44 percent). Regionally, a higher number of CCT relevant projects were identified in the Latin America and the Caribbean and Africa Regions (both 35 percent). However, CCTs are increasingly being adopted within the Africa Region. Independent Evaluation Group | World Bank Group 141 Figure E.2. Activities Supported by Conditional Cash Transfer Project Note: ANY HS = any health system intervention; HMIS = health information and management systems / monitoring and evaluation; P&L= procurement and supply chain logistics; STW = stewardship, regulations/policy/strategy reform; HFIN = health financing; ANY SP = any supply-side intervention; MED = drugs, vaccines and consumables/supply chain; EQUIP = health equipment and labs; INFR = health infrastructure or land acquisition; INCTVES = financial incentives to HS providers; SKILLS = training and skills to HS providers; IEC = information campaign to HS users (households, patients, general population). To what extent and in what ways has World Bank Group support to health services through CCT effectively contributed to the achievement of relevant health services-related goals? The evidence on the effectiveness of World Bank–financed projects supporting CCT interventions is thin since only five investment projects have been evaluated. included in the evaluation portfolio were evaluated through self-evaluation and IEG verification. Overall project implementation ratings find 96 percent of projects as satisfactory or better, while only 4 percent were rated as moderately unsatisfactory. Efficacy ratings for project development objectives involving CCTs are improving over time. The ICRRs efficacy ratings for CCT related sub-PDOs of closed projects between FY05 and FY16, report 83 percent as substantial and 17 percent as modest. These findings suggest an overall substantial impact of World Bank Group interventions on achieving the project development objectives of quality, equity, improved health, institutional strengthening, and access. Of the 198 project objective indicators, the majority report an achievement rate above 75 percent, except for 12 indicators related to quality and equity (table E.1). These indicators were either not achieved, or do not report results. Indicators related to access, improved health, and institutional strengthening are consistently achieved or surpassed. 142 World Bank Group Support to Health Services | Appendix E and Indicators Achievements in World Bank Group Table E.1. Objectives Projects with Conditional Cash Transfer Interventions Indicator Efficacy Projects with Indicators with Achievement Classification of Ratings S+ Indicator Indicator Results Rate PDO Indicators (no. [percent]) (no. [percent]) (no.) (no.) (percent) Access 5 (100) 19 (83) 63 16 73 use — 12 (52) 41 9 75 availability — 8 (35) 13 3 100 affordability — 2 (9) 9 4 50 Quality 1 (0) 6 (26) 11 1 0 Improved Health 4 (80) 6 (26) 14 12 73 Equity — 1 (4) 1 0 0 Institutional 1 (67) 5 (22) 11 1 100 Strengthening Note: PDO = project development objective; S+ = satisfactory or better. — : Not available. Findings from the World Bank Group Portfolio The most common factor cited by projects as important for CCT design is a solid monitoring and evaluation framework. Substantial time, effort, and capacity is required to establish a monitoring and evaluation system for social protection programs. However, well-established monitoring mechanisms—including audits, scheduled verification of co-responsibilities, and feedback loops— are central to effective program implementation, supervision, and accountability. Such is the case in Colombia (P101211), where a systems audit was useful for identifying areas that required improvement in the health management information system (HMIS). The results of the audit were important in confirming the overall acceptable functioning of the HMIS, and in identifying several areas where improvement was needed. The results of the audit laid out the work plan for strengthening the HMIS during project implementation. Indicators to measure progress should be carefully selected and clearly defined. Furthermore, the indicators that are measured for project progress need to consider the availability of data and feasibility to obtain unavailable data. A lack of reliable and timely data will complicate the performance of monitoring and evaluation. Recent World Bank Group strategies call for increased impact evaluations to strengthen the degree of evidence- based project design. A majority (65 percent) of the projects financing included in this review, had planned or conducted IEs. Of the remaining projects, 35 percent (7) did not conduct an IE relevant to the CCT intervention. This includes two projects that conducted IEs for non-CCT interventions embedded in the project. Of those that did not conduct IEs, very few stated this explicitly with the exception of Pakistan who reported that no IE would be conducted due to the difficulty in collecting baseline data for temporary displaced persons. A simple and flexible design is mentioned by 75 percent of closed projects as essential for CCT programming. Colombia (P101211) attributed the program’s success in part to a simple and Independent Evaluation Group | World Bank Group 143 straightforward project design, but noted that the breakdown of indicators by population was perhaps overly ambitious. Panama (P09832), the Lao People’s Democratic Republic (Lao PDR, P114863) and Colombia (P089443) found that an intricate design with ambitious goals often resulted in project delays or poor implementation. Because of the targeting nature of CCTs, the program is likely to cover diverse populations in terms of gender, ethnic or religious backgrounds. A dynamic program that is capable of adapting to local norms is therefore key to maximizing program enrollment and compliance. Colombia (P089443) finds that the operations of CCT programs in urban areas present certain challenges that need to be addressed; similarly, indigenous communities may require adjustments in the standard procedures. The health demand side of Panama’s CCT (P098328) offers culturally appropriate medical practices and medical practitioners who have been trained in cultural sensitivity and local languages. Moreover, the program’s gender dimensions will be strengthened by working beyond women’s roles as caretakers and mothers to emphasize their roles as citizens, entrepreneurs, and professionals. Twenty percent of projects reference evidence on the importance of integrating adaptability into interventions. Strong communication, collaboration, and harmonization are well noted as critical components for CCT programming. A clear definition of roles and functions of each partner at every level is essential for effective coordination. Such is the case in Madagascar (P103606), where the project preparation process involved active collaboration between the World Bank, the MOH, and development partners. The operation promoted a harmonized approach in the sector by supporting the achievement of key outcomes and indicators as outlined in the health sector strategy formulated by the Ministry of Health. This ensured that there was government ownership and partner buy-in from the start. Strong borrower commitment, ownership, and leadership are important such as in Colombia (P089443), where a proactive role of the government in donor coordination was found to be effective, or in Lao PDR (P114863), where proactivity reinforced effective project management. Lastly, collaboration is most effective when underlined by mutual trust and respect, as is the case with Lao PDR (P114863), where there was a mutual respect between the client and World Bank Group teams, resulting in shared commitment to resolve emerging issues. Community mobilization is key to project implementation. Forty percent of projects find that the community itself if a valuable resource for creating awareness, increasing use and achieving health outcomes. Lao PDR (P114863) finds that CCT messaging delivered by trained local volunteers can effectively improve use of key health services and improve critical nutrition behaviors, especially among poor households living in remote areas. Local involvement may also help to counteract faulty information or misunderstanding such as in Colombia (P101211), where faulty information, lack of credibility, and the negative opinions tied to long lines and travel distances impeded project compliance. It has also been noted that CCTs may have positive impacts at the local level when deployed in close-knit communities, as was the case with Panama (P098328). Matching growing demand with supply is an important factor for program success. Forty percent of closed CCT projects noted the importance of ensuring adequate support for supply-side strengthening to meet increasing demand. This is found to be beneficial in Panama (P098328), where 144 World Bank Group Support to Health Services | Appendix E performance-based financing (PBF) for health providers resulted in synergies in implementation with the CCT. Madagascar (P103606) provides an additional example, where health system activities were complemented by support to strengthen the delivery of key health interventions and implementation of supply- and demand-side activities aimed at increasing use of health services especially in rural and remote areas. Evidence from the Literature Portfolio analysis was complemented with effectiveness evidence steaming from the literature addressing CCT results. EGMs were constructed to identify systematic evidence surrounding the interventions and their expected objectives (that is, improve access or quality of health services (see appendix A for a description of the EGM methodology). The CCT EGM is based on 17 systematic reviews, most of which pertain on their impacts on access and improved health. Most of the evidence on CCT stemming from systematic reviews are of good quality and pertains to the programs impact on access to health care and improved health outcomes of the population (figure E.3). The top categories under access were antenatal visits, institutional deliveries number deliveries attended by health personnel/skilled birth attendant, and immunization coverage, which found positive evidence of the effect of CCTs. In terms of health outcomes, again a range of outcomes was studied for impact across systematic reviews. Most of the evidence from systematic reviews on CCTs impact on health outcomes was mixed. Gap Map on the Effectiveness of Conditional Cash Figure E.3. Evidence Transfer Interventions Systematic Reviews Improve Improve Improve Enhance Improve Quality Access Quality Health Efficiency Equity Total High quality 1   1     1 Medium quality 7 6 1 1 13 Low quality 3 1 2 1 1 3 Total 11 1 9 2 2 17 Note: The Evidence Gap Map is based on 15 systematic reviews of conditional cash transfers interventions: Bassani et al. 2013; Boccia et al. 2011; Ellis and Chaffin 2015; Gaarder et al. 2010; Glassman et al. 2013; Gopalan et al. 2014; Hunter et al. 2017; Hurst et al. 2015; Khan et al. 2016; Lagarde, Haines, and Palmer 2009; Målqvist et al. 2013; Manley and Slavchevska 2013; Murray et al. 2015; Ranganathan and Lagarde 2012; Yuan et al. 2014. The mapping of systematic reviews was further complemented by 30 single-impact evaluation studies on CCT programs. Consistent with the EGM findings, the primary focus of health CCT IEs were their effects on improved health (70 percent), and access to health services (50 percent). CCTs in Lao PDR, Peru, and the Philippines show strong impacts on the use of health services, reflecting Independent Evaluation Group | World Bank Group 145 an increase in the proportion of women who used antenatal services, gave birth in health facilities, and who brought their children to health clinics (Tanner, Hayashi, and Li 2015; Perova and Vakis 2009). Evidence from Panama is less robust whereby there was an increase in Papanicolau test screenings, however, no observed impact on the number of visits to health care providers or uptake of vaccinations (Arraiz and Rozo 2011). Improved health outcomes including child health (Barber and Gertler 2008) and nutrition (Angelucci and Attanasio 2013) have shown an overall positive correlation with CCTs, given slight variances among programs. Programs in Bangladesh, Peru, and the Philippines show a significant impact of CCTs on the decrease of stunting among children and the improvement of nutrition knowledge (Perova and Vakis 2009; Ferré and Sharif 2014). Nutritional intake and improved food consumption was also noted in Colombia where beneficiary households could purchase high nutrient food items (Attanasio, Battistin, and Mesnard 2012; figure E.4). of Outcome Classifications in Conditional Cash Transfer Figure E.4. Summary IEs and Health Services Evaluation Portfolio Note: Denominators in the chart above are subsample sizes (that is, the number of unique IEs, or projects in a subsample), and not the number of outcome classifications, which can be multiple per project. IE = impact evaluation. As previously mentioned, every CCT project aimed to address the issue of access. Contrary to IEs, improved health was not the second greatest outcome category in projects and was preceded in extent by quality and institutional strengthening outcomes. Quality was not a focus in CCT IEs, whereas 23 percent of World Bank–financed projects included a relevant outcome. None of CCT IEs or projects was concerned with the issue of efficiency. CCT projects appear to integrate learning from evidence produced in IEs. The World Bank tends to invest in areas where there is strong evidence of positive correlation. For example, 52 percent of objectives relate to increasing access to health services, an area that has been heavily evaluated and often linked to positive results (Lagarde et al. 2009). Furthermore, in addition to integrating evidence in project design, the World Bank Group is also contributing to evaluation literature. 146 World Bank Group Support to Health Services | Appendix E A majority (65 percent) of the projects financing included in this review expressed the intent for conducting an impact evaluation. Performance-Based Financing Evolution and World Bank Group Strategies In the last decade, an increasing number of developing countries have introduced results-based approaches to finance health care and improve the performance of their health systems. The shift from traditional input-based focus to a stronger emphasis on results has become increasingly popular within the international donor community to support countries in achieving the Millennium Development Goals (MDGs) (World Bank 2013; Engineer et al. 2016). PBF programs play an important role in advancing progress toward the Sustainable Development Goals (SDGs), specially contributing to achieve universal health coverage (UHC) given its focus on a package of health services, the expansion of health service coverage, and the increasing access to good- quality health services (WHO 2010; Fritsche et al. 2014) Results-based approaches are relatively recent strategies to support health system performance. The 1997 HNP Sector Strategy envisaged sectorwide reforms and efficient use of nongovernmental resources to promote equitable access to preventive and curative services that are affordable, effective, well managed, of good quality, and responsive to clients, without a particular focus on results. World Development Report 2004: Making Services Work for Poor People brings attention to weak health providers incentives that lead to poor quality service as one of the challenges of low demand for health services, and advocates for generating incentives and strengthening accountability to address these barriers to service delivery (World Bank 2004). The current 2007 HNP Sector Strategy puts greater emphasis on achieving results in the field by increasing the links between project financing and HNP results. The strategy also brings attention to the risk of perverse incentives, biases in surveillance and monitoring reporting, and governance fragility calling for the need to engage with client countries and global partners to identify opportunities to experiment and learn from innovation in results-based project financing (World Bank 2007; Fair 2008). The same year, the HRITF was created to support results-based financing approaches in the health sector to improve maternal and child health around the world. It provides grants to IDA countries for the design and implementation, evaluation, and learning of result-based financing experiences thus contributing to local and global evidence-based policy-making (see in-depth analysis of HRITF in appendixes B). The Theory of Change The theoretical underpinning of the PBF theory of change lies in the principal-agent model enriched by a more complex view on human behavior from behavioral economics (Renmans et al. 2016). The basic rationale is that health workers exert more effort when payments are conditioned to the quantity and quality of the health services provided. Paying health facilities and their staff on this basis would motivate health workers to be more productive and provide better quality of care. Increased quality is expected to enhance patient satisfaction and demand for services, which would result in a higher use Independent Evaluation Group | World Bank Group 147 of health services. Furthermore, the increased use of high quality health services has the potential to translate into significant improvements in the health status of the population. The causal pathway is represented by the nodes “health provider/supply-side behavior changes,” “improved access and quality,” “utilization,” and “health status” in figure E.5.1 Figure E.5. Performance-Based Financing Theory of Change A number of conditions need to be met to achieve the desired outcomes: (i) the health providers targeted by the PBF program are underperforming in the delivery of health services that are incentivized; (ii) the rules governing the PBF program health are understood by health workers to produce the desired behavioral changes; (ii) the health workers and health providers that are targeted by the PBF program have sufficient control over the desired changes (for example, change in the use of inputs); (iii) the use of monetary incentives is culturally acceptable; (iv) health facilities have flexibility and autonomy to allocate better their resources to respond to the new incentive structure; (v) the design features of the PBF program (that is, MIS, verification and counter verification, payment mechanisms) do not produce unintended consequences such as behaviors such as gaming, task trade-off, cherry-picking, and direct misreporting; (vi) the PBF program (that is, the extrinsic motivation) does not crowding-out intrinsic motivation (Benabou and Tirole 2003); (vii) health facilities have flexibility and autonomy to allocate better their resources to respond to the new incentive structure; and (viii) the demand for health services is sufficient to ensure that the improved availability and quality of health services translate into higher use. The Portfolio of PBF Projects The World Bank’s support for performance-based financing is worldwide. The World Bank has supported PBF programs in 51 countries mostly through PFI project financing operations, DPO 148 World Bank Group Support to Health Services | Appendix E budget support, P4R operations, and recipient-executed grants (table E.2). Africa is the leading Region applying this approach to health financing involving 25 countries. It follows the Eastern and Central Europe with 10 countries, the Latin America and the Caribbean Region with seven countries, East Asia and Pacific and South Asia Regions with four4 countries, and one country in the Middle East and North Africa Region. Bank–Financed Projects with Performance-Based Finance Table E.2. World Interventions Project Financing Instrument Active Closed Total Adjustment 0 10 10 Investment 39 28 67 IBRD/IDA 36 24 60 Recipient-Executed A 3 4 7 P4R 2 0 2 Total 41 38 79 Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; P4R = Program-for- Results. Improving access to and quality of health services have been top objectives in health projects supporting PBF programs, accounting for 71 and 44 percent of the portfolio, respectively (figure E.6). Consistent with the notion that PBF programs entails health reforms beyond a change in the payment mechanism, a nonnegligible share of the PBF portfolio (25 percent) has focused also on strengthening institutions. Although PBF programs can be used as an instrument to improve equity in access to health services, less than 30 percent of projects had objective statements mentioning a particular focus on groups that are more vulnerable. This seemingly inattention to equity may reflect a lack of adequate measures of the project’s distributional impacts due to limited country survey data. In fact, very few projects with explicit equity objectives included outcome indicators disaggregated by income level in their results framework, and therefore projects are seldom held accountable for equity-related results at completion. Overall, only 16 percent of the projects had a goal of improving the health status of the country’s population, and few projects aimed at improving efficiency of the health system explicitly. Projects including a PBF component have shown an increased focus toward quality of health services, along with a slight upward trend in the share of projects with development objectives of increasing access to health service. Meanwhile, improving the health status has been declining as an objective. This may not be surprising since achieving results in some health outcomes, particularly mortality rates, may take longer times than the average duration of an operation. However, this should not discourage the measurement of other short-term outcomes related to morbidity (for example, prevalence and incidence of illnesses) which are measurable within the project timelines. Independent Evaluation Group | World Bank Group 149 of the Project Development Objectives of Projects Figure E.6. Frequency Supporting Performance-Based Financing The main health focus of the projects supporting PBF programs has been maternal and child health care including nutrition services to help countries in the achievement of their MDGs. Attention to maternal and child health (67 percent of projects) predominates in all country income levels since even in richer countries preventable maternal and child deaths disproportionately affect the poor. As expected, the projects’ health focus is consistent with countries’ epidemiological transitions and income level. A larger share of projects in upper-middle income countries (25 percent) supports investments to reduce the burden of noncommunicable diseases compared with low-income countries (3 percent) (figure E.7). Although projects supporting performance-based payments have had less explicit attention to efficiency goals compared with other dimensions of health system performance, they have still addressed efficiency indirectly. Almost all (94 percent) of projects focus on strengthening the primary care system by supporting cost-effective interventions in health promotion and prevention. PBF can be used as an instrument to improve efficiency in the health system by setting performance payments such that to incentivize the provision of health services at primary health centers (for example, birth deliveries) instead of at secondary level to allow hospital resources to be used for more complicated cases, as it was the experience in Zimbabwe (World Bank 2013). The analysis of components at project-level reveals that World Bank–financed projects had an integrated approach to PBF that is consistent with the view that PBF cannot be conceived in isolation. Projects largely addressed the main bottlenecks in the capacities of the health system. In addition to incentivizing health workers through performance payments, projects supported a combination of supply-side interventions, like training for health staff to increase their competences and skills (67 percent) and investments in health system capacities in terms of infrastructure (28 percent), equipment (37 percent), and essential medicines and medical supplies (35 percent). Each of these supply-side interventions was expected to improve health workers’ performance, and hence quality of services, by addressing motivation (“can-do” gap—difference between what health staff is capable 150 World Bank Group Support to Health Services | Appendix E Figure E.7. Health Focus by Country Income Level of doing and what they do in practice); skills and knowledge (“know-how” gap—staff cannot do better than what they how to do); and system capacities (“know-can” gap—installed capacity limits health staff practice) (Lemière et al. 2013; Bawo 2015). Almost all projects included activities addressing the entire health system. Consistent with the importance of a strong and reliable HMIS to monitor PBF programs, 76 percent of projects included support for HMIS to strengthen real-time monitoring and evaluation of provider performance, and for the MOH stewardship and regulation functions. The long-term nature of the World Bank engagement in most of the countries included in this review reinforces the alignment of the World Bank support with the theory of change. From a country perspective, it is expected that past investments in building physical and human capacity, strengthening of the stewardship functions, and improving logistics and distribution systems would result in greater synergies and effectiveness. In fact, 49 out of 51 of countries that applied performance-based payments supported by the World Bank, had received support for HMIS. Thus, caution should be taken in interpreting the absence of a bundled support at project level, for example, incentives to providers and support for HMIS, as such investments could have been supported by the World Bank through previous projects. Projects supporting supply-side PBF programs have also blended demand-side approaches to tackle access barriers and encourage use of health services (52 percent). Most of the demand-side support (83 percent) was in the form of information and education campaigns to raise households’ awareness on the importance of preventive care, induce behavior changes toward healthy habits, and bring services closer to needy populations. Half of the time, demand-side interventions included subsidies to households either in the form of cash transfers to incentivize the use of selected health services or, less often, by financing health insurance premiums to encourage enrollment by the poor. In terms of the evolution of project design, a comparison of closed and active investment projects shows that the World Bank has put more emphasis on investments in HMIS, procurement and Independent Evaluation Group | World Bank Group 151 Figure E.8. Activities Supported by PBF Projects Note: ANY HS=any health system intervention; HMIS= health information and management systems/M&E; P&L= procurement and supply chain logistics; STW= stewardship, regulations/policy/strategy reform; MED= drugs, vaccines and consumables/supply chain; EQUIP= health equipment and labs health; INFRA= infrastructure or land acquisition; SKILLS= training and skills to HS providers; ANY DEM=any demand-side intervention; CCT= financial incentives to users with health conditionalities); IEC= information campaign to HS users (households, patients, general population); IS= insurance programs. distribution logistics, medical supplies, and training for health workers. On the demand-side, it appears to be a slight shift in investments from traditional IEC activities toward other more sophisticated approaches such as CCT. To What Extent and in What Ways Has World Bank Group Support to Health Services through PBF Effectively Contributed to the Achievement of Relevant Health Services-Relat- ed Goals? The development effectiveness of World Bank’s projects supporting performance-based payments has been essentially positive. More than 70 percent of closed investment projects have outcome ratings in the satisfactory range (MS+). The highest percentage of satisfactory outcomes have been in Africa (92 percent), Latin America and the Caribbean (75 percent), and South Asia (50 percent), while in Europe and Central Asia and East Asia and Pacific, only one of three projects has been successful. Satisfactory outcomes are more likely in low-income countries (80 percent) than in lower-middle and upper-middle income countries (50 percent and 60 percent, respectively). In achieving project objectives, however, the World Bank has been successful to a lesser extent. Overall, 61percent of the project development objectives have been rated high or substantial, suggesting that a significant number of project outcome ratings are being pulled into the satisfactory range due to their high relevance of project objectives what indicates that the World Bank is doing the right thing. By Region, the outcome-objective gap is relatively high in the Latin America and the Caribbean Region (25 percent), composed of upper-middle income countries; and in the Africa 152 World Bank Group Support to Health Services | Appendix E Region (13 percent) were closed projects pertain to low-income countries, thus indicating that the relevance and efficient implementation are particularly strong and positively impactful in these two Regions compared with South Asia, East Asia and Pacific, and Europe and Central Asia. In terms of specific objectives, the World Bank has made substantial contributions toward improving access to health care services using performance-based payments, and M&E frameworks of evaluated projects accurately measured such results. The share of projects that successfully achieved their access objectives (71 percent) has been higher than that of comparable projects that did not include a PBF component (61 percent). A total of 69 projects (87 percent) supporting PBF interventions include at least one PDO-level indicator measuring progresses toward improvement in access to health services. Key performance indicators related to access were often associated to effective use of services, most of which focus on maternal health services (such as pre- and postnatal care, and birth delivery) and vaccination services. Other dimensions of access like availability of services (for example, specific services and treatments offered, and adequate stock of drugs and medical supplies in Philippines, Bosnia and Herzegovina, Timor-Leste, and South Sudan), and affordability (for example, decreases in household out-of-pocket expenditures for health in Azerbaijan) were less frequently used in projects’ results frameworks. Overall indicator’s targets were accomplished 69 percent of the time. The evidence is relatively scarce with respect to the World Bank’s contribution to improving the quality of health services when supporting PBF programs. Only four out of seven projects (57 percent) had substantial efficacy ratings in quality. Modest quality ratings are a result of insufficient evidence on outcome data (for example, Niger and Azerbaijan) and not achievement of quality indicator (for example, Timor-Leste Health Sector Strategic Plan Support project). On the other hand, there are more investment projects including quality measures than having a developing objective of improving quality. This imbalance between the project objectives and the selected outcome indicators, however, has improved over time. In general, health projects supporting PBF programs included adequate measures of quality of services in their results framework. About half of the projects included an indicator of quality process measuring compliance with medical protocols and indicating what health providers do to support patients’ health (for example, proportion of children treated with oral rehydration therapy in case of diarrhea, provision of vitamin A supplementation for children or folic acid for pregnant women, screening for common diseases and risk factors like tuberculosis, malaria, HIV, cervical cancer, diabetes, and hypertension). Process indicators of quality had a relatively high achievement rate (76 percent and higher than access indicators). This may not be surprising since it usually requires more effort for the health provider to increase service use, which depends on patient choices, than improving the quality of the care provided, which is largely under the control of health staff (Gertler and Vermeersch 2013). The use of structural quality (seven projects) and patients’ satisfaction (five projects) as key performance indicators was less frequent. Structural type of quality indicators measured licensing and accreditation standards required for health facilities as Independent Evaluation Group | World Bank Group 153 well as health providers’ certifications, appropriateness of equipment and supplies, which also overlaps with availability. There is no sufficient evidence on the contribution of the World Bank to improved health status and efficiency because the number of closed projects addressing those objectives is very limited. Few projects have included indicators to measure outcome aspects of quality of care, such as tuberculosis treatment success rate, prevalence of high blood pressure under control. However, socioeconomic characteristics and health care supply factors explain part of the variation observed. Therefore, these factors should be taken into account (see Giuffrida, Gravelle, and Roland 1999). As previously mentioned, very few projects with explicit equity objectives included outcome indicators disaggregated by income level in their results framework. Service use targets by the poor in closed projects only appeared in Philippines, Nepal, and Vietnam. The PBF program in Benin specifically rewards for equity in use by doubling the amount of performance payments when services are provided to the poor. The program used an already established strategy (that is, a poverty certificate issued for recipients of a health equity fund) to identify poor patients. Identification of the poor can be difficult and costly, and therefore differentiating rewards by socioeconomic status of users is not widespread in PBF programs. However, even if equity is rarely monitored with explicit indicators, it is worth noting that the nature of the interventions supported (that is, use of prenatal care, skilled birth attendance, bed net use while pregnant, and so on) tend to benefit the poor (Fritsche et al. 2014). Gender aspects were monitored through sex-disaggregated indicators in 13 projects. Only one project in Armenia disaggregated the percentages of patients screened for hypertension and diabetes by sex. Many projects (77 percent) monitor the percentage of female patients. However, monitor the number of female patients is of limited use if the entire process and how it related to gender health needs is not taken into consideration (van Wijk, Van Vliet and Kolk 1996). It is noteworthy that the M&E indicators described so far are suitable for most health projects and are not specifically linked to the PBF program. Reporting on the specific indicators that triggered performance payments is not systematic in core operational documents. However, when explicitly reported, quantity-of-services targets are largely consistent with the project-level indicators of the results framework (for example, Afghanistan, Argentina, Panama, Democratic Republic of Congo, Dominican Republic, and Zambia). A remarkable example is the Afghanistan’s Strengthening Health Activities project which offered a sophisticated and well-defined PBF M&E framework that combines scorecards indicators according to different dimensions: (i) access, (ii) quality, (iii) client and community, (iv) human resources, (v) physical capacity, and (vi) management system. In more recent projects, progress has been made in linking the PBF program’s performance to the achievement of the development outcomes of the project, thus improving World Bank’s accountability. About 20 projects included measures that summarized performance of PBF programs, typically averages or increases in health facility quality scores indexes, either as outcome 154 World Bank Group Support to Health Services | Appendix E or intermediate outcome indicators. These composite indicators of the quality of care provided at health facility level are usually project specific. and Indicators Achievements in World Bank Group Projects Table E.3. Objectives with Performance-Based Financing Interventions Projects Indicator Efficacy with Indica- Indicators Achievement Classification of PDO Ratings S+ tors Indicators with Results Rate Indicators (n [%]) (no. [%]) (no.) (no.) (%) Access 17 (71) 69 (87) 262 95 69 Utilization 66 (84) 239 84 68 Availability 11 (14) 19 10 90 Affordability 3 (4) 4 1 0 Quality 7 (57) 51 (65) 98 28 68 Content visit 28 (35) 51 17 76 Quality - structural 7 (9) 9 2 50 Patient satisfaction 5 (6) 7 3 33 Quality - PBF 15 (19) 15 1 100 Efficiency 2 (50) 7 (9) 10 — — Improved health 5 (80) 14 (18) 19 11 82 morbidity 10 (13) 11 4 100 mortality 5 (6) 8 7 71 Institutional strength- 10 (60) 19 (24) 43 19 74 ening Equity (income) — 7 (9) 11 5 60 Gender (sex-disag- 13 (16) 14 3 33 gregated) Gender (female 61 (77) 134 59 71 specific) Performance of PBF component Behavior change 8 (10) 9 2 100 Number of benefi- 13 (16) 16 3 33 ciaries Note: PBF = performance-based financing; S+ = satisfactory or better. A basic presumption of PBF programs is that health workers’ behavior change is a key issue to improve health results, yet there is insufficient attention to health workers’ motivation, absenteeism, and acquired competences. While projects’ results framework included output-level indicators to measure progress in capacity improvements (such as training, quality assurance frameworks, rehabilitated health facilities), only three projects in Benin, Liberia, and Pakistan include indicators related to supply-side behavior changes. Remarkably, the Benin project included three different measures for motivation of health workers, absenteeism rates, and competency score of health staff on maternal and neonatal health. Behavioral indicators usually refer to demand-side changes: while 8 projects measured behavior, changes related to exclusive breastfeeding practices and tobacco Independent Evaluation Group | World Bank Group 155 consumption, many more (35 projects) included use indicators with a behavior change component (for example, mostly on the use of condoms and other contraceptive methods, and the use of mosquito bed nets). Findings from the World Bank Group Portfolio The most important element of a successful PBF program, identified in 40 percent of projects, is a flexibility design that allows for midcourse adjustments and leaves room to health providers to address implementation constraints the innovation and creativity solutions (see for example Afghanistan (P112446), Panama (P106445), Sudan (P098483), and Nepal (P117417). Another critical element for success is to integrate the PBF mechanisms into existing institutional setups and structures. Good examples were identified in Argentina (P095515), the Dominican Republic (P106619), and Madagascar (P103606). Particularly important is harmonizing PBF verification systems with existing national HMIS to ensure accurate, timely and quality data reports like in the Vietnam experience (P095275). Solid M&E framework and performance verification by a third party was explicitly mentioned as essential PBF implementation element by 30 percent of projects, see Afghanistan (P112446 and P110658), Argentina (P095515), Burkina Faso (P093987), Zambia (P145764), and the Dominican Republic (P106619). Previous experience and having sufficient capacity to manage the PBF program (for example, fiduciary and contract management capacity at both national and subnational levels) were identified as important elements in about 30 percent of the projects. Feasibility study and prepilots were used in various settings (for example, Congo (P106851) and Zambia (P145764)) to identify whether PBF programs could be introduced. In Afghanistan, the experience in managing performance contracts accumulated through various projects (P110658, P078324, and P112446) with the support of international experts contributed to the success of the program. In the Dominican Republic (P106619), the experience accumulated through the CCT programs helped the managing the PBF program. In Zambia (P145764), a PBF prepilot and existing strong expertise has informed the design of a larger RBF pilot. The World Bank team of experts who led the RBF pilot design in Zambia had good knowledge of emerging experience from Rwanda and other countries adopting RBF approaches. About 20 percent of projects stated governments’ institutional capacity and mechanisms as critical in the PBF implementation. Local capacity to manage NGO contracts effectively and efficiently through P4P arrangements made a difference in the cases of Afghanistan (P110658, P078324, and P112446). Clients’ adequate fiduciary, contract, and financial management capacity at national and subnational levels are reported particularly important in the implementation of PBF projects. 156 World Bank Group Support to Health Services | Appendix E Evidence from the Literature Similar to CCT interventions, an EGM based on systematic reviews was constructed to highlight the concentration of evidence among relevant health system objectives. Evidence from systematic reviews of PBF interventions were of medium of low quality; evidence should be used with caution. Within the eight systematic reviews focusing on performance-based financing, program impacts were most reported on access to health services and the quality of health services (see figure E.9). Under access, the primary indicator was the number of institutional births. While the results were generally mixed, one high quality systematic review conducted by Witter et al. (2012) found positive evidence however, the quality of primary studies was rated low. The systematic reviews of PBF on quality of health services are all low quality except for one of high quality; and the indicators were generally related to structural aspects of quality. Figure E.9. Evidence Gap Map on the Effectiveness of PBF Interventions Systematic Reviews Improve Improve Improve Enhance Improve Quality Access Quality Health Efficiency Equity Total High quality   1     1 Medium quality 3 Low quality 4 3 1 2 4 Total 4 4 1 2 8 Note: The Evidence Gap Map is based on 8 systematic reviews of performance-based financing interventions: Blacklock, Claire, et al. 2016; Das, Ashis, et al. 2016; Eichler, Rena, et al. 2013; Eldridge, Cynthia and Natasha Palmer 2009; Renmans D, et al. 2016; Suthar, Amitabh B., et al. 2017; Turcotte-Tremblay et al 2016; Witter, S., A. et al. 2012. The body of knowledge on the effectiveness of performance-based payments is growing as shown by an increasing number of impact evaluations. The World Bank is regularly generating new evidence on different program experiences through rigorous impact evaluations using experimental and quasi-experimental designs. From 79 projects supporting PBF interventions, 25 planned for an impact evaluation most of which are still ongoing. As of 2016, the HRITF financed 34 RBF impact evaluations, 28 of them accompany country pilot grant and 6 are stand-alone impact evaluations. Eleven impact evaluations distilling evidence on PBF programs has been finalized and are included in this review. The comparison of impact evaluation topics with project-level objectives indicate good alignment between where the available evidence is and what are the main goals of the health projects supporting PBF programs (see figure E.10). Independent Evaluation Group | World Bank Group 157 of Outcome Classifications in Performance-Based Figure E.10. Summary Financing Impact Evaluations and Health Services Evaluation Portfolio Note: Denominators in the chart above are subsample sizes (that is, the number of unique IEs, or projects in a subsample), and not the number of outcome classifications, which can be multiple per project. IE = impact evaluation. The majority of impact evaluations have focused on the potential of PBF programs to increase access to health care services finding in general a positive impact, yet there are variations across countries and across maternal and child care services. For example, PBF programs increased the use of family planning methods in Burundi and Cameroon, but not in Afghanistan, Zimbabwe, and the Democratic Republic of Congo. Prenatal care use also increased in Argentina and Burundi as a result of PBF, but there was not significant effect in Rwanda, Afghanistan, Cambodia, the Democratic Republic of Congo, and Zimbabwe; while postnatal care increased in Zimbabwe. Evidence about PBF impacts on skilled birth attendant and institutional delivery also varies across programs, it was positive in Rwanda, Burundi, Cambodia, and Zimbabwe, while no differential effect was found in Cameroon and Afghanistan. Child immunization rates increased in Cameroon, but not in Rwanda, Afghanistan, Cambodia, the Democratic Republic of Congo, Zimbabwe, and Burundi. In Rwanda, the PBF program had a positive effect on the number of preventive care visits by young children, and the likelihood of being tested for HIV (Basinga et al. 2011; Gertler et al. 2014; Bonfrer et al. 2013; World Bank 2014d; de Walque et al. 2015; Engineer et al. 2016; Van de Poel et al. 2016; Friedman et al. 2016; World Bank 2017). As it was discussed at project-level, impact evaluations use a variety of quality measures when estimating the PBF program impacts showing mixed results. Systematic reviews conclude that the impact of PBF programs on quality of care is still unclear (Witter et al. 2012), but several reviewed programs found improvements in quality measures (Chalkley et al. 2016). For instance, PBF programs were found to improve the quality of health care processes by increasing the likelihood of receiving tetanus vaccines in prenatal visit in Rwanda, Argentina and Zimbabwe, and improving the overall health facility quality score in Rwanda, and Burundi (Basinga et al. 2011; Bonfrer et al. 2013; Gertler et al. 2013; Friedman et al. 2016). However, as previously highlighted, quality checklists are 158 World Bank Group Support to Health Services | Appendix E mainly composed by structural indicators. In Cameroon, as well, most of the positive impacts were observed on structural quality (for example, equipment availability and health staff presence), but no impact was found on the completeness of service provision during antenatal care and child health consultations (World Bank 2017). Compromising quantitative and qualitative approaches, the impact evaluation of the Zimbabwe’s PBF program found mixed results regarding quality gains. Despite PBF facilities showed improvements in availability of medicines and equipment and increases in all standardized indexes, differences with the control group were not always statistically significant. Positive impacts were found on the likelihood pregnant women had a urine test and tetanus injection, and improvements in quality measures occurred by the second year of the RBF pilot, thus suggesting the importance of a learning phase to achieve results. Direct observations of prenatal care at health facilities, however, reported no differences in the quality of care with respect to control group facilities (Friedman et al. 2016). Modest impacts on quality were found in Afghanistan and the Democratic Republic of Congo. In the first case, an impact evaluation found positive results in three measures of technical quality of care (that is, PBF providers spent more time with patients, conducted more complete histories and physical examinations and provided more counseling to patients), although it did not find impact on any of the other 17 Balanced Scorecard indicators at health facilities that were less directly under health worker control (Engineer et al. 2016). In the second case, several measures of quality were poorer in PBF facilities than in the control group (that is, quality and quantity of equipment, quality index based on interviewers’ observation; availability of vaccines) (World Bank 2014d). Among the studies that estimated the impact of PBF on perceived quality, patient satisfaction increased in Cameroon and Zimbabwe according to exit interviews, but no impact was found in Afghanistan, the Democratic Republic of Congo, and Burundi. The evidence linking performance payments to better health population outcomes is thin and mixed. For example, Gertler and others (2013) reported positive impacts of the PBF program on birth outcomes (that is, low birth weight) in Argentina, which is a factor that is closely linked to neonatal morality. However, a second study found no additional impact of temporary financial incentives on birth weight and premature birth, despite the large effect of providers’ incentives on early initiation of prenatal care (Celhay et al. 2015). PBF led to a reduction in children’s underweight and stunted measures in Rwanda and Zimbabwe, but those measures deteriorated in PBF districts in the Democratic Republic of Congo. Incentives had no significant effect on other child health and behavior changes in Zimbabwe (for example, episodes of diarrhea, sought advice for children health concerns, complying with antimalarial treatment) (Gertler and Vermeersch 2012; Friedman et al. 2016; World Bank 2014d). Evidence from PBF experience in Cambodia find no impact on neonatal mortality, which is quite sensitive to high quality maternal and child health services typically incentivized by PBF programs. Lack of strong evidence on health gains cast doubts about the potential for incentivized institutional delivery in a context of poor quality of services evidenced by lack of equipment and trained personnel (Van de Poel et al 2016). Impact evaluations do not tend to include costs estimates turning cost-effectiveness of PBF programs a big gap in the literature. While programs increase financing, and entail considerable administrative and monitoring costs, their value for money with respect to more traditional input- Independent Evaluation Group | World Bank Group 159 based financing is yet to be ascertained, and therefore is a priority for future prospective research. A recent systematic review of the value for money of PBF interventions found that the studies were not full economic evaluations of PBF programs, and that alternative interventions to strengthen the capacities of the healthcare system were not considered (Turcotte-Tremblay et al. 2016). Recent studies, including knowledge products generated by the World Bank, have progressed in unpacking the “black box” of complex PBF programs into the specific program elements and intermediate effects. Since PBF programs tend to trigger changes in different dimensions (motivation, enhanced supervision, managerial autonomy, data collection), observed effects are no easily attributed to payments linked to performance alone imposing research challenges. There has been raising concerns, particularly in the absence of cost-effectivity data, that in low-income settings a straight increase in health workers’ salaries would boost their motivation and consequently their performance. Impact evaluations supported by the HRITF have started to disentangle incentives effects from resource and supervision effects. To ensure resource neutrality between PBF and non-PBF facilities the design introduced equivalent input-based budgets in the latter group equivalent to the amount of performance payments to PBF facilities in Rwanda and the Democratic Republic of Congo. The recent randomized experiments in Zambia and Cameroon included a more sophisticated design of 3 and 4 different arms, respectively, to differentiate PBF, from increasing financing, and enhanced monitoring in the Cameroon case. In practice, however, additional financing received by control facilities in Zambia turned out to be lower due to administrative bottlenecks. In Cameroon, differences between PBF facilities and those receiving additional financing coupled with enhance supervision turned out to be subtle (Shen et al. 2017; World Bank 2017). These de jure and de facto differences may explain how implementation challenges affect the effectiveness of the PBF captured by impact evaluations (see box E.1). The still nascent literature on the effects of PBF on health worker’s outcomes in low-income settings suggest mixed results (Renmans et al. 2016). Following the example of Cameroon, the availably of more qualified human resources improved, both in PBF facilities and in those receiving additional financing not linked to performance, with respect to the other two groups, suggesting that monitoring alone is not enough to reduce absenteeism. The Zambia evaluation did not find evidence that extrinsic motivation would crowd-out intrinsic interests. Interestingly, satisfaction with compensation was found to be higher in the PBF group as expected, although job satisfaction was higher in the enhanced financing group. High workload, exacerbated by chronic staff shortages, ultimately compromised health care workers’ motivation to provide high-quality services (Shen et al. 2017). In Afghanistan, results suggest that mechanisms behind performance payments to change provider motivation did not occur since indexes of motivation and job satisfaction were same in both groups (Engineer et al. 2016). Positive effects in absenteeism were found in the Democratic Republic of Congo, however, staff attendance reduced several months after the pilot ended, which suggests that incentives need to be thought of as a permanent policy. Health workers in PBF facilities were less satisfied with the job because of increased workloads and concerns about the volatility of 160 World Bank Group Support to Health Services | Appendix E health facility revenues (World Bank 2014d). The PBF program in Zimbabwe resulted in higher health workers’ satisfaction with respect to their increased compensation and qualitative findings show that Explaining Findings from the Literature on Performance- Box E.1. Factors Based Financing Programs Performance-Based Financing Design and Implementation ƒƒ Insufficient incentive payments: the relatively low-power of performance payments in relation to guaranteed salary negatively influence health workers’ motivation and limits performance-based financing (PBF) gains (Engineer et al. 2016; Fritsche et al 2014; Ssengooba et al. 2012; World Bank 2014; Rudasingwa and Uwizeye 2016). ƒƒ Timeliness of payments and revenue shortfalls: interruptions in the payment schedule distort heath facility revenue differences between PBF and non-PBF facilities (World Bank 2014). Delays in governments input-based financing caused revenue shortfalls leading facilities to rely on RBF funds for expenditures (Friedman et al. 2016). ƒƒ Health providers’ control: the potential effects of PBF programs are larger for services over which health staff has greater control, and hence the marginal cost of effort is lower. PBF should pay more for those services which greater marginal cost of effort (Gertler and Vermeersch, 2013; Van de Poel et al. 2016). ƒƒ Already high baselines: the room for improvement influences program impact with higher potential improvement enabling stronger impacts, and raise concerns about the efficiency of subsidizing these particular targets (Gertler and Vermeersch, 2013; Fried- man et al. 2016; World Bank 2017; Chalkley et at 2016; Van de Poel et al. 2016). ƒƒ De jure vis-à-vis de facto I – verification weaknesses: less than planned community verifications throughout the pilot, and limited financial sanctions associated with fraud- ulent over-reporting reduce incentives to perform (World Bank 2014b). ƒƒ De jure vis-à-vis de facto II –contamination of treatment groups: differences between intervention groups can be subtle when neighboring facilities from different groups learn from each other and intervention design is not fully grasped by staff and man- agement (World Bank 2017; Shen et al. 2017; World Bank 2014). Households’ bypass- ing behaviors (that is, not seeking care at the closest facility) also leads to estimates which are below the true causal effect of the intervention (World Bank 2017). Health Workers’ Behavior ƒƒ Understanding the PBF program: difficulties in communicating to health workers and limited understanding of how payments were calculated may have undermined the potential effects of the P4P intervention (World Bank 2014; Engineer et al. 2016). Independent Evaluation Group | World Bank Group 161 Box E.1, continued. ƒƒ Decreasing motivation and job satisfaction: higher workload as an imperative to earn points added pressure on health workers (Shen et al. 2017); increased patient load, contributing to a higher workload and consequent burnout (Friedman et al. 2016); sen- timent that the district supervision visit was too frequent and sometimes too stringent (Shen et al. 2017). ƒƒ Perverse incentives: Gaming and task trade-offs has been observed in several PBF programs (for example, Kalk et al. 2010; Binyaruka et al. 2015; Janssen et al. 2015). Cherry-picking strategies (that is, the strategic choice of patients) have been observed less frequently (see Lannes et al. 2015; Skiles et al. 2013). Active manipulation of re- ports has also been observed (Kalk et al. 2010; Khim and Annear 2013). Contextual Factors ƒƒ Inattention to demand side: supply-side incentives for providers were not sufficient to increase use when geographic barriers, preference for alternative treatment options, or existent of user fees act as a barrier on the demand-side (World Bank 2017; Engineer et al. 2016; World Bank 2014). Higher effects were found when PBF was accompanied by the demand-side interventions like distribution of vouchers to cover user fees (Van de Poel et al. 2016). ƒƒ Interactions with other programs: concomitant interventions supported by other programs (such as Expanded Program for Immunization or HIV/AIDS) may crowd-out (Friedman et al. 2016) or boost the effects of the PBF program (de Walque et al. 2015). ƒƒ Health facility autonomy: changes in organizational practices resulting from changes in the incentive payments, such as expanding the scope of work of community health workers, may encourage households’ seeking care behavior (Celhay et al. 2015). ƒƒ Inefficient allocation of human resources: lack of consistency between resource de- ployment to some facilities with the catchment population, increased workers pressure and workload and reduce job satisfaction (Friedman et al. 2016). they were strongly motivated by incentives. However, they also expressed their dissatisfaction with untimely disbursements, the level of incentives relative to tasks, the higher patients’ flows leading to higher workload, the lack of clear roles and lines of reporting, and limited capacity of supervisors. These findings on health worker motivation are puzzling given the positive effects of the program on some access and quality indicators (Friedman et al. 2016). Pandemic Preparedness and Control Evolution and World Bank Group Strategies The World Bank supported member countries in preparing for and controlling several pandemic threats over FY05–16, such as the avian influenza, also known as “bird flu,” the 2009 H1N1 flu or 162 World Bank Group Support to Health Services | Appendix E “swine flu,” the Middle East respiratory syndrome, the 2014 West Africa Ebola virus disease outbreak, and the Zika virus outbreak in Latin America and the Caribbean. The World Bank approved in 2016 the Global Program on Avian Influenza Control and Human Pandemic Preparedness and Response (GPAI) that channeled Banks own and donors’ resources into avian influenza control and human pandemic preparedness projects. The 2007 HNP strategy recognizes the need to strengthen health systems to fight human immunodeficiency virus / acquired immune deficiency syndrome and other pandemics. The following year the World Bank together with WHO, OIE, FAO and UNSIC prepared the global strategy “One Health” aimed to strengthen country systems in the veterinary and human public health areas, and the bridges between them. In May 2016, the World Bank launched the Pandemic Emergency Financing Facility, an innovative insurance-based mechanism, which will provide surge funding, in the form of grants, to low-income countries to respond to rare, high-severity disease outbreaks. The Theory of Change Effective pandemic preparedness and control (pandemics) requires collective response and a global health governance to fulfil four critical functions: surveillance, protection of the population from the circulating virus, effective response to outbreaks, and communication (Lee and Fidler 2007). The 2005 International Health Regulations provide the legal and political underpinning for the required global response.2 Research and development and access to diagnostics, vaccines, and treatments are crucial to health security, and cannot be left to market forces (Heymann 2015). UHC strengthen country-level pandemics through two transmission channels. First, by lowering financial barriers stimulating demand for health services and facilitating early case detection of outbreaks. Second, UHC protects people from impoverishing healthcare expenditures, which, in turn, reduce their long-term risk of contracting communicable diseases (Jain and Alam 2017). Zoonosis represents the most significant threat for pandemics. The “One Health approach”3 would reduce health risks at animal-human-environment interfaces. However, broad institutional changes - and ownership of these changes across the various ministries, departments, and interest groups with a stake in disease control - are required (Okello 2014). Specific health services strengthening for pandemics include management, diagnostic capacity of clinical laboratories, training of laboratory technicians, equip and maintain Intensive Care and Isolation Units, supply chain management for vaccines and antiviral drugs, availability of trained human resources, including epidemiologist and a reserve corps of trained personnel and volunteers. Change in the behavior of the population are required to reduce the spread of the diseases among the population. For example, travel advisories and precautions, screening of persons arriving from affected areas, closing schools, restricting public gatherings, quarantine of exposed persons and isolation of infected persons. Cultural adaptation and support from respected community leaders who can lead local engagement efforts, and community workers who speak local languages are key for adopting pandemic containment measures as shown during the Ebola pandemic (see figure E.11). Independent Evaluation Group | World Bank Group 163 A number of assumptions need to be met to achieve the desired outcomes: (i) frontline human resources would continue provide health services notwithstanding the risk of contagious; (ii) honest, accurate and timely information are provided to the public; (iii) the population and the health workforce respond to behavior change interventions (for example, information and incentives) in the expected manner; (iv) the broader economy and the key functions of the state are resilient to the pandemic outbreak. Figure E.11. Pandemic Preparedness and Control Theory of Change Avian Flu An uncoordinated early response delayed limiting the spread of the virus which is still active today. In terms of project preparation and activation, avian influenza projects were prepared quickly, with a median time of 4 months from concept note to approval. The average time elapsed from outbreak to project activation was 275 days. Partnership arrangements were quite similar for all projects with FAO taking the lead in animal health and WHO for human health. Countries obtained technical assistance from other agencies such as the European Commission, USAID, CDC, OIE, WHO, UNICEF and the Australian, Canadian, and UK bilateral agencies to help implement the GPAI projects. The first wave of World Bank-funded avian influenza projects focused on the consolidation of animal health services (for example, Vietnam, Armenia, West Bank and Gaza) and some projects established compensation funds and created insurance programs to manage the economic losses of culling the infected animals. The project design of the second wave of projects followed a template consisting of four components: (i) animal health; (ii) human health; (iii) communications; and (iv) project management. The animal and human health components are of equal importance in controlling the disease outbreak. However, the human health components tend to have a slower start than the animal health components except in cases where the human health component was integrated with an existing health project. The slow implementation of the human health component was affected by a waning in momentum with a decrease in the social mobilization mechanisms that followed the drop 164 World Bank Group Support to Health Services | Appendix E Table E.4. Countries Receiving Support from the World Bank under the GPAI Middle East Latin America East Asia and Europe and and North and the Carib- Africa Pacific Central Asia Africa bean South Asia Cameroon; Con- Cambodia, Chi- Albania, Arme- Djibouti Arab Argentina, Afghanistan, go, Dem. Rep.; na, Indonesia, nia, Azerbaijan, Republic of Bolivia, Brazil, Bangladesh, Congo Rep.; Lao PDR, Mon- Bosnia and Egypt, Islamic Chile, Costa Bhutan, India Liberia; Malawi; golia, Myanmar, Herzegovina, Republic of Iran, Rica, Domini- Nepal, Sri Lanka Mauritania; Vietnam Georgia, Koso- Jordan, Syrian can Republic, Mozambique vo, Kyrgyz Re- Arab Republic, Haiti, Honduras, Niger; Nigeria; public, Moldova, Tunisia, West Mexico, Nicara- Sierra Leone; Romania, Ta- Bank and Gaza gua, Paraguay, Togo; Uganda; jikistan, Turkey, Uruguay Zambia Turkmenistan, Uzbekistan Source: World Bank 2014c, 5. Note: Included are 30 Rapid Assessments costing less than $100,000/assessment and are not investment projects. GPAI = Global Program on Avian Influenza Control and Human Pandemic Preparedness and Response. in the number of confirmed cases in 2007–08, the limited of evidence on the transmission of the virus from person to person, and the reduced perceived importance of avian influenza versus other communicable and noncommunicable diseases. The long-term capacity building required for pandemic preparedness requires long preparation time for adequate analytical work or for coordinating relevant multisector interventions. The quick preparation and standardized approach adopted to address the pandemic emergency, does not fit well with the long-term objective of increase pandemic preparedness. An alternative would have been to launch a quick emergency project followed by a complementary support aiming at longer-term objectives, such as institutional reform and capacity building project with the broader pandemic preparedness focus. On the other hand, pandemic preparedness built by this emergency operation will be rapidly eroded in the absence of a program (externally or internally funded) that goes beyond the life of the project, for securing the sustainability of preparedness to deal with future outbreaks (Tajikistan). The communication components of the avian flu projects were generally well conceived. However, there were some unexpected responses that lead to unproductive panic, insufficient public knowledge, and erosion of faith in public authorities. Practical behavior change activities enforced by demonstration interventions involving civil society groups, should be added to complement traditional multimedia messaging (see project in Moldova where the communication and information campaign was part of an Europe and Central Asia-wide behavioral change strategy). Project management capacity (for example, procurement expertise at the Project Management Unit PMU) was an implementation concerns frequently cited in ISRs and in aides’ memoirs of supervision missions, even if project management was usually rated as Moderately Satisfactory and Satisfactory. Multisector cooperation was difficult as projects usually involved multiple ministries and complex execution modalities. In Vietnam, the collaboration between the Ministry of Health and the Ministry Independent Evaluation Group | World Bank Group 165 Table E.5. Factors Related to the Performance and Outcomes of Avian Flu Projects Moderately Satisfactory, Satisfactory, or Highly Moderately Unsatisfactory or Unsatisfactory Out- Satisfactory Outcomes comes Proactive government support as evidenced by short Failure to adjust compensation rates may result in weak preparation time and government processing toward incentives to report disease outbreaks early. loan/credit effectiveness. Outdated institutional and human resource process- Operational decisions decentralized to lower levels lead es to conduct efficacious surveillance and detection to more project ownership at the lower government actions. levels but potentially offset by higher transaction costs. Tensions develop between the need for ensuring a rap- The engagement of international organizations to id implementation start up in the face of an imminent support implementation of an emergency project has threat and the need to ensure that the implementation benefits in terms of technical rigor, but also poten- experience is embedded in ongoing operations of line tial consequences in terms of reduced government ministries. ownership. There is a high likelihood that government budget Multisector project implementation increases the con- allocations will not be able to sustain operations put in trol of the infection. place when development programs are largely exter- Behavior change strategies using TV communications, nally funded. public awareness clips, multimedia campaigns and Delays in signing agreements with international organi- regional video conferencing. zations result in slower startups. Cross-ministry cooperation between Ministry of Health Loss of momentum and declining government commit- and Ministry of Agriculture and Rural Development. ment and donor pledging when the pandemic shows Full time PMU staff integrated in project management signs of slowing down and other health emergencies or and low PMU staff turnover. political events intervene. A fast-track procurement process for emergency Collaboration with NGOs can be very productive, operations to speed up implementation and the use of especially at the grassroots level, but can encounter bilateral agencies with faster procurement procedures. compatibility difficulties with project line ministries. An ability to rapidly implement pharmaceutical interven- Slow uptake of human health component except in tions (vaccination and use of antiviral medication) and cases where the human health component became an nonpharmaceutical interventions (physical distancing integral part of an existing health project. policies such as quarantine and school or workplace Limited outreach to civil society especially in support of closures), behavior change strategies. Monitoring and evaluation with a robust results frame- Insufficient national leadership to ensure cross-ministry work used as a management tool for effective project cooperation, implementation Long delays in appointing and mobilizing PMU staff at the central and, especially at the lower levels, and resignations and re-assignments resulting in frequent PMU staff turnovers and loss of skills. Not having an experienced procurement expert con- tributes to delays in implementation and an inability to meet emergency goals. Delays in issuing permits to construct or upgrade clinical laboratories and shortage of technical capacity of staff to operate and maintain the equipment. Weak epidemiological surveillance and weak biosecu- rity levels on poultry farms provide conditions in which the mutation of the virus continues to prevail. Results frameworks being developed but not used by project management as a tool for effective project implementation partly due to voluminous and poorly focused indicators and many countries not having a culture of measuring progress and documenting results. 166 World Bank Group Support to Health Services | Appendix E of Agriculture and Rural Development was effective. Support from the leaders in the two ministries facilitated information sharing, joint analysis of results and efficient use of resources. Low turnover, and the level of integration of the PMU within the main sectoral ministries contributed positively to speed up implementation and to establish effective links across components and sectoral ministries (see Tajikistan). Specific challenges: (i) effective supply chain management of antiviral drugs with limited lifespan; (ii) timeless of permits for the construction of laboratories; (iii) capacity to staff to operate and maintain laboratory equipment (see Nepal). The capacity to control an avian influenza outbreak is related to the capacity of the health system and to the level of health service coverage (that is, the level of UHC). Pandemic control interventions include pharmaceutical interventions (for example, deliver drugs and vaccines), behavior change interventions (for example, physical distancing policies such as quarantine and school or workplace closures) and laboratory capacity. Therefore, the capacity to provide manage the relevant part of the health system, including the ability to retain the necessary staff determine the capacity of the country to respond to a pandemic. Epidemiological surveillance remains the most important but also the weakest disease control and prevention tool. The introduction of the influenza-like illness parameter into grassroots human health surveillance programs in avian influenza client countries is a valuable addition to effective disease surveillance. Advanced biosecurity laboratories that provide a high degree of biosecurity, including air tight seals, controlled ventilation with directional air flow, and filtered air exhaust are a necessary ingredient of a comprehensive pandemic preparedness and response. However, because of the large costs and technical expertise required to build, maintain and operate them, a region approach that allow concentrating the limited resources available is recommended. Weakness in the project monitoring and evaluation (M&E) framework were often mentioned (12 percent of M&E components rated unsatisfactory). The review of 20 Emergency Project Papers indicated that only 11 had acceptable results frameworks and monitoring plans. Bangladesh had an excellent M&E framework. Too often an M&E system is set up but not fully utilized for effective management of the project (Albania). Where a project is aimed at mitigating a disease threat with an uncertain probability of occurring, the results framework should cover both scenarios – when the disease strikes and when it does not. Intermediate outcomes need to be monitored to assess whether capacity is improving, and whether there are sustainable changes in preparedness, irrespective of whether the threat materializes (Tajikistan). Stronger management oversight is needed to ensure that project M&E is used as a management tool for effective project implementation. The table below summarizes the factors that have been related to projects’ performance and outcomes (World Bank 2014c). Ebola Virus An uncoordinated early response, which exposed the overwhelmed public health capacity of the region and claimed the lives of thousands was followed by one of the most successful global partnerships Independent Evaluation Group | World Bank Group 167 between foreign and local governments and multinational aid organizations to stem an international health crisis (Siedner and Kraemer 2017). The pandemic is now in a postpandemic period. The World Bank was a key actor of the Ebola virus response in the West African region (Siedner et al. 2015). The World Bank supports response and recovery, which includes restoring basic health services, helping countries get all children back in school, farmers back planting in their fields, businesses back up and running, and investors back into the countries. A top priority for World Bank Group support is to build a strong and well-trained health workforce; build resilient health systems that can deliver essential, quality care in even the most remote areas; improve disease surveillance; and quickly detect, treat and contain future outbreaks. As a global coalition led by WHO, prepared the plan to contain the Ebola outbreak, the World Bank Group quickly restructured ongoing health projects and, together with the African Development Bank, committed on September 2014, just after 28 days from the WHO declaration of PHIC, $105 million for the Ebola Emergency Response Project (P152359). A $285 million additional financing was approved two months later in November 2014. The World Bank Group seconded a senior public health specialist to WHO to assist in the coordination of the technical and financial efforts of these two institutions. In total the World Bank Group mobilized $1.62 billion ($1.17 billion from IDA and at least $450 million from IFC) to support Ebola response and recovery efforts in the three West African countries hardest hit by Ebola: $260 million for Guinea; $385 million for Liberia and $318 million for Sierra Leone. The initial $518 million commitment comprised $390 million from the World Bank Group’s IDA Crisis Response Window; $110 million from national IDA; and $18 million reallocated from existing health projects. Findings from the World Bank Group Portfolio Systemic weaknesses of the health systems and services in the three most affected countries – including insufficient funding, an inadequate workforce, poor infrastructure, shortages of medicines and supplies and weak health information and disease surveillance systems – all contributed to the spread of Ebola and undermined efforts to respond (Kieny, Evans, and Schmets et al. 2014). It is worth comparing the different outcomes of the Ebola outbreaks in Nigeria, Senegal, and Mali, where differently from Guinea, Liberia and Sierra Leone, could mount a successful and rapid response to EDV before it spiraled out of control. The key elements for success in controlling the epidemic are: (i) fast and thorough tracing of all potential contacts; (ii) ongoing monitoring of these contacts; and (iii) rapid isolation of potentially infectious contacts (Fasina et al. 2014). All three countries had their own high-quality laboratories, facilitating the rapid detection or discarding of cases.4 Contact tracing was rigorous and most identified contacts were monitored in isolation. Local staff and existing infrastructures were used in innovative ways. For example, Mali used medical students with training in epidemiology to increase staff numbers for contact tracing. All three countries established emergency operations centers and 168 World Bank Group Support to Health Services | Appendix E recognized the critical importance of public information campaigns that encouraged community cooperation (World Health Organization 2015). In Nigeria, the Polio Emergency Operations Centre and its vast experience and resources served as a springboard for the Nigeria’s Ebola response. The center operated an Incident Command System, which involved a plethora of actors, including government and donors, but bypassed bureaucracies. The Polio Emergency Operations Centre was therefore quick to make decisions and respond, as well as use real-time data to trace cases and control the epidemic. The speed of the response to an outbreak is key to control the epidemic and to avoid it spirals out of control. If the global partnership in the Ebola pandemic had responded earlier after the first signs of an uncharacteristic outbreak, it is likely that the number of lives lost, the impact on health infrastructure, and the magnitude of the eventual response could have been drastically diminished (Siedner et al. 2015). The Pandemic Emergency Financing Facility launched by the World Bank Group in 2016 would provide financial resources supporting quick implementation of a global rapid response strategy that includes the recruitment and training of health workers, strengthens early warning and detection systems, monitors and evaluates the evolution of the pandemic to provide rapid feedback to public health agencies on the effectiveness of the public health and clinical response. Containment of the avian flu pandemic and ending the outbreak of the Ebola virus would not have occurred without a global coalition. Technical assistance from WHO and US CDC, financing from several donors channeled by the World Bank were among the key ingredients of the coalition. Partnerships would have been stronger and more effective if a global warning and response system had been in place, a global governing institution coordinating the response, a financing program at the ready including a plan for contributions from various countries. Calls to action have been raised to build a global warning and response system for pandemic outbreaks (Gostin and Friedman 2015) as well as to create a global institution empowered and funded to coordinate a global response system (Gates 2015). World Bank efforts that are currently under way to support in-country pandemic preparedness and control comprise: ƒƒ 31 health sector projects in the pipeline, which offer ready opportunities for strengthen- ing country resilience; ƒƒ Phase III of the REDISSE project which is under preparation and will add 5 West African countries to the 7 that were already covered under REDISSE I and II; ƒƒ and the East Africa Public Health Laboratory Network project, which covers 5 countries. In addition, the World Bank has successfully leveraged global support for a number of new initiatives to support our work on preparedness during the past 24 months, including: ƒƒ The World Bank-Japan Joint UHC Initiative under which the World Bank is supporting 10 countries to accelerate UHC implementation and improve pandemic preparedness; and Independent Evaluation Group | World Bank Group 169 Technical assistance to a number of countries in East Asia on strengthening the financial and institutional sustainability of health security, with financial and technical support from bilateral donors. Public-Private Interactions For the purposes of this analysis, public-private interactions (PPIs) were defined as policies undertaken by the public authorities that impact the private sector; regulatory and licensing regimes aimed at the private sector; and public financing of privately provided service provision (for example, contracting out of services, public-private partnerships, and public insurance systems purchasing privately provided services).5 As such, this analysis of World Bank Group’s support for PPI-related activities covers: ƒƒ World Bank projects financing that help strengthen the government’s stewardship of the health system, and particularly of the private sector ƒƒ IFC AS support to governments in undertaking public-private partnerships (PPPs) that deliver health services6 ƒƒ IFC IS that supports private providers of publicly financed health services Evolution of World Bank Group Strategies Over the last decade and a half, World Bank Group has issued key documents that reflect its evolving strategy toward PPI in health. The first of these is IFC’s health strategy of 2002 (IFC 2002). IFC’s goal in this strategy was to ensure close collaboration with the World Bank and its alignment with the World Bank’s objectives of improving health outcomes, protecting the population from the impoverishing effects of ill health, and enhancing the performance of health services. It also noted that solving the challenges of health care in low and middle-income countries could not be left to the public sector alone because of governmental fiscal constraints but also the poor quality, ineffectiveness, and inefficiency of the public sector in many health systems. It highlighted that many governments around the world were seeking to increase the role of the private sector in the provision of care while complementing the activities of the public sector. Subsequently, the World Bank’s 2007 health strategy, recognized the importance of the private sector in health systems (World Bank 2007). It acknowledged the need to ensure that an effective regulatory framework exists in countries for public-private collaboration to improve health systems and highlighted that “public policy is still not attuned to ensuring public-private complementarity and synergies and effective resource use in the health sector.” While the strategy recognized the World Bank’s comparative advantages in this area, it acknowledged that institutional challenges needed to be addressed to allow the World Bank to better support governments that are interested in engaging the private health sector. It said that the “[World] Bank Group has a potential comparative advantage for contributing to client country development of sound public policy toward the private sector, but the current tendency of the World Bank HNP sector to focus on the public sector and of IFC to focus 170 World Bank Group Support to Health Services | Appendix E on business development for the private sector has created a vacuum in the World Bank Group in terms of supporting client country development of public policy toward the private health sector.”7 The strategy recognized that the World Bank’s capacity in this area will need significant reinforcement and strengthening if its full potential for supporting client countries is to be achieved and it proposed that improving the policy environment for public-private collaboration in the next 18–24 months was a key step in implementation of the strategy. IFC’s 2007 Health Strategy for Sub-Saharan Africa (SSA), explicitly built on the World Bank’s 2007 (IFC 2007). It aimed to increase access to health-related goods and services in SSA by supporting the development of a higher-quality and more robust private health sector in SSA. More recently was a joint technical briefing presented to the World Bank Group Board in 2015 (World Bank and IFC 2015). It argued that (i) UHC cannot be achieved without the private sector; (ii) World Bank Group is uniquely placed to help governments harness the private sector to contribute to UHC; and (iii) World Bank Group can realize internal synergies through a more coherent approach. The document also highlighted the areas where World Bank Group will seek to improve its support to governments and the private sector to improve UHC, including: (i) helping governments to expand service coverage through mechanisms that harnessed existing private providers (for example, PPPs, contracting, vouchers, social franchising, telemedicine); (ii) mobilizing global knowledge and capital to scale up innovative service delivery models; (iii) partnering with large, integrated health care providers to improve system level efficiency and quality and help them further reach the poor; iv) improving stewardship and knowledge of mixed health systems; (v) providing timely and relevant response to country teams related to private sector engagement demands/needs and the provision of high quality technical assistance (based on demand from country teams); (vi) developing standards of practice and an evidence base on private sector engagement in areas relevant to client engagements; (vii) promoting peer-to-peer learning on PPI for policy makers, investors, and World Bank Group staff; and (viii) developing a research agenda for evidence-based interventions by World Bank Group and others. Overall Theory of Change for Public-Private Interaction Given that all systems concerned with the provision of health services comprise both publicly- provided provision and provision by nonstate actors, the theory of change underpinning this evaluation is that improving and expanding the provision of health services requires a focus on, inter alia, improving the interaction between the public and private sectors, including: the role of government in planning for the overall system of provision; government regulation of the private sector to ensure public resources are not misdirected and that private provision does not lead to exploitation through excessive provision and charges or through unsafe care; and government contracting with, purchasing from, and partnering with the private sector to expand availability and quality of health services provided to the public. Independent Evaluation Group | World Bank Group 171 Figure E.12. Public-Private Interaction Theory of Change World Bank Project Financing Support for PPI Stewardship Function Evidence from the Literature: Strengthen the Stewardship Role for PPI The literature suggests that the concept of government acting as a “steward” of their overall health systems has been around for some time. In their World Heath Report in 2000, the World Health Organization (WHO) identified four core functions of any health system: finance, resource generation, service delivery, and stewardship (WHO 2000). In a subsequent paper in 2002, WHO defined stewardship as “providing vision and direction for the health system, collecting and using intelligence, and exerting influence – through regulation and other means.”8 The function of stewardship, of course, entails much more than PPI. And the extent to which PPI makes sense in any given country will depend on the role the private sector plays or has the potential to play in that country. But wherever the private sector plays a substantial role, there is a clear role for government in engaging the private sector in the manner the WHO describes. Despite the experience with a rapidly growing number of interventions, evidence “on which to make wise policy decisions concerning the private sector is often weak or absent” (Horton and Clark 2016, 540). Montagu and Goodman (2016) study concludes that: ƒƒ Banning the private sector where demand for services is high is unlikely to succeed 172 World Bank Group Support to Health Services | Appendix E ƒƒ Implementation and enforcement of regulation to constrain private sector providers in low-income countries is inadequate (with some notable exceptions) ƒƒ Regulation is not effective when low quality (underqualified provision) are the only cred- ible source of care for large populations. In such circumstances, a subsidized health service that is recognized by users as being of adequate quality is needed ƒƒ Inspections role is rare and often not happening mostly due to lack of capacity or re- sources of frontline inspectors and imperfect access to information ƒƒ Accreditation is increasingly common in middle-income countries and often works if it is linked to reimbursements ƒƒ Policies are relatively effective when they are compatible with the financial incentives of providers; that is, when they allow them to pursue their own interests and objectives while at the same time achieving public goals The Portfolio of Projects Supporting the Stewardship Function for PPI World Bank Group support for stewardship in PPI projects is limited. Between 2005 and 2016, there were 46 World Bank projects financing in health that included private sector-related stewardship components (World Bank 2016). This represented 7 percent of total World Bank HNP portfolio. While there have been World Bank–financed projects with private sector components, it seems that very few World Bank–financed projects aimed in a cohesive way at improving government’s ability to better engage with the private sector. In 2014, in internal discussions within World Bank Group also highlighted that there are few existing projects explicitly aimed at improving the government’s ability to engage the private sector (Ibid). In those discussions, some reasons stated constraining this are: lack of consensus about the private sector role, limited financing has been available for private sector analytical work; and recruitment practices have been skewed toward people with public/academic backgrounds (Ibid). The country case studies confirm the limited upstream support provided through World Bank project financing. In Romania, the inpatient sector is still mostly public but private hospitals are visible in the market and growing. However, the government, especially the MOH, is still focusing mostly on public sector hospitals. The public-private referral process is inefficient. Public-private interaction for hospitals and large private networks is limited at most and there is no mention in World Bank Group strategies and there have not been projects with an intention to support MOH in understanding what steps could be taken or what is the private sector role. In Bangladesh, the public sector also uses the private and NGO sectors to enhance public service delivery. Government provides grant allocation (both one-off and yearly for recurrent costs) to many nongovernment entities involved in health services delivery. The World Bank has done little in this area, more analysis is required to fully assess the roles and relationships between the public, private, NGO, and informal health service providers. While the Ministry of Health and Family Welfare has responsibility for the overall health system in Independent Evaluation Group | World Bank Group 173 urban areas and, public primary health service delivery in urban areas is the responsibilities of the respective local government bodies. Since 2010, the World Bank Group Health in Africa Initiative (HIA) has provided support to governments in SSA to understand and engage the private sector in health service delivery. Some examples of the projects that this initiative has undertaken are summarized in the box E.2. Despite this focus, HIA has some considerable limitations. It is geographically constrained to a small group of countries in SSA; it is a specific initiative and its work is not fully integrated into World Bank project financing broader work; and, as it is donor-funded, its facing long-term sustainability challenges. Based on IEG’s interviews with World Bank staff, it seems that the World Bank, outside of the HIA team, has only one individual who is dedicated to upstream advice to governments on PPI issues. This level of resource allocation does not seem to match the ambition that World Bank has in the area of upstream PPI advisory work. Box E.2. Health in Africa Policy Work Expansion of social health insurance to include the poor in Kenya, Ghana, and Nigeria using innovative approaches, including through the private sector South Sudan: Training of nurse anesthetists using a public-private partnership (PPP) model Uganda: Establishment of regulations for accredited drug sellers. Helped to bring 6,000 informal drug sellers into formal market and improve quality of medicine Over 15 legal reforms in multiple countries to improve regulations of the health sector and increase private sector participation in delivering improved quality goods and services Support to the development of public-private partnership frameworks for engagement of private sector in delivering health goods in Kenya, Uganda, South Sudan and Nigeria Support to the creation of the East Africa HealthCare Federation and national private sector federation in 6 countries—a regional platform for engagement between governments and private sector. To What Extent Has World Bank Group Support to Health Services through PPI Effectively Contributed to the Achievement of Relevant Health Services-Related Goals? The development effectiveness of World Bank’s projects supporting PPI stewardship function has been limited and its effectiveness below than the overall Health services portfolio. Out of 24 closed investment projects, 63 percent had outcome ratings in the satisfactory range (MS and above). The highest percentage of satisfactory outcomes have been in Africa (86 percent), South Asia 174 World Bank Group Support to Health Services | Appendix E (71 percent), while in Europe and Central Asia and East Asia and Pacific and Latin America and the Caribbean Regions had 60 percent, 50 percent and 33 percent respectably. Effectiveness success of stewardship is above 70 percent in support to client countries. IEG reviewed the achievement of the indicators that had stewardship components. This specific aspect of strengthening health system functions, which also identify a subset of projects seeking public-private interactions (PPIs), was identified in only 7 percent of World Bank–financed projects. The indicators used to monitor success in the integration was achieved in 70 percent of the cases. However, they were often output-level indicators, such as developing a health sector strategy that comprise the private sector, developing guidelines, policies, or regulations relevant to private providers (for example, licensing and accreditation of health facilities or individual health workers). Some examples of stewardship support and its achievement are presented in box E.3. Box E.3. Examples of World Bank Group Stewardship Support and Its Success In Azerbaijan, the healthcare system remained largely unreformed, and continued to function according to the old Soviet centralized norms, resulting in poorly pooled and inequitably allocated resources. The World Bank helped improve the country’s health system stewardship by developing a health policy framework and a national drug policy, supporting accreditation and licensing programs, and developing a mechanism of quality control and assurance. These efforts were successful due to the government’s commitment and continuous engagement resulting in key background studies and institutional reforms undertaken and ultimately facilitating project preparation and implementation. During the time of the project, health indicators in Moldova remained well below European Union averages as evidenced by some of the highest incidence of chronic diseases in the region. In this context, the government was committed to make the Ministry of Health (MOH) the steward of the health system to implement effective polices and address these critical health issues. The World Bank helped strengthen the MOH’s stewardship capacities to formulate and implement health policies and accelerate the health reforms in primary care, hospital rationalization, payment system for medical facilities, public-private partnerships (PPPs). A conducive government environment toward the development of the MOH stewardship capacities, along with a strong design based on extensive analytical work played a crucial role in the project. One of the challenges for Albania’s health sector was improving the stewardship of the health system to effectively address the growing incidence of noncommunicable diseases. To face them, the MOH and the Institute of Public Health (IPH) needed systemic changes. As a result, the World Bank strengthened the roles of the MOH and and IPH by developing strong formulation and performance monitoring functions in both agencies, developing and implementing a system to monitor provider Independent Evaluation Group | World Bank Group 175 Box E.3, continued. performance, and establishing a licensing and re-licensing program for physicians and health facilities. A strong quality of supervision, along with government flexibility and strong design, was instrumental to find solutions and alternatives to make the project successful. In Cambodia, critical areas such as maternal health had not improved, malnutrition reduction was slow, and quality of care and effectiveness was low during the time of the project. Health sector analyses in Cambodia pointed toward the need for the government to provide stronger leadership and stewardship of the health sector. In this context, the World Bank strengthened health system stewardship functions by developing policy packages and strengthening institutional capacity, increasing private sector regulation and partnerships, and improving governance and stewardship functions of national programs and centers overseeing the Second Health Strategic Plan. To successfully achieve the stewardship objective, the government’s commitment played a crucial role by proactively mobilizing resources and creating additional positions to effectively support project implementation. In the Philippines, there were major disparities in health outcomes across provinces in the Philippines and across income levels at the time of project appraisal. As a result, a national health strategy was approved to organize health reform initiatives into financing, regulation, service delivery, and governance. In this context, the World Bank set out improve health system performance by providing local health system reform grants to selected provinces. However, the project was not successful because there was no clearly defined framework, along with lack of accountability, for implementing agencies. It was further complicated by the limited capacity of agencies at district and state levels. In Turkey, The World Bank has supported the government’s health sector reforms and implementation strategy for the past decade through a series of Adaptable Program Loans which were an integral part of the Government of Turkey’s covering the period 2003 to 2013. One of the main components of the programs, have been to restructure the Ministry of Health for effective stewardship. Results have taken years to materialize with a series of restructurings. Stewardship component were adjusted to focus on strengthening the Strategic Planning and Policy Development Unit of the MOH and the development of a performance management framework for autonomous health facilities. This included the establishment of quality assurance and accreditation policies and procedures and establishing monitoring and evaluation capacity. The reorganization of the Ministry of Health to enable it to focus on policy formulation, regulation and monitoring was completed under the second APL. However, due to the disconnect between the primary objective of assisting the Ministry of Health to become a more effective steward of the health system, and many of the activities (which were largely aimed at improving the quality of service at the primary care level), it is difficult to establish direct link between activities and results. 176 World Bank Group Support to Health Services | Appendix E IFC AS Support to Governments Undertaking Public-Private Partnerships The Theory of Change The PPP model seeks to respond to market failures while minimizing the risk of government failure. Economic theory suggests that private ownership is to be preferred where competitive market prices can be established IMF 2004). Under such circumstances, the private sector is driven by competition to sell goods and services at a price that consumers are willing to pay, and by the discipline of the capital market to make profits. However, various market failures (natural monopoly or externalities and so on) can justify government ownership, for example, in roads, water distribution, or education. But governments—which deliver these services because of market failure in the first place—may subsequently struggle with government failure, as they may have difficulties operating efficiently and containing costs, or they lack the capability to achieve a desired quality standard, or both. In other words, government failure can simply substitute—or may follow—market failure. These arguments can be used to motivate PPPs as an instrument of combining the relative strength of government and private provision in a way that responds to market failure but minimizes the risk of government failure. Figure E.13 depicts a conceptual framework for PPPs in health in terms of combining the relative strengths of government and private provision. This naturally has to take into account health systems context and enabling factors. The basic rational is that the selection of the adequate private operator through transparent and open competition will bring managerial and financial capacities to operate with an entrepreneurial spirit and with the combined role of the public actors including social responsibility public accountability and financial resources will result into high quality health infrastructure and services that will translate in to improved availability and better quality of care. Increased quality is expected to enhance patient satisfaction and demand for services, which would result in a higher use of health services and ultimately contribute to UHC. The objective here is that the selection of a private operator through competition will combine managerial and financial capacities with the public sector’s financial resources and focus on social responsibility and public accountability to improve health infrastructure and services (Roehrich 2014) resulting in improved availability and better quality of care. PPPs work when private sector actors can use their management skills and capacity for innovation to improve efficiency and quality standards. Efficiency gains play an important role in increasing value for money through PPPs. Governments pay a fee to the private partner for the services provided (often split into usage fees and availability payments), which the private sector uses to pay operating costs and interest charges and to repay debt and a return on equity. In cases where efficiency increases offset the higher capital costs of the private sector, the PPP may have a higher value for money and hence be the preferred option for the government. Such efficiency effects may include improved analysis during project selection, better planning, on-time and on-budget implementation, improved construction expertise, and adequate maintenance. PPPs, if implemented well, can therefore help overcome inadequate infrastructure, which constrains economic growth, particularly in developing countries (World Bank 2012). Independent Evaluation Group | World Bank Group 177 Figure E.13. Conceptual Framework for Public-Private Partnerships Evidence from Literature The evidence available in considering the overall success or otherwise of PPPs in health is limited. On the one hand, the literature points to the negative effects on public budgets because of contingent liabilities not being adequately assessed, insufficiently reported, or accounted for off-balance sheet. On the other, there is some available positive evidence of the improved quality of the service through PPPs (Bhatia 2006; Basu, Andrews, Kishore, et al. 2012), but there is mixed evidence on technical quality (Basu, Andrews, Kishore, et al. 2012; Bennett, Hanson, Kadama et al. 2005,). The Portfolio of IFC PPP Advisory Projects About 70 percent (47 out of 67) of IFC’s AS portfolio are PPP transaction advisory projects to governments. Most of IFC’s PPP advisory support has gone to middle-income countries in South Asia, Africa and Europe and Central Asia. This is consistent with the notion that PPP’s are most effective when executed in relatively more mature markets. The main priority of IFC’s PPP advisory practice is to achieve contract closure, that is, to have the public sector enter a contract with the private sector for the provision of services. However, in reality, this is only the start of any project as these contracts typically run for long periods of time (typically 10 178 World Bank Group Support to Health Services | Appendix E to 30 years). So, the projects have longer-term objectives whose achievement can only be properly assessed at the end of this contract period. In terms of the objectives of IFC’s PPP advisory practice, improving access to health has been the top priority. About 90 percent of projects included this objective. 55 percent of projects included improving quality. Only a small percentage of projects had a specific focus on efficiency (12 percent). Private sector development is the top nonhealth related objective. An analysis of trends of project’s goals suggests steady support for improving access and quality. IFC’s PPP advisory projects have mainly supported general health (55 percent of the projects) while 26 percent of projects supported specific noncommunicable diseases (mainly diabetes and cancer). This is not surprising since most of the support (63 percent of projects) is related to hospital and clinics in middle-income countries. The predominant health focus in projects’ interventions is consistent with countries’ epidemiological transitions and income level. The higher the level of gross domestic product per capita the more is the support to general health and noncommunicable diseases. With regards to the level of care supported since 2010 there seems to be a trend away from supporting tertiary care and toward secondary and primary care. To What Extent Has World Bank Group Support to Health Services through PPP Advice Effectively Contributed to the Achievement of Relevant Health Services-Related Goals? A higher percentage of health PPPs reached commercial closure compared with PPP’s overall. The review of all closed projects in terms of the success of IFC AS up to the point of bringing the PPP transaction to contract closure found that 64 percent of all IFC AS mandates9 (16 out of 25) reached contract closure. In arriving at this point, 83 percent of all mandates proceeded to tender with support from IFC AS, bids were received in 67 percent of all mandates, and 63 percent of all mandates proceeded to commercial closure. This is higher than the 50 percent achieved in IFC’s overall PPP portfolio. Among projects that achieved contract closure, the most significant success factors are project design and government commitment. While IFC’s PPP advisory projects have stated long-term development objectives, their achievement against those objectives cannot be measured at project closure because those can only be measured reached a few years after PPP is actually operating. This makes it difficult to properly assess the performance of the PPP portfolio against these longer-term goals. IEG reviewed transactions after project closure based on available external evidence, country case studies, interviews and postcompletion monitoring studies independently commissioned by IFC. Overall, IFC’s PPP projects are contributing to improve access and quality. Out of the 16 projects (20 PPPs)10 that reached contract closure, seven projects (11 PPP’s or 55 percent) are currently operating. IEG reviewed nine (of the 11 sufficiently mature PPP’s) transaction after project closure. Results show specific evidence on access in particular availability in 83 percent of the projects and all of the projects provide evidence on structural quality. Independent Evaluation Group | World Bank Group 179 However, there is insufficient information available to judge aspects of access (such as affordability), efficiency, sustainability and fiscal burden of the PPP’s because postcompletion reports, though all have minimum requirements, they lack a common methodology and a clear framework to measure long-term results. Figure E.14. Results of IFC AS Supporting Public-Private Partnerships The main challenges that affected the performance of PPPs are health systems−related. Inadequate referral process (Brazil, Lesotho), retention of health care professionals (Romania, Lesotho, Mexico), delays in government payment (India, Lesotho), inadequate calculation of government contribution (Brazil), delays in matching human resources availability with infrastructure (Mexico), insufficient government capacity to manage PPPs (Romania, Lesotho). Factors of success. The most important elements of a successful PPP are flexibility of the contracts governing the partnership and commitment of government, including capacity to deal with midcourse corrections when necessary. Government commitment and flexibility to modify contracts have contributed to the long-term sustainability for some PPPs (Brazil, Romania, India). 180 World Bank Group Support to Health Services | Appendix E and Challenges of IFC Advisory Services Supporting Figure E.15. Results Public-Private Partnerships Note: IFC = International Finance Corporation. IFC IS to Private Companies Offering Public-Private Interaction Evidence from Literature on Private Companies Offering PPI A recent paper published in the Lancet breaks the private sector into “four stylized private provider types: (i) the low-quality, underqualified sector that serves poor people in many countries; (ii) not- for-profit providers that operate on a range of scales; (iii) formally registered small-to-medium private practices; and (iv) the corporate commercial hospital sector, which is growing rapidly and about which little is known” (McPake and Hanson 2016, 622). IFC generally invests in the fourth of these stylized provider types. The paper assesses the literature insofar as it addresses the performance of the corporate commercial health sector and the contribution of that sector to achieving UHC. It ultimately concludes that “evidence is patchy and inadequate” (McPake and Hanson 2016, 626) to draw any conclusions relating to these questions. The article also suggests that IFC (and the Commonwealth Development Corporation) should invest more in understanding the broader impact of their investment activities in the health sectors in low and middle-income countries. Achievement of UHC requires pooled, mainly public financing, but can be compatible with various roles for private health providers, under effective public stewardship. Success in stewardship of the health system through the transition to UHC in pluralistic health systems will require policies that recognize the links between the public and private sectors and that work at the system level to improve performance throughout” (McPake and Hanson 2016, 628–629). This conclusion not only reinforces the point, made in numerous times in this report, regarding the importance of the stewardship role, but also makes clear the importance of investing in the private companies that Independent Evaluation Group | World Bank Group 181 have a role in the driving toward UHC. Unfortunately, there is insufficient literature to clearly say which types of these companies, and which types of PPI interactions, make the most sense. The Portfolio of PPI Projects Investing in Companies Offering PPI IFC made 89 investments in hospitals, clinics, specialty chains and medical laboratories between 2005 and 2016 into companies that provide health services. Of these, 48 (53 percent) were into companies that were “mixed” that is, they served both private markets as well as provided services to the public sector; three (6 percent) were into pure public-private partnerships – all of them in Turkey; and 41 were into companies that only served private markets. (see box below). To What Extent Has World Bank Group Support to Private Companies Offering PPI Effec- tively Contributed to the Achievement of Relevant Health Services-Related Goals? The development outcome of IFC’s projects supporting PPI investments has been positive than the overall IFC investments in health. Out of 12 evaluated investment projects, 83 percent had a satisfactory or better development outcome rating. This result is significantly higher than the rest of IFC investments11 (67 percent). Satisfactory outcomes are more likely in low-income (100 percent) than in upper-middle- and lower- middle-income countries (78 and 75 percent, respectively). In terms of specific objectives, the IFC-PPI has no significant differences with that off the entire portfolio toward improving access and quality to health care services. The share of projects that successfully achieved their access objectives (73 percent) show no significant differences with that of the overall IFC portfolio (73 percent). Key objectives related to access were often associated to availability (for example, number of beds increased, increase in coverage). and use of services (Number of patients reached, number of patients treated). However, other important dimensions of access such as affordability (for example) and effective use was less frequently used in projects’ monitoring and evaluation frameworks. Similarly, the share of projects that successfully achieved their quality objectives (78 percent) has not been more significant than that of the IFC health services portfolio (73 percent). Key indicators related to quality is often associated with structural quality (for example, certification such as iso 9000, licensing and accreditation standards required for health facilities) but very little emphasis is put in technical quality and process measures of quality (see Summary table 2 below). and Indicators Achievements in International Finance Table E.6. Objectives Corporation Investment Projects Type Indicator No PPI =16 PPI =12 Totals Access =25 Access (affordability) No Indicator 3 3 Access (availability) Number of Beds Increased 4 4 Number of companies Access (availability) 1 1 2 invested through the fund Number of drugs manufac- Access (availability) 1 1 tured 182 World Bank Group Support to Health Services | Appendix E Table E.6, continued. Type Indicator No PPI =16 PPI =12 Totals Access (Utilization) Number of patients reached 3 5 8 Access (Utilization) Number of patients treated 1 2 3 Equity =4 Equity No Indicator 3 1 4 Quality =17 Structural Quality Accreditation 6 4 10 Quality No Indicator 2 3 5 Structural Quality Number of Staff Trained 2 3 5 Job Creation = 13 Job Creation New Jobs created 8 5 13 Private Sector Development =9 Private Sector Development Various Indicators 5 4 9 Note: IFC =International Finance Corporation. While good, these results, do not inform the extent of the contribution to PPI objectives. A review of hospitals and specialty chains interventions have PPI characteristics show that in practice private hospitals tend to rely more on out-of-pocket resources than on public financing. The main reasons for this is uncertainty of public funding, the lack of adequate pricing of services and lack of adequate regulation. (see box E.4). As markets mature However, opportunities to invest more in more integrated within the public system are starting to emerge in some developing countries. There are However, examples where the projects have been more successful to include public funding successfully (Turkey). Box E.4. Examples of Limited Contribution to Public-Private Interaction Objectives In India, an IFC-supported client did not progress as the expected rate of government health insurance subsidies for patients below the poverty line was not sufficient for sustainable private sector engagement, such as the REACH model. In Romania, private sector provider relies on out-of-pocket expenditures as public funding is capped two low and is constrained by public health care budgets. In Macedonia, IFC supported a private hospital whose income depended solely on government resources (state health fund), which posed a high risk for the project if government delayed payments. The client should have tried to broaden its customer base. A project supported in China needed to apply for accreditation by the Chinese state insurance but this never materialized so as a result the company had to rely on private payments. Independent Evaluation Group | World Bank Group 183 Box E.4, continued. Public-private joint venture hospital in China included establishment of public-private partnership model by forging joint venture s with leading public hospitals. For public hospitals, the joint venture with private providers was expected to free up public sector resources that would have otherwise been used for the treatment of those who can pay for services. This was one of the stated aims of the Chinese government which encouraged more private sector health care. Since the joint venture had partial success government reduced its focus on partnering with private sector. In Colombia, the health insurance plan covers 95 percent of the population. However, the IFC client is facing liquidity issues due to delays on payments from government and insurance/Administration agents. The current collection cycles of is about 200 days, which is outside of the companies control. Faced with this long collection period, the client has managed to diversify its mandatory health plan patient mix with some private insurance and foreign patients. Currently the company has about 5 percent of its profits from contracts abroad with Caribbean governments. Findings from the PPI Portfolio World Bank project financing upstream advice. The World Bank’s intent in providing upstream advice in PPI has been clear since 2007 and was restated in its 2015 technical briefing to the Board. Its comparative advantage in this area is also clear. However, to date, its follow through has not matched this intent. No serious resources have been applied to the challenge of bringing expertise and advice to governments in an area – private sector engagement – that the World Bank historically has found ideologically challenging. Overall, therefore, we must conclude that the World Bank has some considerable work to do to improve its relevance in the provision of advice to government on PPI. IFC AS to PPP. These projects are generally successful in bringing transactions to commercial closure. Results of mature PPPs shows achievement the expected access and structural quality objectives. However, there is insufficient information available to judge as specific aspects of technical quality, affordability, efficiency and sustainability of the PPP’s. PPP’s must be done taking into account a holistic view of the health system and elaborated how will they contribute to the system overall. Main challenges supported PPP’s encounter were health system related. But most important factors of a successful PPP’s. But still Qualified sponsor and committed government and flexibility to make midcourse corrections contracts is crucial over time. PPPs in health is an area of activity where there is very little evidence as to the circumstances in which those arrangements work and those where they do not. Few operating PPP’s offer valuable lessons but knowledge generated and disseminated by IFC has so far been limited. The postcompletion reports are useful but are not systematic and largely lack comparability among PPP’s and baselines/targets at project design IFC’s IS Services. While IFC’s investments in health have been overall successful and they are contributing to expand access through the increase in coverage and structural quality. There is insufficient evidence to judge critical aspects of access such affordability of resources and important 184 World Bank Group Support to Health Services | Appendix E aspects of quality. PPIs shows few full successful integration of the supported private providers in the publicly financed health system. While IEG found that there is close to 59 percent of projects offer some sort to PPI (financing from governments). The assessment shows few full successful integration of the supported private providers in the publicly financed health system. Often private providers rely on full out-of-pocket payment, which limit financial affordability for the poor. This often happens because of the insufficient availability in the public budgets and because of the low pricing assigned to the services. References Angelucci, M., and O. Attanasio. 2013. “The Demand for Food of Poor Urban Mexican Households: Under- standing Policy Impacts Using Structural Models.” American Economic Journal: Economic Policy 5 (1): 146–205. American Veterinary Medical Association. 2008. One Health: A New Professional Imperative: One Health Initiative Task Force: Final Report. Schaumburg, IL: American Veterinary Medical Association. Attanasio, O., E. Battistin, and A. Mesnard. 2012. “Food and Cash Transfers: Evidence from Colombia.” The Economic Journal 122 (559): 92–124. Arraiz, I., and S. Rozo. 2011. “Same Bureaucracy, Different Outcomes in Human Capital? How Indigenous and Rural Non-Indigenous Areas in Panama Responded to the CCT.” IDB (Inter-American Devel- opment Bank) Office of Evaluation and Oversight Working Paper OVE/WP-03/11, IDB, Washing- ton, DC. http://idbdocs.iadb.org/WSDocs/getdocument.aspx?docnum=36180403. Bassani, D.G., P. Arora, K. Wazny, M. F. Gaffey, L. Lenters, and Z. A. Bhutta. 2013. “Financial Incentives and Coverage of Child Health Interventions: A Systematic Review and Meta-Analysis.” BMC Public Health 13 (3): S30. Bawo, L., K. Leonard, and R. Mohammed. 2015. “Protocol for the Evaluation of a Quality-Based Pay for Performance Scheme in Liberia,” Implementation Science 10 (9): 1–11. Barber, S. L., and P. J. Gertler. 2008. “Empowering Women to Obtain High-Quality Care: Evidence from an Evaluation of Mexico’s Conditional Cash Transfer Programme.” Health Policy and Planning 24 (1): 18–25. Brad Herbert Associates. 2012. Health in Africa Mid-Term Evaluation: Final Report. Washington, DC: World Bank. Basinga, P., P. J. Gertler, A. Binagwaho, A. L. Soucat, J. Sturdy, and C. M. Vermeersch. 2011. “Effect on Maternal and Child Health Services in Rwanda of Payment to Primary Health Care Providers for Performance: An Impact Evaluation.” The Lancet 377 (9775): 1421–1428. Basu, S., J. Andrews, S. Kishore, R. Panjabi, and D. Stuckler. 2012. “Comparative Performance of Private and Public Healthcare Systems in Low-And Middle-Income Countries: A Systematic Review.” PLoS Medicine 9 (6): e1001244. Independent Evaluation Group | World Bank Group 185 Benabou, R., and J. Tirole. 2003. “Intrinsic and Extrinsic Motivation.” The Review of Economic Studies 70 (3): 489–520. Bennett, S., K. Hanson, P. Kadama, and D. Montagu. 2005. “Working with the Nonstate Sector to Achieve Public Health Goals.” WHO (World Health Organization) Making Health Systems Work: Working Paper 2, WHO, Geneva. Bhatia, B., and N. Gupta. 2006. “Lifting Constraints to Public-Private Partnerships in South Asia.” Pub- lic-Private Infrastructure Advisory Facility Gridlines Note 6, World Bank, Washington, DC. Blacklock, C., E. MacPepple, S. Kunutsor, and S. Witter. 2016. “Paying for Performance to Improve the Delivery and Uptake of Family Planning in Low- and Middle-Income Countries: A Systematic Review.” Studies in Family Planning 47 (4): 309–324. Boccia, D., J. Hargreaves, K. Lönnroth, E. Jaramillo, J. Weiss, M. Uplekar, J. D. H. Porter, and C. A. Ev- ans. 2011. “Cash Transfer and Microfinance Interventions for Tuberculosis Control: review of the Impact Evidence and Policy Implications.” International Journal of Tuberculosis and Lung Disease 15 (6): S37–S49. Bonfrer, I., R. Soeters, E. van de Poel, O. Basenya, G. Longin, F. van de Looij, and E. van Doorslaer. 2013. “The Effects of Performance-Based Financing on the Use and Quality of Health Care in Burundi: An Impact Evaluation.” The Lancet 381 (S2): S19. Celhay, P., P. Gertler, P. Giovagnoli, C. Vermeersch. 2015. “Long-Run Effects of Temporary Incentives on Medical Care Productivity.” Policy Research Working Paper 7348, World Bank, Washington, DC. Chalkley, M., A. Mirelman, L. Siciliani, and M Suhrcke. 2016. “Paying for Performance for Health Care in Low- and Middle-Income Countries: An Economic Perspective.” Center for Health Economics Research Paper 140, University of York, York, U.K. Das, A., S. S. Gopalan, and D. Chandramohan. 2016. “Effect of Pay for Performance to Improve Quality of Maternal and Child Care in Low-and Middle-Income Countries: A Systematic Review.” BMC Public Health 16 (1): 321. de Walque, D., P. Gertler, S. Bautista-Arredondo, A. Kwan, C. Vermeersch, J. Bizimana, A. Binagwaho, and J. Condo. 2015. “Using Provider Performance Incentives to Increase HIV Testing and Counseling Services in Rwanda.” Journal of Health Economics 40 (2015): 1–9. Eichler, R., K. Agarwal, I. Askew, E. Iriarte, L. Morgan, and J. Watson. 2013. Performance-Based Incentives to Improve Health Status of Mothers and Newborns: What Does the Evidence Show?” Journal of Health, Population, and Nutrition 31 (4 Suppl. 2): S36. Eldridge, C. and N. Palmer. 2009. Performance-Based Payment: Some Reflections on the Discourse, Evi- dence and Unanswered Questions.” Health Policy and Planning 24 (3): 160–166. Ellis, C. M., and J. Chaffin. 2015. “Evaluations of Outcomes for Children and Youth from NGO-Supported Microeconomic Interventions: A Research Synthesis.” Enterprise Development and Microfinance 26 (2): 104–121. 186 World Bank Group Support to Health Services | Appendix E Engineer, C., E. Dale, A. Agarwal, O. Alonge, A. Edward, S. Gupta, H. Schuh, G. Burnham, and D. Peters. 2016. “Effectiveness of a Pay-for-Performance Intervention to Improve Maternal and Child Health Services in Afghanistan: A Cluster-Randomized Trial.” International Journal of Epidemiology 45 (2): 451–459. Fair, M. 2008. “From Population Lending to HNP Results: The Evolution of the World Bank’s Strategies in Health, Nutrition, and Population.” IEG Working Paper 2008/3, World Bank, Washington, DC. Fasina, F. O., A. Shittu, D. Lazarus, O. Tomori, L. Simonsen, C. Viboud, and G. Chowell. 2014. “Transmis- sion Dynamics and Control of Ebola Virus Disease Outbreak in Nigeria, July to September 2014.” Euro Surveillance 19 (40): 20920. Ferré, C., and I. Sharif. 2014. “Can Conditional Cash Transfers Improve Education and Nutrition Outcomes for Poor Children in Bangladesh? Evidence from a Pilot Project.” Policy Research Working Paper 7077, World Bank, Washington, DC. Fiszbein, A., and N. Schady. 2009. Conditional Cash Transfers: Reducing Present and Future Poverty. Washington DC: World Bank. Friedman, J., A. Das, and R. Mutasa. 2016. “Rewarding Provider Performance to Improve Quality and Coverage of Maternal and Child Health Outcomes. Zimbabwe Results-Based Financing Pilot Program Evidence to Inform Policy and Management Decisions.” Report No: 106518-ZW, World Bank, Washington, DC. Fritsche, G., Soeters, R., and Meessen, B. 2014. Performance-Based Financing Toolkit. Washington, DC: World Bank. Gaarder, M. M., A. Glassman, and J. E. Todd. 2010. “Conditional Cash Transfers and Health: Unpacking the Causal Chain.” Journal of Development Effectiveness 2 (1): 6–50. Gates, B. 2015. “The Next Epidemic—Lessons from Ebola.” New England Journal of Medicine 372 (15): 1381–1384. Gertler, P., and C. Vermeersch. 2012. “Using Performance Incentives to Improve Medical Care Productiv- ity and Health Outcomes.” Policy Research Working Paper 6100, Impact Evaluation Series 60, World Bank, Washington, DC. Giuffrida, A., H. Gravelle, and M. Roland. 1999. “Measuring Quality of Care with Routine Data: Avoiding Confusion between Performance Indicators and Health Outcomes.” BMJ 319 (7202): 94–98. Glassman, A., D. Duran, L. Fleisher, D. Singer, R. Sturke, G. Angeles, J. Charles, B. Emrey, J. Gleason, W. Mwebsa, and K. Saldana. 2013. “Impact of Conditional Cash Transfers on Maternal and Newborn Health.” Journal of Health, Population, and Nutrition 31 (4 Suppl. 2): S48. Gopalan, S. S., R. Mutasa, J. Friedman, and A. Das. 2014. Health Sector Demand-Side Financial Incen- tives in Low- and Middle-Income Countries: A Systematic Review on Demand- and Supply-Side Effects.” Social Science & Medicine 100: 72–83. Independent Evaluation Group | World Bank Group 187 Gostin, L. O., and E. A. Friedman. 2015. A Retrospective and Prospective Analysis of the West African Eb- ola Virus Disease Epidemic: Robust National Health Systems at the Foundation and an Empow- ered WHO at the Apex.” The Lancet 385 (9980): 1902–1909. Heymann DL, Chen L, Takemi K, Fidler DP, Tappero JW, Thomas MJ, Kenyon TA, Frieden TR, Yach D, Nishtar S, Kalache A. 2015. Global health security: the wider lessons from the west African Ebola virus disease epidemic. The Lancet. 15;385(9980):1884–901. Horton, R., and S. Clark. 2016. “The Perils and Possibilities of the Private Health Sector.” The Lancet 388 (10044): 540. Hunter, B. M., S. Harrison, A. Portela, and D. Bick. 2017. “The Effects of Cash Transfers and Vouchers on the Use and Quality of Maternity Care Services: A Systematic Review.” PLoS One 12 (3): e0173068. Hurst, T. E., K. Semrau, A. Gawande, and L. R. Hirschhorn. 2015. “Demand-Side Interventions for Maternal Care: Evidence of More Use, Not Better Outcomes.” BMC Pregnancy and Childbirth 15 (1): 297. IFC (International Finance Corporation). 2002. Investing in Private Health Care: Strategic Directions for IFC. Washington, DC: The World Bank Group. ———. 2007. Health Strategy for Sub-Saharan Africa. Washington, DC: The World Bank Group. ———. 2011. Healthy Partnerships: How Governments Can Engage the Private Sector to Improve Health in Africa. Washington, DC: The World Bank Group. IMF (International Monetary Fund). 2004. Public-Private Partnerships. Washington, DC: IMF. Jain, V., A. Alam. 2017. “Redefining Universal Health Coverage in the Age of Global Health Security.” British Medical Journal Global Health 2 (2): e000255. Kalk, A., F. A. Paul, and E. Grabosch. 2010. “‘Paying for Performance” in Rwanda: Does It Pay Off?” Tropi- cal Medicine and International Health 15: 182–90. Khan, M. E., A. Hazra, A. Kant, and M. Ali. 2016. Conditional and Unconditional Cash Transfers to Improve Use of Contraception in Low- and Middle-Income Countries: A Systematic Review.” Studies in Family Planning 47 (4): 371–383. Khim K., Annear, PL. (2013). Strengthening district health service management and delivery through inter- nal contracting: lessons from pilot projects in Cambodia, Social Science and Medicine, 96: 241–9 Kieny, M. P., D. B. Evans, G. Schmets, and S. Kadandale. 2014. “Health-System Resilience: Reflections on the Ebola Crisis in Western Africa.” Bulletin of the World Health Organization 92 (12): 850–850. Lagarde, M., A. Haines, and N. Palmer. 2009. “The Impact of Conditional Cash Transfers on Health Out- comes and Use of Health Services in Low- and Middle-Income Countries.” Cochrane Database of Systematic Reviews Oct 7 (4): CD008137. 188 World Bank Group Support to Health Services | Appendix E Lee, K., and D. Fidler. 2007. “Avian and Pandemic Influenza: Progress and Problems with Global Health Governance.” Global Public Health 2 (3): 215–234. Lemière, C., G. Torsvik, O. Mæstad, C. Herbst, and K. Leonard. 2013. “Evaluating the Impact of Re- sults-Based Financing on Health Worker Performance.” HNP Discussion Paper 98269, World Bank, Washington, DC. Målqvist, M., B. Yuan, N. Trygg, K. Selling, and S. Thomsen. 2013. “Targeted Interventions for Improved Equity in Maternal and Child Health in Low-and Middle-Income Settings: A Systematic Review and meta-analysis.” PLoS One 8 (6): e66453. Manley, J., S. Gitter, and V. Slavchevska. 2013. “How Effective Are Cash Transfers at Improving Nutritional Status?” World Development 48: 133–155. McPake, B., and K. Hanson. 2016. “Managing the Public-Private Mix to Achieve Universal Health Cover- age.” The Lancet 388 (10044): 622–630. Miake-Lye, I. M., S. Hempel, R. Shanman, and P. G. Shekelle. 2016. “What Is an Evidence Map? A Systematic Review of Published Evidence Maps and Their Definitions, Methods, and Products.” Systematic Reviews 5 (1): 28. Montagu, D., and C. Goodman. 2016. “Prohibit, Constrain, Encourage, or Purchase: How Should We En- gage with the Private Health Care Sector?” The Lancet 388 (10044): 613–621. Murray, S. F., B. M. Hunter, R. Bisht, T. Ensor, and D. Bick. 2012. Demand‐Side Financing Measures to Increase Maternal Health Service Utilisation and Improve Health Outcomes: A Systematic Review of Evidence from Low- and Middle-Income Countries.” JBI Database of Systematic Reviews and Implementation Reports 10 (58): 4165-4567. Okello, A. L., K. Bardosh, J. Smith, S. C. Welburn. 2014. “One Health: Past Successes and Future Chal- lenges in Three African Contexts.” PLoS Neglected Tropical Diseases 8 (5): e2884. Perova, E., and R. Vakis. 2009. Welfare Impacts of the “Juntos” Program in Peru: Evidence from a Nonex- perimental Evaluation. Washington, DC: World Bank Ranganathan, M., and M. Lagarde. 2012. “Promoting Healthy Behaviours and Improving Health Outcomes in Low- and Middle-Income Countries: A Review of the Impact of Conditional Cash Transfer Pro- grammes.” Preventive Medicine 55: S95–S105. Renmans, D., N. Holvoet, C. Orach, and B. Criel. 2016. “Opening the ‘Black Box’ of Performance-Based Financing in Low- and Lower-Middle-Income Countries: A Review of the Literature.” Health Policy and Planning 31: 1297–1309. Roehrich, L. G. 2014. “Are Public-Private Partnerships a Health Option? A Systematic Literature Review.” Social Science & Medicine 113: 110–119. Independent Evaluation Group | World Bank Group 189 Rudasingwa, M., and M. Uwizeye. 2017. “Physicians’ and Nurses’ Attitudes toward Performance-Based Fi- nancial Incentives in Burundi: A Qualitative Study in the Province of Gitega.” Global Health Action 10:1, 1270813, DOI: 10.1080/16549716.2017.1270813 Ssengooba, F., B. McPake, and N. Palmer. 2012. Why Performance-Based Contracting Failed in Uganda: An “Open-Box” Evaluation of a Complex Health System Intervention.” Social Science & Medicine 75 (2): 377–383. Shen, G., H. Nguyen, A. Das, N. Sachingongu, C. Chansa, J. Qamruddin, and J. Friedman. 2017. “Incen- tives to Change: Effects of Performance-Based Financing on Health Workers in Zambia.” Human Resources for Health 15: 20. Siedner, M. J., L. O. Gostin, H. H. Cranmer, and J. D. Kraemer. 2015. Strengthening the Detection of and Early Response to Public Health Emergencies: Lessons from the West African Ebola Epidemic.” PLoS Medicine 12 (3): e1001804. Siedner, M. J., and J. D. Kraemer. 2017. The End of the Ebola Virus Disease Epidemic: Has the Work Just Begun?” The Lancet Global Health 5 (4): e381–e382. Suthar, A. B., J. M. Nagata, S. Nsanzimana, T. Bärnighausen, E. K. Negussie, and M. C. Doherty. 2017. “Performance-Based Financing for Improving HIV/AIDS Service Delivery: A Systematic Review.” BMC Health Services Research 17 (1): 6. Tanner, J., R. Hayashi, and Y. Li. 2015. “Improving Coverage and Utilization of Maternal and Child Health Services in Lao PDR: Impact Evaluation of the Community Nutrition Project.” IEG Working Paper 2015/2, World Bank, Washington, DC. Turcotte-Tremblay, A., J. Spagnolo, M. De Allegri, and V. Ridde. 2016. “Does Performance-Based Financ- ing Increase Value for Money in Low- and Middle- Income Countries? A Systematic Review.” Health Economics Review 6: 30. Van de Poel, E., G. Flores, P. Ir, and O. O’Donnell. 2016. “Impact of Performance-based Financing in a low-resource setting: a decade of experience in Cambodia.” Health Economics 25: 688–705. van Wijk, C. M. G., K. P. Van Vliet, and A. M. Kolk. 1996. “Gender Perspectives and Quality of Care: Toward Appropriate and Adequate Health Care for Women.” Social Science & Medicine 43 (5): 707–720. Vellez, M. 2015. Contracting-out Primary Health Care Services using Performance-Based Payments: An evaluation of the Honduras’ Experience. https://ssrn.com/abstract=2631766 or http://dx.doi. org/10.2139/ssrn.2631766 Witter, S., A. Fretheim, F. L. Kessy, and A. K. Lindahl. 2012. “Paying for Performance to Improve the Delivery of Health Interventions in Low-and Middle-Income Countries.” Cochrane Database of Systematic Reviews Feb 15 (2): CD007899. World Bank. 2004. World Development Report 2004: Making Services Work for Poor People. Washington, DC: The World Bank Group. 190 World Bank Group Support to Health Services | Appendix E ———. 2007. Healthy Development: The World Bank Strategy for Health, Nutrition, & Population Results. Washington, DC: The World Bank Group. ———. 2011. Social Safety Nets: An Evaluation of World Bank Group Support Since, 2000–2010. Indepen- dent Evaluation Group. Washington, DC: World Bank. ———. 2012a. Resilience, Equity, and Opportunity. Social Protection and Labor Strategy. Washington, DC: World Bank. ———. 2012b. Public-Private Partnerships Reference Guide, Version 1.0. Washington, DC: World Bank. ———. 2013. Using Results-Based Financing to Achieve Maternal and Child Health: Progress Report. Washington, DC: World Bank. ———. 2014a. Social Safety Nets and Gender: Learning from Impact Evaluations and World Bank–Fi- nanced Projects. Washington, DC: The World Bank ———. 2014b. “World Bank Policy Note Impact Evaluation on Performance-Based Financing in Haut-Ka- tanga District, Democratic Republic of Congo. Washington, DC: World Bank. ———. 2014c. Responding to Global Public Bads. Learning from Evaluation of the World Bank Experience with Avian Influenza 2006–13. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016. Approach Paper: An Evaluation of World Bank Group Support to Health Services. Indepen- dent Evaluation Group. Washington, DC: World Bank. ———. 2017. Cameroon Performance-Based Financing Impact Evaluation Report. Washington, DC: World Bank. World Bank and IFC (International Finance Corporation). 2015. Technical Briefing to the Board: Joint World Bank Group Approach to Harnessing the Private Sector in Health. Washington, DC: The World Bank Group, April 24 World Health Organization. 2000. World Health Report: Health Systems Financing: The Path to Universal Coverage. Geneva: World Health Organization ———. 2002. Towards Better Stewardship: Concepts and Critical Issues. Geneva: World Health Organiza- tion. ———. 2010. World Health Report: Health Systems Financing: The Path to Universal Coverage. Geneva: World Health Organization. ———. 2015. Successful Ebola responses in Nigeria, Senegal, and Mali. Geneva: World Health Organiza- tion. http://www.who.int/csr/disease/ebola/one-year-report/nigeria/en. Yuan, B., M. Målqvist, N. Trygg, X. Qian, N. Ng, and S. Thomsen. 2014. “What Interventions Are Effective on Reducing Inequalities in Maternal and Child Health in Low-and Middle-Income Settings? A Systematic Review.” BMC Public Health 14 (1): 634. Independent Evaluation Group | World Bank Group 191 1 The review is not covering the political economy surrounding the decision-making process to adopt a PBF program. For example, Van de Poel et al. (2015) suggests that the Cambodian government opted for contracting-in (thus PBF) instead of a contracting-out, because both of budgetary consideration and the reservation about devolve health service provision to international NGOs. Concerns about the perception of privatization steaming from contracting international NGOs (Vellez 2015). 2 The 2005 IHR have the scope to prevent, protect against, control and provide a public health response to the international spread of disease in ways that are commensurate with and restricted to public health risks, and which avoid unnecessary interference with international traffic and trade. 3 One Health is the collaborative effort of multiple disciplines—working locally, nationally, and globally—to attain optimal health for people, animals and the environment (American Veterinary Medical Association, 2008). 4 In Nigeria, the World Bank Group had supported financially the “Partnership for Polio Eradication” between April 2003 and March 2011 through four projects with a cumulative total credit of $190.4 million. 5 This is broadly consistent with the assessment framework used in “Healthy Partnerships: How Governments Can Engage the Private Sector to Improve Health in Africa” (IFC 2011). 6 It excludes Health Financing PPPs and Studies and Regional PPPs. 7 Ibid, page 44 8 Towards Better Stewardship: Concepts and Critical Issues, WHO, 2002, page 1 9 This review excludes health insurance PPPs, studies and regional PPP’s The reason there are a different number of projects and PPPs is that two projects included more than one PPP. 10 Specifically, a project in Mexico include two PPPs and a project in Romania included four PPPs. 11 Excluding Pharmaceuticals, Funds, Health Financing projects. 192 World Bank Group Support to Health Services | Appendix E Appendix F. Case Study Analysis of Selected Countries Country case studies are developed to assess the alignment between World Bank Group support to health services and countries needs and priorities; the extent to which synergies and complementarities among the different type of World Bank Group support to health services were achieved; and the role and contribution of the World Bank Group within the country-level partnership supporting health services. The Independent Evaluation Group (IEG) conducted six in-depth analysis of country-level support, from which three involved country visits and three were desk-based reviews. Selection of countries was based on a purposeful sample based on geographic coverage, income level, fragility, mix government capacity to manage development assistance, mix of hi/low World Bank Group support to total health expenditures, complexity of the donor network and capacity of the recipient country (see appendix A for details on sampling selection). The scatterplot of the two variables (that is, complexity of the development partners’ network and the capacity of the recipient country) and highlights the six countries selected for in-depth analyses are located (see figure F.1). Table F.1 presents the characteristics of the selected countries. Figure F.1. Country-Level Partnerships: Complexity and Capacity of the Recipient Country Note: The two variables are normalized around zero. Complexity of the development partners network is measured by the number of actors providing DAH. Higher values denote more complex networks. Government capacity is measured using the World Bank country policy and institutional assessment rating for the health sector. Higher values denote lower capacity. 193 Table F.1. Characteristics of the Countries Selected for the Case Studies World Bank Group Support as a percentage of Total Health Financing Country (percent) Income Region Fragility Liberia 1.3 Low Income AFR Yes Philippines 0.3 Lower-Middle Income EAP No Romania 0.8 Upper-Middle Income ECA No Brazil 0.1 Upper-Middle Income LCR No Republic of Yemen 0.7 Lower-Middle Income MNA Yes Bangladesh 1.8 Lower-Middle Income SAR No Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. World Bank Group Alignment to Country Needs World Bank Group’s alignment to country needs is assessed considering: (i) The alignment of projects with country priorities (Health and system needs); (ii) Alignment with World Bank Group country strategies and country health and system needs. Table F.3 presents the results of the analysis. Table F.3. Alignment to Country Needs How Well Are the Identified Country Priorities Were World Bank Group Country (as Stated in National Strategies) Reflected in Strategies Aligned with Health and System Country World Bank Group Portfolio? Needs? Liberia World Bank Group projects have partly supported Yes, World Bank Group strategies were the country development needs. Health Sector mostly in line and have partly supported the Review (HSR): (i) strengthen the policy framework country development needs. and selected management functions of the Min- istry of Health and Social Welfare; and (ii) improve preservice training and selected components of the basic package of health services. Bangladesh World Bank financed project objectives were con- Yes/Mixed. World Bank Group Country sistent with country needs and the government’s Partnership Strategies since 2006 initially strategies. The World Bank–supported SWAPs emphasized achievement of the MDGs and sought to address critical needs in improving later indicated increasing support for system- public service delivery to meet the Millennium ic reforms. Development Goals. The SWAPs also supported broad, longer-term improvements in the health system. However, there was a relatively lower focus on urban health services than warranted by Bangladesh’s rapidly increasing urban population. Republic of The World Bank–financed projects were consis- Yes. The World Bank’s support is in response Yemen tent with the country needs. The World Bank’s to the country need and priorities which strategy was service delivery and improved evolved after crisis. access and the MoPHP had set its highest priority to address the high rates of child mortality and maternal mortality, as well as addressing disease-specific health needs such as the high prevalence of schistosomiasis and malaria. 194 World Bank Group Support to Health Services | Appendix F How Well Are the Identified Country Priorities Were World Bank Group Country (as Stated in National Strategies) Reflected in Strategies Aligned with Health and System Country World Bank Group Portfolio? Needs? Brazil Projects at the federal level have attempted to Yes, but demand has been volatile. follow identified health concerns but in every case, government has shifted focus and/or decided to use its own funds, leaving World Bank–financed projects less relevant, canceled, or with unsat- isfactory outcomes. At the state level: World Bank–financed projects with health focus in 8 of Brazil’s 27 States/entities, mostly broader public services projects with health components, varied types of objectives and results. This is in the context of a drought of health project financing in Latin America and the Caribbean in general. With the fiscal crisis in Brazil, the World Bank’s focus is now on efficiency of spending, mainly through analytic work Philippines The World Bank’s support is well-aligned with the The World Bank’s strategies have been well- country’s priorities. (1) The focus of the largest aligned with country priorities. project is on improving governance (introducing better ways to procure commodities) and health care financing (improving PhilHealth). In both cas- es, this was matched by financing.) The second major theme is primary health, with a focus on safe motherhood. This has been the theme of an investment loan (for a pilot in five provinces) and a trust fund project. Romania Yes aligned. Both National Health Strategies ad- Yes. World Bank Group strategies were opted by the Government of Romania in FY06–16 mostly in line and have partly supported the had very broad objectives which addressed the country development needs, stressed the many health sector challenges and accommodat- need to focus on health system redesign and ed most World Bank Group projects. increased spending efficiency. Note: SWAP = sectorwide approach program. The World Bank Group’s Role in Country-Level Partnerships World Bank Group’s role in country-level partnerships is assessed considering: (i) its financing role as a percentage of total country health expenditures and compared with the role of the other development partners; (ii) its coordination role; and (iii) its technical leadership. Table F.4 presents the results of the analysis. Independent Evaluation Group | World Bank Group 195 Table F.4. The World Bank Group’s Role in Country-Level Partnerships Technical Financing Substantial (World Bank Assistance Country or Group as a percentage of Total Health and Capacity Expenditure) Coordination Knowledge Low Government 3/3 1/3 0/3 Capacity Liberia 1.3. Limited, Good with Ebola but Limited World Bank Group is the largest con- still, high level of duplication and tributor of total financial inflows to the fragmentation among donors’ government activities Bangladesh 1.8. Limited, Fiduciary role has domi- Selected In all SWAPs, the World Bank was the nated the support largest single contributor among the development partners Republic of 0.8. Limited before crises a stronger Limited Yemen The financing role after the crisis has collaboration with UNICEF and been critical. The World Bank is one of WHO was achieved because the few remaining development partners these agencies were grant (along with WHO and UNICEF) active in recipients as well as the managing the country. It has canceled all undis- and implementing entities on an bursed commitments in all other sectors exceptional basis but expanded scope for health projects High Govern- 0/3 0/3 1/3 ment Capacity Brazil 0.05 Limited Selected Limited, multiple federal-level health projects were scaled back or canceled because the government preferred to proceed with its own resources. Philippines 0.284 Limited Selected during crisis Romania 0.812 Limited Limited significant fiscal government constraints, and the lack of capacity to absorb funds from European Commission 196 World Bank Group Support to Health Services | Appendix F Table F.5. How Relevant Has World Bank Group Support to Health Services Been to the Main Health Needs and Priorities? How well are the identified Country Priorities (as Stated in National Strategies) Reflect- Were World Bank Group ed in the World Bank Group Country strategies aligned with Country Context/Challenges Country Strategies World Bank Group Priorities Portfolio? Health and system needs Use World Bank Group country PAD (World Bank), Board Re- Was support aligned with Use Literature review, Health Use National Health priorities (country partnership port (IFC) Portfolio Review, and country strategies and Health sector ESW). Strategies: strategies) CAS reviews Interviews and system needs Romania •Frequent changes in central Both National Health Strategies World Bank Group strategies Yes, aligned. Both National Health Yes, World Bank Group strate- leadership and legislation prevent adopted by GoR in FY06–16 had are mostly in line and have partly Strategies adopted by GoR in gies were mostly in line and have needed changes in the health very broad objectives. The national supported the country develop- FY06–16 had very broad objec- partly supported the country sector. •Governance issues• health strategy adopted in 2004 ment needs and stressed the tives which addressed the many development needs, stressed the Lack of sustainable government included health sector reforms, need to focus on health system health sector challenges and need to focus on health system commitment to reform: in the past hospital rationalization, hospital redesign and increased spend- accommodated most World Bank redesign and increased spending 26 years,26 Health Ministers and modernization, ambulatory and ing efficiency. IFC’s support in Group projects. Decisions regard- efficiency. 17 NHIH presidents in Romania). primary care strengthening, strategies have stayed pretty ing the health system are taken Became European Union (EU) community care development, much the same: Each of the by the Government and Ministry member state since 2007•The improved and increased health country strategies indicated that of Health without necessarily fol- health system is still mostly financing, private sector develop- IFC would seek opportunities to lowing the national health strategy centralized. Most Providers are ment, decentralization, improved invest in health services. or any other existing strategies. private except for hospitals. prevention, better coordination, The national health strategy Public provision of services is not human resources planning, and approved in 2004 was planned to mandatory for private provid- so on the national health strategy be implemented, monitored and ers. HR: Significant Migration adopted for the period 2014–2020 evaluated by a Committee but of Doctors and Nurses due to has an overall scope, six basic this never happened. Govern- low salaries and EU accession. principles and many objectives ment Programs had no mention The existing health workforce is organized by three strategic areas: about the NHS in their Health inequitably distributed. Insufficient public health, health services chapters. The national health Drs in Rural areas. Formal mech- and cross-section measures. strategy 2014–2020 has to be Independent Evaluation Group | World Bank Group anism for dialogue exists between The action plan goes into much evaluated every year through an public and private providers and detail and provides indicators and implementation report and the NHIH but in practice dialogue is targets but budgets are not clearly first report was published in July 197 inefficient. Equity issues. provided for every item. 2016 for the previous year. How well are the identified Coun- 198 try Priorities (as Stated in National Were World Bank Group Strategies) Reflected in the World Country strategies aligned with Country Context/Challenges Country Strategies World Bank Group Priorities Bank Group Portfolio? Health and system needs Use World Bank Group country PAD (World Bank), Board Re- Was support aligned with Use Literature review, Health Use National Health priorities (country partnership port (IFC) Portfolio Review, and country strategies and Health sector ESW). Strategies: strategies) CAS reviews Interviews and system needs Liberia Nearly 14 years of civil unrest and From humanitarian aid to devel- 1. CS 2007–2010: Improving World Bank Group projects have Yes, World Bank Group strategies instability left Liberia with a weak- opment 2011–14.The second Na- the delivery of basic services partly supported the country were mostly in line and have part- ened and seriously dilapidated tional Health and Social Welfare (including health) was one of development needs. HSR: (i) ly supported the country devel- health system incapable of ade- Plan and Policy 2011–21 focuses the four key pillars of both the strengthen the policy framework opment needs. In the first phase quately responding to the health to the quality of health and social iPRS and the PRS. 2. CPS and selected management func- of reconstruction focused on (i) needs of women and children. welfare services, to be delivered 2011–2017 Improved capacity of tions of the Ministry of Health and strengthen selected management System challenges: Liberia is still close to communities in a manner health service delivery in selected Social Welfare (MOHSW); and (ii) functions of the MOHSW; and (ii) near the bottom of the rankings responsive to patient needs, and secondary-level health facilities improve preservice training and improve preservice training and on almost all comparative health with management delegated to (CPS, p25)(a) improve the quality selected components of the basic selected components of the basic indicators. The key health system lower administrative levels. Core of care for services with proven package of health services. package of health services (that World Bank Group Support to Health Services | Appendix F constraints: • Inefficient supply functions of the Ministry (planning, effectiveness; (b) increase the is, Infrastructure and Equipment, chain management system • research and development) and availability of qualified graduate and Human Resources). Inadequate number and limited the tertiary hospital subsector physicians (pediatricians, obste- skills of health workforce • Low were strengthened. The Ministry tricians, general surgeons, and availability of/limited access to is now the only line agency to internal medicine specialists, with and demand for adequate health receive direct funding from part- cross-cutting focus on anesthe- facilities • Lack of accountability ners (for example, the Pool Fund siology); (c) enhance the clinical and inadequate monitoring and comprising DFID, Irish Aid, GTZ competencies and motivation of supervision systems • Weak and Swiss cooperation). This is mid-level cadres (nurses, mid- leadership, management, and due in large part to the strength wives, and physician assistants), governance capacity at the and commitment of the Ministry and (d) improve provider account- county and district levels • Youths leadership and good harmoniza- ability mechanisms related to and adolescents’ issues • Health tion of partner assistance both the achievement of results, Transition and the Double Burden and health worker performance at of Disease. selected facilities Bangladesh • Over the past 15 years, •The government has shown World Bank Group Country World Bank financed project Yes/Mixed. World Bank Group Bangladesh saw substantial consistent commitment to Partnership Strategies since 2006 objectives were consistent with Country Partnership Strategies improvements in health outcomes improving health service delivery initially emphasized achievement country needs and the govern- since 2006 initially emphasized and achieved most health-related and advancing health outcomes. of the MDGs and later indicated ment’s strategies. The World achievement of the MDGs and MDGs. • However, progress has As part of its revised HNPSP increasing support for systemic Bank–supported SWAPs sought later indicated increasing support been lacking in some areas: that (2003–2011), the government reforms. Each of the country to address critical needs in im- for systemic reforms. Each of is, neonatal mortality remains identified several broad strategies strategies indicated that IFC proving public service delivery to the country strategies indicated substantially above targets; child to guide its approach to modern- would seek opportunities to meet the MDGs. The SWAPs also that IFC would seek opportuni- malnutrition still remains high. • izing health services and improv- invest in health services. In 2010 supported broad, longer-term ties to invest in health services. Bangladesh achieved positive ing health outcomes.). In 2011, the government identified achiev- improvements in the health World Bank Group projects health outcomes at relatively the government identified priority ing universal health care as the system. However, there was a were responsive to government low public cost: expenditure on objectives in health services. In main goal of its new health sector relatively lower focus on urban commitment to improving health health in 2014 was 2.8% of gross 2015, Bangladesh embraced the plan. The plan aims to increase health services than warranted by outcomes and to government domestic product, compared Sustainable Development Goals the number of poor people en- Bangladesh’s rapidly increasing strategies and priorities in health with an average of 4.4% in South (SDGs) for 2030, including SDG rolled in Phil Health and improve urban population. In addition, in services. The initial SWAPs were Asia. • Various factors over the 3 which focuses on ensuring the outpatient. retrospect, a greater emphasis largely donor-driven but govern- last decade served to improve health and promoting well-being on helping strengthen public ment ownership has increased demand for health services: rising (World Bank PID 2017) A specific oversight of private sector and substantially over the period. overall per capita incomes; rapid objective is to achieve universal informal health service providers growth of the garment industry health coverage, which encom- was warranted. The avian flu re- that employed a high proportion passes assuring access to HNP sponse project’s objectives were of women; widespread commu- services without causing financial relevant, although weaknesses nity empowerment programs by hardship. were apparent in the project’s NGOs; access to safe water sup- design. ply; access to microcredit; and rural infrastructure development. • On the supply side, Bangladesh has developed a pluralistic health service supply sector, with public, private-for-profit, NGO, and informal service providers having broad reach across the country. • Governance issues are pervasive, both in the public sec- Independent Evaluation Group | World Bank Group tor and outside the public sector. 199 How well are the identified Coun- 200 try Priorities (as Stated in National Were World Bank Group Strategies) Reflected in the World Country strategies aligned with Country Context/Challenges Country Strategies World Bank Group Priorities Bank Group Portfolio? Health and system needs Use World Bank Group country PAD (World Bank), Board Re- Was support aligned with Use Literature review, Health Use National Health priorities (country partnership port (IFC) Portfolio Review, and country strategies and Health sector ESW). Strategies: strategies) CAS reviews Interviews and system needs Philippines • Macroeconomic instability. The government has a relatively The World Bank’s strategies have The World Bank’s support is The World Bank’s strategies have Although the economy is relatively consistent and stable health been well-aligned with country well-aligned with the country’s been well-aligned with country strong now, it has a history of sector strategy. In 1998, the priorities. The 2005 Strategy’s priorities. (1) The focus of the priorities. The 2005 Strategy’s instability. • Governance and Philippines launched the Health overarching objective was to largest project is on improving overarching objective was to corruption. Corruption is endem- Sector Reform Agenda. This “[help] to build public institutions governance (introducing better “[help] to build public institutions ic, leading to both inefficiencies was a major health sector reform that serve the common good.” ways to procure commodities) that serve the common good.” and a high level of administrative with many aspects, ranging from The project also supported the and health care financing (improv- The project also supported the controls to reduce the risk of demand-side reform (expand- strategy’s focus on social inclu- ing PhilHealth). In both cases, strategy’s focus on social inclu- corruption. • Local conflict. While ing PhilHealth) to supply side sion, specifically “improved per- this was matched by financing. sion, specifically “improved per- restricted to certain areas, it (regulation, accreditation, and so formance of national institutions The current ASA is focused on formance of national institutions World Bank Group Support to Health Services | Appendix F can be intense. • Inequality and on) and greater decentralization. and increased access for the strengthening PhilHealth as did and increased access for the poverty. While poverty has gone The World Bank was not much poor and disadvantaged groups the canceled (never taken to the poor and disadvantaged groups down, it is still relatively high. In- involved in this process. to basic service.” Regarding Board) loan. The health section in to basic service.” Regarding equality has increased. • Natural This was replaced in 2005 by regional inequality, the strategy the development policy loan (DPL) regional inequality, the strategy disasters. The country is on a ma- FOURmula ONE for Health, which focused on “[providing] greater series also focused on PhilHealth. focused on “[providing] greater jor typhoon zone and has been was an attempt to simplify the voice and improved access for (2) The second major theme is voice and improved access for severely affected over the years. reform and keep it more focused. the poor and disadvantaged primary health, with a focus on the poor and disadvantaged Health Sector: • The Philippine The reform that aimed to align in the planning and delivery of safe motherhood. This has been in the planning and delivery of government is quite committed health sector reform with the education, health, and other the theme of an investment loan education, health, and other to health system reform, as part broader public expenditure and basic services at the local level.” (for a pilot in five provinces) and a basic services at the local level.” of a long-term political agenda. • governance reforms. It had four (The key projects were approved trust fund project. (The key projects were approved There are significant governance strategic areas: (i) health system under this Strategy).and inpa- under this Strategy).The current issues in much of the public delivery; (ii) Health regulation; tient benefits package. For the World Bank Group’s partnership sector, which affects the health (iii) health finance; and (iv) good 2011–2016 plan, the government strategy (2015 to 2018) has a sector. • The private sector has management. The priorities have has identified achieving universal broad theme of the strategy is to always played a major role in remained the same. The two sub- health care as the main goal. The ensure that the poor receive the providing healthcare at all levels. sequent presidents maintained all current World Bank Group’s part- benefits of recent growth in the • Private insurance is well estab- the main features, changing levels nership strategy (2015 to 2018) Philippines. Objective 2.1 focuses lished and traditionally available of intensity or adding new has a broad theme of the strategy on improving access of basic to the wealthier groups. • Public elements. The government’s most is to ensure that the poor receive services, particularly increasing health insurance has been recent development strategy, the the benefits of recent growth in health insurance coverage and available for formal sector public Philippine Development Plan— the Philippines. Objective 2.1 fo- supporting the government’s uni- and private workers. • Spending 2017–2022 identifies the need to cuses on improving access of ba- versal health care objective (World on public health has increased continue to focus on improving sic services, particularly increas- Bank, 2014). The government substantially in the past five years. reproductive health and fighting ing health insurance coverage designed the directions of the • Traditionally there has been a other communicable diseases. and supporting the government’s health strategy before the World shortage of nurses due to foreign The plan calls for continued universal health care objective Bank was active. employment, however, now this increase in investment in health (World Bank, 2014). is less of problem as salaries have increased. • The public services and continued growth health sector is decentralized to in the coverage in public health the municipal level. The national insurance. government continues to play a major role in setting priorities, financing, and in procurement. • The demographic and epidemio- logical transitions are incomplete. Brazil Typical upper-middle income 1988 Constitution made health The World Bank’s strategies have Projects at the federal level have Yes, but demand has been country profile: almost universal care a basic right. 1994 Family been well-aligned with country attempted to follow identified volatile. immunization coverage; 98% Health Strategy, SUS in 1996, priorities. CAS 2003–2007: health concerns (for example, women receive prenatal care; provides for universal integrated Focused on supporting a country family health in late 2000s, hospital 98.1% births with skilled atten- primary care: preventive and health strategy centered on modernization, regional health dant in 2012. By early 2010s, basic health care using multi- universal access to primary network formation, AIDS) but SUS (single public financing sys- disciplinary professional teams, health care (this has remained in every case, government has tem) had contributed to improved each with a specific geographic the core of the health strategy. shifted focus and/or decided to health outcomes, increased cov- catchment area. there was also a focus on the use ts own funds, leaving World erage of primary care, increased Entirely publicly funded. poor both urban and rural. Focus Bank–financed projects less rele- equity, but was on access coverage. In the vant, canceled, or with unsatisfac- not demonstrable impact on new CPS (2008–2011) the focus tory outcomes. At the state level: quality of care. Increasing shifted to quality, efficiency and World Bank–financed projects concern with noncommunicable performance accountability and with health focus in 8 of Brazil’s diseases, aging. Challenges shifted away from single disease 27 States/entities, mostly broader with coverage in frontier regions programs that lacked links to the public services projects with (Amazon, Northeast), urban poor overall health system and greater health components, varied types Independent Evaluation Group | World Bank Group (slums), particularly due to human engagement at subnational level. of objectives and results. This is in resource (doctor) shortages. 50% CPS 2011–2015 focus in low-in- the context of a drought of health of financing and 80% of service come households through public project financing in LAC in general. 201 provision is private. Country and private channels and regional With the fiscal crisis in Brazil, How well are the identified Coun- 202 try Priorities (as Stated in National Were World Bank Group Strategies) Reflected in the World Country strategies aligned with Country Context/Challenges Country Strategies World Bank Group Priorities Bank Group Portfolio? Health and system needs Use World Bank Group country PAD (World Bank), Board Re- Was support aligned with Use Literature review, Health Use National Health priorities (country partnership port (IFC) Portfolio Review, and country strategies and Health sector ESW). Strategies: strategies) CAS reviews Interviews and system needs not homogenous; States and service delivery networks /referral the World Bank’s focus is now municipalities are main providers mechanisms and provision of on efficiency of spending, mainly of health care. analytic work. the focus Brazil is a through analytic work. Overall, large and sophisticated borrower, it is foreseen less health-specif- and the World Bank’s project ic project financing due to low financing envelope has become demand, and also issues from the small in relation to federal budget. supply side (with the focus on IDA Attempt to maximize val ue-add- and a shift instead to knowledge ed by engagement with subna- generation with middle-income tional jurisdictions where technical countries). Overall IFC has been technical, financial support and responsive to country concern knowledge services most valu- for increased quality/efficiency of World Bank Group Support to Health Services | Appendix F able -- there’s a sense that the services, but caution warranted: World Bank can contribute more private insurance not available to at the state/municipal all, and equity concerns remain level, closer to where services valid in a very unequal country. IFC are delivered, and can better invested in large hospital network, customize service delivery within meeting demand for quality ser- framework of federal regulations. vices from the one-quarter of the population with private insurance, consolidating and improving management of small (mostly rural) hospitals to avoid shutdowns, investing significantly in training of health workforce. Another IFC investment in laboratory network providing MRIs outside urban areas where there has been no access to these diagnostic ser- vices, significantly increasing use of teleradiology with centralized physicians and technicians. Republic of Yemen Country Context: The Republic MoPHP had set its highest Service delivery and improving The World Bank–financed Before crisis - The World Bank’s of Yemen was a low-income priority to address the high rates access. Country Engagement projects were consistent with support is in response to the country until 2008. It became a of child mortality and maternal Note F17–18 - Support Service the country needs. The World country need and priorities. The lower-middle income country in mortality, as well as addressing Delivery. Provide emergency Bank’s strategy was service World Bank’s interventions have 2009. It became a FCV country disease-specific health needs support to preserve local service delivery and improved access and been focused on service delivery in 2011. Ongoing conflict has such as the high prevalence of delivery capacity in health. This the MoPHP had set its highest and maternal and child health left 18.8 million people in need schistosomiasis and malaria. To will be done in full partnership priority to address the high rates to achieve MDG 4 &5. This was of humanitarian assistance and formulate a strategic approach with UN institutions. Particular of child mortality and maternal in line with MoPHP priority of placed overwhelming strain on for these priorities, in 2005 the focus on women and youth. The mortality, as well as addressing addressing the high rates of child the country’s health system at MoPHP and its development two ongoing projects in the health disease-specific health needs mortality and maternal mortal- a time when it is needed most. partners initiated a dialogue to sector, the Schistosomiasis Con- such as the high prevalence of ity, as well as addressing dis- Less than 45% of health facilities revisit the health sector strategy trol Project and the Health and schistosomiasis and malaria• ease-specific health needs such are fully functioning and at least and evaluate the governments Population Project, will continue The reduction of the number of as the high prevalence of schisto- 274 facilities have been damaged and donors’ approaches to to be implemented by the WHO maternal and neonatal deaths as somiasis and malaria. The World or destroyed during the current improve health sector perfor- and UNICEF. These projects are well as the number of deaths of Bank used its early project fi- conflict. • Healthcare workers mance and its efficiency and a expected to provide effective children under five is one of the nancing operation (Health Reform have been forced to relocate and health sector review (HSR) was implementation avenues for main expected results listed in Support Project 2002–2009) to the ones still in post have not proposed. The objectives for the coordination with other financiers the 2010–2025 National Health pilot specific health interventions received their salaries regularly GoY’s health sector during this (for example, the Global Alliance Policy. This is supported by the (malaria and schistosomiasis) as in around six months. Medical period were very general. The for Vaccines and Immunization) in Maternal and Newborn Voucher well as demand-side models (out- supplies are chronically in short objectives of the Ministry of Public expanding the coverage of critical Project.• Given the critical health reach innovations). A stand-alone supply despite extensive support Health and Population (MoPHP) vaccines and medical supplies needs in the field (during ongoing Schistosomiasis Control Project from WHO and Health Cluster national health strategy 2010–25 to the Republic of Yemen using crisis), the World Bank lifted— was approved FY10. After crisis members, further complicating are: 1. Fulfilling a better health lev- the outreach model of service on an exceptional basis—the - Essential inputs to the health the delivery of lifesaving health- el for the entire population of the delivery. CAS FY10–13 Further suspension of disbursements for facilities and outreach teams have care in the country. • Almost Republic of Yemen in cooperation strengthen delivery of health ser- two health projects, namely the become scarcer and, in many 14.5 million people, including with the other sectors.2. Facilitat- vices. In health, the sectoral focus Schistosomiasis Control Project places, nonexistent. This is most two million internally displaced, ing access to quality health care will be on strengthening capacity and the Health and Population evident in: (a) severe shortages of lack access to clean water, services equally to all people.3. and efficiency for the delivery of a Project• In light of these im- essential medicines and medical sanitation and hygiene services, Increasing the performance level basic package of health services, mediate priorities and pressing supplies required with huge increasing the risk of infectious of the health system and the improving delivery of reproductive healthneeds, the World Bank has disruptions in procurement, trans- diseases such as acute watery efficiency of work and workers at health services, and reducing responded by supporting two port and supply chain capabilities; diarrhea, malaria and scabies. • all levels.4. Proper response to the incidence of schistosomiasis. projects: (i) the Schistosomiasis (b) diminished safe potable water The number of healthcare provid- the population needs and provi- CAS 2006. 64. Pillar Two: Improv- Control Project to decrease the from the public domain and lack Independent Evaluation Group | World Bank Group ers in Republic of Yemen (either sion of appropriate health care ing human development through high prevalence and intensity of of essential fuel, power, main- public or private) providing quality services.5. Raising the awareness more efficient service delivery and infection of schistosomiasis in tenance, water pumps among service is very limited. level on the health matters improved safety nets. This pillar partnership with WHO and the others; (c) insufficient operational 203 How well are the identified Coun- 204 try Priorities (as Stated in National Were World Bank Group Strategies) Reflected in the World Country strategies aligned with Country Context/Challenges Country Strategies World Bank Group Priorities Bank Group Portfolio? Health and system needs Use World Bank Group country PAD (World Bank), Board Re- Was support aligned with Use Literature review, Health Use National Health priorities (country partnership port (IFC) Portfolio Review, and country strategies and Health sector ESW). Strategies: strategies) CAS reviews Interviews and system needs The health facilities often lack ad- and contributing to decreasing would focus on improving access Schistosomiasis Control Initiative; and logistical resources for essen- equate equipment and supplies, the population growth rate and and quality of health. and (ii) the Yemen Health and tial health and nutrition programs and this is compounded by a facing the social detriments of Population Project to contribute at first-level referral centers, scarcity of physicians and other health development along with to the acceleration of the achieve- especially for emergency obstetric medical professionals, including supporting better quality of life.6. ment of MDG 4 & 5 (reduction in and maternal care as well as the poor quality and reduced Mobilizing extra resources to fund childhood mortality and improve- referral nutrition services, further standards of medical training in the health services and focus on ment of maternal health) through risking the lives of hundreds of the country. Among those Yeme- social health insurance. support for key initiatives targeted thousands. Consequently, the nis that can afford to, many travel to improve access to and use Expanded Program for Immuni- abroad for private treatment. • of maternal, neonatal and child zation and national vaccination Limited existence and functional- health (MNCH) in selected campaigns have been interrupt- ity of formal insurance programs governorates with poor MNCH ed, threatening the reemergence World Bank Group Support to Health Services | Appendix F (public sector mandatory health indicators.. of some vaccine preventable dis- insurance program is nonfunc- eases. Also, pockets of new dis- tional; some employment-based eases that are usually associated programs (private and public with conflict -stricken countries companies) provide limited bene- (for example, cholera and tra- fits); limited existence of informal choma). The recently approved insurance programs. Emergency Health and Nutrition Project addresses these needs. What has the role of the World Bank Group been in global and country-level partnerships supporting health services? For this section, plan to do interviews. (1) MOH official (for example, there is a unit in MOH that would typically deal with partnerships). (2) Interview task team leaders of relevant projects. (3) Health specialist of the country (4) At a minimum three development partners. What Are the Com- World Bank World Bank Group What are the Complemen- plementarities or Group Coor- Technical Knowledge Main tarities or Synergies between Synergies of World World Bank dination Role Role (a Proxy for Institutional Coordination World Bank Group Institu- Bank Group with Other Group Financing (Proxy for Con- Convening Power as Actors Mechanisms tions and Instruments? Donors? Role vening Power). Technical Authority) Ask stakeholders: In (health systems, Was World Bank Group noncommunicable support adequately sequenced Ask stakeholders of and communicable Dimensions to look at: Across 1) very successful disease, Health service GP’s? Between IFC and World collaborative projects in delivery, awareness, Bank? By type of support which the World Bank is communication cam- Look at the main Describe if there are formal or Health Systems then Service involved. 2) In the sector, paigns) who are the Actors in the informal coordination mecha- Delivery? Adequate sequenc- what is the collaborative Measure the actors considered to country. This nisms between development ing (that is, Analytical support initiatives in which the World Bank Ask stakeholders be knowledge leaders may include partners. If there are who leads (ASA) followed by World Bank/ coordination has been Group financial in how many in the country. To multilaterals, the coordination role. (that is, IFC project, World Bank health successful. 3) among support relative activities did whom do you turn to bilateral, NGOs, government MOH, a bilateral, systems strengthening followed the most important to government the World Bank MOH what are the top Include the TOP a multilateral, and so on). What by IFC investment? IFC Advi- collaborative initiatives, Health spending. played a leading, three players. HS, SD, 10. Describe is the role of the government in sory Services followed by IFC does the World Bank Compare to what proactive, reac- Awareness, education them in table 1. this: strong or weak? investment? feature among them. others are doing tive role info campaign. Romania Six Multilaterals There is lack of coordination Support is relevant and timely Largely project driven. The World Bank Throughout the Given Romania’s need active, Two mechanisms among DPs and (although it does not make a EIB has been a partner Group has a evaluation period, for sustainable reforms, bilateral donors lack of current leadership to sustainable impact. There were of the World Bank broader focus on there have been the unique World Bank are active. Six undertake the role. MOH co- important delays in implemen- Group for the whole health services some instances Group role as a knowl- NGOs, three pri- ordinates separately with DPs. tation and many design adjust- period, cofinancing compared all the of coordination edge provider and vate companies But the role of the government ments. World Bank and IFC APL2. However, more other multilateral/ among develop- policy adviser is key, (CSR)EAP are is weak, although there are support have aligned objectives recently World Bank bilateral organiza- ment partners but not sufficiently ac- the most active, departments, committees, over time. The original CPS is supporting equip- tions that are active including with knowledged and used with less than project management units in envisioned a phased reduction ment in building three in Romania and World Bank, by MOH and other $10M support the structure of various min- in IFC investment to invest se- regional hospitals and has had a strong but it has been development partners. given annually istries (MOH, MoF, Ministry of lectively focusing on opportuni- EIB is supporting with financing role in largely project World Bank Support between them. European Funds). ties to support projects outside infrastructure but there absolute terms. driven. EIB has is relevant and timely Independent Evaluation Group | World Bank Group There are hun- Bucharest and complementing is no apparent coordina- World Bank has been a partner of but does not make a dreds of other the proposed IBRD IPF within tion among development unique position in the World Bank sustainable impact. NGOs and the sector. partners. that supports Group for the 205 What Are the Com- World Bank World Bank Group 206 What are the Complemen- plementarities or Group Coor- Technical Knowledge Main tarities or Synergies between Synergies of World World Bank dination Role Role (a Proxy for Institutional Coordination World Bank Group Institu- Bank Group with Other Group Financing (Proxy for Con- Convening Power as Actors Mechanisms tions and Instruments? Donors? Role vening Power). Technical Authority) private compa- Due to risks associated with Also, IMF had close col- Romania with both whole period, nies supporting the European debt crisis and laboration with the World upstream and cofinancing local health the economic downturn, IFC Bank Group between downstream func- APL2. However, services through assumed a countercyclical role, FY09–12 in support for tions. On a year more recently sponsoring, but with selective private sector the health reform under- average, World World Bank the sums are investments, including health. taken under the DPLs. Bank Group has is supporting small. DPLs1–3 supported a provided morefi- equipment in comprehensive reform nancing that all the building three agenda, and the policy main development regional hospitals dialogue was supported partners in Roma- and EIB is by an array of technical nia, although exact supporting with assistance through a support of EC infrastructure joint program of World funding in health but there is no Bank, IMF, and EC. is hard to assess. apparent coor- World Bank Group Support to Health Services | Appendix F According to dination among interviews, World development Bank is recognized partners. Also, among players as IMF had close one of the main collaboration actors providing with the World financing in the Bank Group and sector. World Bank EC between consults and en- FY09–12 in gages key actors support for the on specific areas health reform but coordination under the DPLs. is rather project driven with UNICEF and WHO, EIB (APL2), IMF, EC. Liberia 62 organizations 1. Health Sector. Three mo- World Bank support focused World Bank Group sup- There are two dis- World Bank con- The MOH and WHO (4 Governmental dalities sector wide support on few key aspects and was port is complementary tinct financial flows: vening services are the two top knowl- Institutions, 15 (DfID Irish aid) with earmarked generally incremental. For and synergic with other one is through the in Liberia have edge leaders, followed Multilaterals funding supervision and coordi- example, the support to human DPs. It focused in three government and fostered Liberia by the organizations in 12 Bi-laterals nation that goes through the resources development started key areas: (i) Strength- the other is directly participation in the second layer such and 31 NGOs budget. Fairly uncoordinated with the HSR project in 2007 en policy making and to implementing the IHP+. On as USAID, UNICEF & Foundations) support through a lot of NGO’s that provided salary for clinical management functions; NGOs Lead the April 13, 2016, and UNFPA, and the were identified mainly funded by USAID but teachers at medical school (ii) Infrastructure and financial manage- Dr. Bernice World Bank Group through this MOH does not have that much and critical staff, and financed equipment; and (iii) ment assessment. Dahn, Minister is positioned in the process. connection or power over that. the renovation of the medical Human resources for Lead financial of Health and third layer. Knowledge World Bank is hands off of school and the nurse anesthe- health. World Bank resources mobiliza- Social Welfare leadership is measured implementation. tists training. The HSS in 2011 Group support to infra- tion (see Ebola re- of Liberia signed by in-degree, which supported further human re- structure and equipment sponse and Global the compact in is the number of in- sources through the creation of in particular, focuses in Financing Facility). Washington, DC, coming connections is the graduate medical residency the area where there is The World Bank in the presence positioned in the third program and PBF incentives. the largest fiscal gap – Group is the larg- of Dr Margaret layer with UNICEF and Graduate medical residency 60–70%. est contributor to Chan, Direc- UNFPA. Distribution of programs were not available; the financial flows tor-General of knowledge leadership thus it is considered an effec- to the government. the World Health is uneven. This means tive strategy to strengthen HR In 2016, The Organization and that fewer percentage as it reduces the need to move World Bank Group Dr Tim Evans, of organizations are abroad for specialization and provided $55 mil- Senior Director considered as knowl- reduce potential brain drain. lion to the health for the Health, edge leaders in the Finally, during the Ebola out- sector in Liberia Nutrition and network, while many break, the EERP project sup- and $20 million Population Glob- others are not con- ported the Human Resources directly went to the al Practice, at sidered as knowledge Scale up (hazard payment and government, which the World Bank leaders death benefit) health worker’s accounts for 25% Group. census, MOH staff support for of total financial medical education and labora- inflows to the tory training, AMD renovation. government. • HNP and SPL and Agricul- ture. The EERP financed the provision of cash transfer to Independent Evaluation Group | World Bank Group support communities affected by the Ebola virus. The EERP financed the provision of seeds and the construction of 207 What Are the Com- World Bank World Bank Group 208 What are the Complemen- plementarities or Group Coor- Technical Knowledge Main tarities or Synergies between Synergies of World World Bank dination Role Role (a Proxy for Institutional Coordination World Bank Group Institu- Bank Group with Other Group Financing (Proxy for Con- Convening Power as Actors Mechanisms tions and Instruments? Donors? Role vening Power). Technical Authority) warehousing facilities to restart the economic activities in the agriculture sector that virtually collapsed during the Ebola outbreak. • HNP and IFC. The IFC AS to establish a National Diagnostic Center PPP could represent a joint World Bank/IFC project as it is envisaged that USD2 mil- lion from the EERP would be allocated to this project. Bangladesh 2006–11 HNP- The SWAP mechanism proved Yes. The series of SWAP’s Most of the DP’s active For both the The World Bank Key distinctive con- World Bank Group Support to Health Services | Appendix F SP: CIDA, DFID, an effective means of coor- reflected a coordinated effort to in the health sector oper- SWAPs, the played a leading tributions that donors EU, USAID, dinating the contributions of support a common sequenced ated in coordination with World Bank was role in the con- consistently mention SIDA, JICA, various development partners set of objectives since the each other through the the largest single sortium of DPs of the World Bank GTZ/KFW, Neth- around common objectives. 1990s. SWAP framework. Some contributor among that participated included its overall erlands, WHO, Several DP coordination DPs (that is, USAID, the DPs. The in the SWAP leadership and lead in UNICEF, UNFPA, mechanisms were embedded ADB) had projects out- World Bank also programs. The financial management UNDP, GFATM, in the SWAP. (i) the overall Joint side the SWAP. However, administered the World Bank also and procurement Gavi, Islamic Cooperation Framework. (ii) the these other activities multidonor trust administered the areas. Representatives Development Task Groups for each thematic were broadly in line with funds to support multidonor trust of the DPs that met Bank.2011–17 area in the health sector that the SWAP objectives implementation of funds to support with the IEG evaluation HPNSDP: EC, and approach in health each SWAP. The implementation mission expressed DFID, JICA, services (mostly large World Bank played of each SWAP. consistent appreciation CIDA, SIDA, infrastructure projects). a leading role in the The World Bank of the leadership and AusAID, USAID, consortium of DPs was also active technical contributions GIZ, KFW, WHO, that participated in other forums of the UNICEF, UN- in the SWAP pro- such as the An- FPA, UNAIDS, grams. The World nual /Mid-term GFATM, Ga- Bank’s leadership Program Review vi-HSS. 2011–17 in the Steering HPNSDP: EC, sets the agenda for technical SWAP program Committee World Bank in the DFID, JICA, discussions by among the DPs helped encourage and Technical SWAPs, particularly in CIDA, SIDA, and government; and (iii) Local participation and Assistance Com- terms of financial man- AusAID, USAID, Consultative Group meetings mobilize additional mittee. How- agement and procure- GIZ, KFW, for GOB-DP dialogue on financing from ever, the World ment. In Bangladesh, WHO, UNICEF, strategic issues. However, the other development Bank Group the SWAP did include UNFPA, UN- SWAPs were not comprehen- partners. However, played mainly technical assistance to AIDS, GFATM, sive and the government also after attempts to a fiduciary role, implement some ambi- Gavi-HSS. implemented other develop- support health which created tious reforms, but most ment projects in the health sector reforms, the implementation were not implemented sector outside the SWAP, on program largely challenges that (World Bank 2014) its own and supported by slipped into pre- have dominated DPs. The result was a hybrid serving the status the nature of its approach where DP’s funded quo, focusing on support. other projects as well. In increasing financing general, however, these other and service delivery. activities were broadly in line In addition, the with the SWAP (mostly large complicated nature infrastructure). of the administrative arrangements creat- ed significant chal- lenges during the implementation of the program (World Bank 2014e). Philippines   The government is open to The projects were well coor- The donors have coor- The World Bank’s According to In the Philippines, the coordination and supports ef- dinated. They were carried dinated their focus and provided financing interviews, the technical assistance forts to promote coordination. out simultaneously. There was cooperate. There does during a fiscal World Bank is a provided by the World Coordination has been done little coordination between the not appear to be any crisis, although respected part- Bank to build the uni- on a case by case basis. The World Bank and IFC. important overlap. The much of the ner. With a small fied targeting system government is quite supportive EC and the World Bank funding came after number of part- (Listahanan) was a key of development assistance have worked closely the worse years. ners, there is no element in the success and improving donor efficiency. together, providing The World Bank’s need for a lead of its national CCT Donors have good relations some cofinancing (that contribution was or coordinator. program and its subsi- Independent Evaluation Group | World Bank Group with the government and each is, Asian Development relatively small The government dized health insurance other. The capacity of the Bank, European Com- compared with the does some coor- program (PhilHealth) government is mixed. mission, and German entire health bud dination role. 209 What Are the Com- World Bank World Bank Group 210 What are the Complemen- plementarities or Group Coor- Technical Knowledge Main tarities or Synergies between Synergies of World World Bank dination Role Role (a Proxy for Institutional Coordination World Bank Group Institu- Bank Group with Other Group Financing (Proxy for Con- Convening Power as Actors Mechanisms tions and Instruments? Donors? Role vening Power). Technical Authority) government). The ADB get and tends to and World Bank worked be sporadic from in parallel to strengthen a budget point of local government’s role, view. with the World Bank providing demand-side financing (PhilHealth) and the ADB providing supply-side financing (on l project financing to support local providers). Brazil Main actor is Government sets strategy and Also strong coordination be- Minimal. In some cases, 2000s and early The World Bank The World Bank ap- the government. strongly coordinates donors. tween GPs (health, education, donors have contribut- 2010s, World does not play a pear to be recognized World Bank Group Support to Health Services | Appendix F Most previous Prefers not to mix donors on social development, and more ed to the same project Bank was major convening role. as a technical leader by major donors same project because of differ- recently water) at the state (World Bank and IADB health donor. Its interaction stakeholders mention- -- World Bank, ent rules/procedures. level, half-dozen multisectoral on Bolsa Familia), but World Bank/donor with other part- ing different knowledge IADB, WHO/ projects at state/municipal government negotiates financing much ners is minimal. products. Multiple key PAHO, BMGF, level with health component with them separately -- less important in The government knowledge products, USAID, UNFPA, (Ceara, Acre, Rio, Rio Grande prefers to keep separate last several years, coordinates de- including landmark Japan, Italy, do Norte, Bahia, Rio state). different donors with dif- as BR finances the velopment part- studies on noncommu- Germany -- have Between World Bank and IFC: ferent rules/procedures. sector itself. BR ners and keeps nicable diseases (2005, scaled back in There is very, very little coordi- No cofinancing. UNAIDS no longer eligible their activities P078826), hospital last 4–5 years as nation. Even within Bahia, the has technical working for Global Fund in separate, largely performance/efficien- BR has financed hospital PPP has been com- group on HIV, and World 2011; some bilat- to manage cy (2008, P078797/ programs with its pletely separate from several Bank staff tries to attend eral donors phased different rules/ P106246), aging (2011, own resources. World Bank–financed projects. meetings when possible out as well around procedures. P116850), 20-year More emphasis IFC investment officers say but finds them not that time. Some assessment of SUS being placed only that they comment on the particularly useful. World emergency techni- (2013, P120864), on triangular World Bank’s CPS, and that Bank and PAHO interact cal assistance (US cancer care (2015, relationships, the World Bank/IFC did a joint at conferences, work on CDC) for Zika. P133175). Cancer helping BR to roundtable at the Brazil Invest- same issues, but there study was path-break- emerge as donor ment Forum -- this is not are “no incentives ing in use of “narrow” BMGF still a high level of coordination/ to interact.” IADB does cancer issue to under- finances “Grand interaction. .ASA: Excellent not see what the clear stand broader health Challenges sequencing, with World Bank value-added would be system challenges. Brazil” to cata- analytic work setting the stage of working together with These studies have lyze innovative for and supporting multiple World Bank. aligned with and driven research by projects and overall govern- government strategy. actors within the ment strategy (see “World Bank country. Group Technical Knowledge Role”). Republic of Yemen Before crisis the There are some formal coor- Coordination between the There are instances of Although IDA Coordination be- According to inter- main donors dination mechanisms which World Bank & IFC: There was uniqueness of the World assistance for the tween the World views, there is no in the health were in place before the crises a joint World Bank and IFC Bank Group and some period 2003 to Bank Other Do- stand-alone ASA. No sector were: but these were not working project Global Partnership on areas were donors work 2013 that is, before nors: According particular mention UNFPA, UNICEF, as expected. According to Output-Based Aid for Safe with the same health the crisis was to the Interviews by stakeholders. The WHO. The major the WHO Yemen Cooperation Motherhood. IFC designed focus. Initiatives to low - about 7% for Emergen- World Bank has a long bilateral partners Strategy for 2008–2013 (writ- the project and the World work together given the of the total health cy Health and engagement in the are the European ten in 2010), program include Bank validated that the service donors strengths. WHO overseas develop- Nutrition Project, Republic of Yemen’s Commission and the following: (a) The National was provided. Although the and UNICEF remain the ment assistance. the World Bank health sector in provid- Governments of Reproductive Health Steering two private hospitals were two leading agencies After crisis, IDA as- partnerships with ing basic health. The Germany, Italy, Committee and Technical determined to have adequate in the sector having the sistance has been the United Na- World Bank is currently Japan, Nether- group. (b) The Interagency capacity at the hospital level, most extensive field critical. The World tions Children’s addressing the cholera lands, Oman, Coordinating Committee on capacity to deliver services at presence in terms of Bank closed 11 Fund (UNICEF) epidemic in the Repub- Saudi Arabia, immunization meets quar- the clinic level was unclear due planning and direct im- operations in the and the World lic of Yemen. DFID and United terly to plan and monitor to lack of prior experience. This plementation capacities Republic of Yemen Health Organiza- States of Ameri- implementation of planned risk of inadequate staff capacity through their strategic and channeled the tion (WHO) under ca. Gavi, Global immunizations activities as was underestimated, which, partnerships and coordi- funds ($200 million) the ongoing IDA Fund, and Bill well as coordinate all efforts alongside the lack of clear nation with the majority to the health sector Health and Pop- and Melinda and support. It is made up of understanding of the project of donors. World Bank under the Emer- ulation Project Gates Founda- partners supporting immuniza- concept by the two hospitals is canalizing the support gency Health and and the Schisto- tion. After the tion and some national sectors. (including the handling of com- through the two leading Nutrition Project somiasis Project crisis, the main (c) The UN system has finalized plicated obstetric cases), led agencies. Except for P161809 (January have proved to donors were the second United Nations to significant implementation malaria, which is being 17, 2017). be successful. WHO & UNICEF. Development Assistance challenges. The World Bank’s supported entirely by the Independent Evaluation Group | World Bank Group Other donors Framework. However, the involvement was in providing Global Fund the World were the EU, report also identified following independent audits of the Bank, the other partners DfID, and the development assistance service being provided. are all addressing the 211 What Are the Com- World Bank World Bank Group 212 What are the Complemen- plementarities or Group Coor- Technical Knowledge Main tarities or Synergies between Synergies of World World Bank dination Role Role (a Proxy for Institutional Coordination World Bank Group Institu- Bank Group with Other Group Financing (Proxy for Con- Convening Power as Actors Mechanisms tions and Instruments? Donors? Role vening Power). Technical Authority) Islamic Devel- challenges due to the presence The program was started with critical basic health opment Bank of many donors undertak- the idea that (i) the program will needs of the Yemeni (IsDB), Saudi, ing collaboration using their become self-sustainable, (b) population. The World USA, and UAE project approach and creating the government will take over Bank and the other do- governments their own demands. Despite and subsidize it, or (c) find an- nors overlap in providing have pledged the existing opportunities for other partner to replace Global basic health such immu- and/or com- collaboration, many agencies Partnership on Output-Based nization, child and ma- mitted financial still work on their own, thereby Aid. None of these options ternal health. The World support to the creating avenues for overlap materialized and the program Bank is the only donor Yemeni health and duplication. was stopped. financing schistosomia- system. After Coordination between the sis. EU is a Community crisis, the World World Bank Health GP and Health Worker network Bank is in Other GP’s: Before the crisis for building communi- close collabo- there was no coordination ty resilience which is World Bank Group Support to Health Services | Appendix F ration with UN with other sector/GPs. Most complementarity to the agencies—as recently, the Emergency implementation of the is the case with Health and Nutrition Project World Bank–financed UNICEF and will complement the ongoing projects in the health WHO. interventions offered by the sector. World Bank, namely, amely, the Emergency Crisis Response Project on demand- and sup- ply-side nutrition interventions, as well as the ongoing efforts by other international and local partners. This will be achieved through a design that strongly emphasizes the following: (i) addressing development needs of the population, building on the successful preconflict IDA engagement in the health sector and a need to hasten and scale up in response to the exacerbated needs resulting from conflict; (ii) enforcing IDA’s role in partner coordination, monitoring and learning through wel-structured third party monitoring activities that will contribute to plan- ning of the post-conflict and recovery phases; (iii) supporting the health system components represented by the local public institutions (GHOs, DHOs, community and outreach teams) and NGOs involved in the health service provision to preserve functionality and operability of the health system, an approach that has been well tested and used by IDA in Afghanistan and in the Horn of Africa; and (iv) adopting a multisectoral approach with the ongoing IDA interventions. The Health GP is preparing a joint project with Water GP to address cholera epidemic in the Republic of Yemen. Independent Evaluation Group | World Bank Group 213 World Bank Support to Health Services in FCV Countries: Liberia and the Republic of Yemen The World Bank stopped operations in Liberia in 1984 after the country fell into arrears and despite many efforts, it took the World Bank 23 years to fully re-engage. The country descended into a civil war in 1989. In 2003 a Comprehensive Peace Agreement was signed by the major warring factions. Much of the physical health infrastructure and equipment was destroyed, and there was a severe shortage of physicians and nurses. The World Bank co-led the reengagement effort with the United Nations Development Group in 2003. In March 2004, the World Bank approved the Country Re-engagement Note for Liberia, focused on restoring the functionality of the state and rebuilding infrastructure. However, new IDA project financing could still not be approved because of the arrears. The joint IDA/African Development Bank Interim Strategy Note approved in 2007, sets the stage for full normalization of relations with the international financial institutions. In June 2007, the World Ban k approved the Health Systems Reconstruction (P105282) project to: (i) strengthen the policy framework and selected management functions of the Ministry of Health and Social Welfare; and (ii) improve preservice training and selected components of the basic package of health services. In the health sector, Liberia has successfully moved beyond humanitarian relief and has embarked on the rebuilding of its health system (World Bank 2012, 76). In the Republic of Yemen, despite the long cycle of violence the World Bank has for the most part remained engaged in supporting health services. In June 2014, the armed conflict between the government and militias started to spread across much of the country. Prompted by the rapidly deteriorating crisis and the security situation in the field, on March 11, 2015, the World Bank suspended all disbursements under IDA-financed projects and Recipient-Executed Trust Funds. In December 2015, given the critical health needs in the field, the World Bank lifted—on an exceptional basis—the suspension of disbursements for two health projects, the Schistosomiasis Control Project (P113102) and the Health and Population Project (P094755), to allow for an arrangement with two specialized UN agencies for procurement and distribution of essential drugs, medical supplies, and related activities. The two projects are being implemented by the World Health Organization (WHO) and the United Nations Children’s Educational Fund (UNICEF). Local resilience and community-driven mechanisms are useful for short-term assistance to local communities in fragile and conflict-affected situations. Community-based organizations are used in the Republic of Yemen to provide health services in areas where health staff is limited or in areas with large concentrations of internally displaced people (see World Bank 2013). These organizations can open space for line agencies (closed for security reasons) to connect with local communities through subproject support; this has helped establish positive relationships with rebel groups; and enhanced the World Bank’s credibility, trust, and acceptance by state and nonstate actors. On the other hand, in postconflict Liberia, the reliance on institutional channels with weak institutional capacity to deliver health services (for example, limited managerial capacity, insufficient understanding of World Bank procedures, lack of in-house procurement capacity) was identified as a factor in the initial slow pace 214 World Bank Group Support to Health Services | Appendix F of the Health Systems Reconstruction project (only 14.1 percent of the resources were disbursed in the first two years of implementation;(World Bank 2012, 8). References World Bank. 2012. Liberia Country Program Evaluation: 2004–2011. Independent Evaluation Group. Wash- ington, DC: World Bank. ———. 2013. World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States. Indepen- dent Evaluation Group. Washington, DC: World Bank. Independent Evaluation Group | World Bank Group 215 Appendix G. Social Network Analyses Social Network Analysis of Actors in the Liberia Health Sector Background, Objectives, and Scope As part of the overall evaluation, an illustrative case study of the World Bank Group’s positioning in the health sector at the country level was undertaken. Liberia was one the selected country case study. Liberia is characterized by a large number of development partners supporting the health sector and with a government with limited institutional capacity (see appendix E). The purpose of the exercise was twofold: (1) to better understand the relationships among key organizations involved in supporting the health sector in Liberia; (2) to understand how the World Bank Group positions itself in the health sector in Liberia in relation to other organizations. The exercise focused on two key dimensions: knowledge leadership and financial flows. Using social network analysis (SNA), visual network maps were generated that provide insights into how organizations relate to each other with regard to these two dimensions. Methodology Several sources of information were used to identify key organizations in the health sector in Liberia. First, the team used aide memoires of recent World Bank Group missions to Liberia and membership information of the health sector coordination committees and technical working groups, which were provided by the Health Nutrition and Population Global Practice. After compiling a longlist of potential key organizations, the team conducted consultations with experts with in-depth knowledge about the sector such as the World Bank’s task team leader of health projects in Liberia, the World Bank’s Health Specialist based in Monrovia, and the Manager of the World Bank’s Project Management Unit at the Ministry of Health. Sixty-two organizations (4 governmental institutions, 15 multilaterals, 12 bilaterals, and 31 nongovernmental organizations [NGOs] and foundations) were identified through this process. A standardized questionnaire was developed and administered to the selected organizations to collect the necessary data. The team asked two sets of questions. For knowledge leadership: “Which organizations do you consider to be “knowledge leaders” in the Liberian health sector? Which organizations produce the most credible and useful knowledge for your work in health? Which organizations do you turn to when you need technical advice? Which organizations’ publications do you often read to gain new information?” For financial flows: “How much funding did your organization receive from / provide to the listed organizations in the last year (2016, calendar or fiscal year) in $, millions?” To boost response rates, the standardized questionnaires were administered in the framework of a face-to-face meeting with relevant representatives from the selected organizations. The final response rate was 87 percent (54 out of 62). The team used Cytoscape as SNA software (Smoot et al. 2010) to calculate the network metrics and to draw the network maps. For knowledge leadership, Organic layout was used to develop the 216 network map and degree centrality was used as a metric to measure knowledge leadership of the key organizations. For financial flows, the layout was determined by the team purposefully to illustrate the distinct financial flows to the government institutions and to the NGOs. Limitations One of the key limitations of SNA is the construct validity of the data used to measure the variables of interest. In this study, the team tried to achieve a high level of construct validity of findings by collecting data on the basis of a customized standardized questionnaire that was administered to all the selected organizations. Findings about Knowledge Leadership Finding 1: The Ministry of Health and World Health Organization (WHO) are considered to be the two top knowledge leaders, followed by a ‘second layer’ of organizations such as United Stated Agency for International Development (USAID), the United Nations Children’s Fund (UNICEF), and the United Nations Population Fund. The World Bank Group is positioned in the third layer. Knowledge leadership is measured by in-degree centrality, which is the number of incoming connections. The more a specific organization is referred to by others as a knowledge leader, the more incoming connections an organization has in the network map. Figure G.1 shows the knowledge leadership network and table G.1 shows a ranking of organizations on the basis of this variable. Figure G.1. Knowledge Leadership Network in the Health Sector in Liberia Note: Node size represents in-degree. Yellow = government institutions; green = multilaterals; red = bilaterals; blue= nongovernmental organizations and foundations. Independent Evaluation Group | World Bank Group 217 Table G.1. Knowledge Leadership Ranking (Top 10) Organization Sector In-degree Ministry of Health Government 49 WHO Multilateral 38 USAID Bilateral 26 UNICEF Multilateral 22 UNFPA Multilateral 19 The World Bank Group Multilateral 15 International Rescue Com- NGO 13 mittee CDC Bilateral 12 Global Fund to fight AIDS, Multilateral 11 Tuberculosis and Malaria Partners In Health NGO 11 Note: CDC = U.S. Centers for Disease Control and Prevention;NGO = nongovernmental organization; UNFPA = United Nations Population Fund; UNICEF = United Nations Children’s Fund; USAID = United Stated Agency for International Development; WHO = World Health Organization. Finding 2: Distribution of knowledge leadership is uneven. Among the 54 organizations, the highest value of in-degree centrality is 49, whereas the lowest is 0. Looking at the distribution of in-degree centrality, 25 percent of the organizations account for 70 percent of the total in-degree centrality and 50 percent of the organizations represent 90 percent. This means that a relatively low percentage of organizations is considered a knowledge leader in the network, while many others are not considered as knowledge leaders. Figure G.2. Distribution of In-Degree 218 World Bank Group Support to Health Services | Appendix G Figure G.3. Scatterplot of In-Degree and Budget Size Finding 3: Financial volume to some extent plays role in knowledge leadership. As shown in Figure G.3, budget size is positively correlated with in-degree centrality. This suggests that financial volume plays a role in knowledge leadership but not a decisive one (as the position of the World Bank Group, one of the largest funders, positioned in the third layer of knowledge leaders, suggests). Finding 4: The network of multilateral and bilateral organizations is more cohesive than that of NGOs. Cohesion of a network is measured by average degree centrality, which is the number of direct connections an actor has on average in the network. Average degree centrality among multilateral and bilateral organizations is 6.0 while that of NGOs is 3.7 (figures G.4 and G.5). Independent Evaluation Group | World Bank Group 219 Figure G.4. Multilateral and Bilateral Organizations Network Note: Node size represents in-degree. Green = multilaterals; red = bilaterals. Figure G.5. Nongovernmental Organizations Network Note: Node size represents in-degree. 220 World Bank Group Support to Health Services | Appendix G Findings about Financial Flows Finding 1: There are two main distinct financial flows of support to the health sector: one that runs through the government and the other that goes through NGOs. As shown in figure 6, some multilaterals including the World Bank Group, WHO, Global Fund and UNICEF, and a few bilateral organizations such as USAID, Irish Aid and the U.K. Department for International Development provide funding to government institutions. There is also an almost parallel set of flows to implementing NGOs. Finding 2: The World Bank Group is the largest financial contributor to the government. In 2016, The World Bank Group provided $55 million to the health sector in Liberia and $21 million went directly to the government, which accounts for 25 percent of total financial inflows to the government (table G.2). Given the fact that a part of the financial flows from the Ministry of Health to other elements of the Government originates from World Bank Group, this percentage would be even larger in reality (figure G.6). Finding 3: USAID is the largest financial contributor to implementing NGOs. In 2016, USAID provided $87 million to the health sector in Liberia and $74 million went directly to implementing NGOs, which accounts for 62 percent of total financial inflows to the NGOs (table G.2). The numbers differ from what was reported in the Ministry of Health’s Resource Mapping Report because our exercise Figure G.6. Financial Flows in the Health Sector in Liberia Note: Node size represents in-degree. Yellow = government institutions; green = multilaterals; red = bilaterals; blue= nongovernmental organizations and foundations. Independent Evaluation Group | World Bank Group 221 captured the financial flows provided by USAID and related parts of the US government, which were not covered in the Resource Mapping. Table G.2. Top Five Sources of Financial Flows ($, millions) Total Financial Outflows Directly to Government Directly to NGOs Development Partner Amount Development Partner Amount Development Partner Amount USAID/US 87 World Bank Group 21 USAID/US 74 World Bank Group 55 WHO 20 EU 8 WHO 24 Global Fund 11 Global Fund 8 Global Fund 18 USAID 10 World Bank Group 6 DFID 11 UNICEF 8 DFID 5 Total 261 Total 85 Total 119 Note: Total financial outflows include flows between development partners such as multilaterals to multilaterals and bilaterals to multilaterals. DFID = U.K. Department for International Development; EU = European Union; NGO = nongovernmental organizations; UNICEF = United Nations Children’s Fund; USAID = United States Agency for International Development; WHO = World Health Organization. Conclusions SNA is a technique to visualize and analyze a system of actors, in this case all key organizations supporting the health sector in Liberia, on the basis of particular variables of interest. The analysis has generated useful insights into the dimensions of knowledge leadership and financial flows in the health sector in Liberia. For knowledge leadership, it was found that the Ministry of Health, WHO, USAID and a few other multilateral organizations are considered to be the leading organizations. The World Bank Group’s role is less prominent in this regard. For financial flows, two distinct sets of financial flows were observed: one from multilateral and bilateral development partners to the government and the other from multilateral and bilateral development partners to NGOs. The World Bank Group is the largest financial contributor to health services and health infrastructure provided through the government. Social Network Analysis of Webometric Introduction, Methodology, and Data Webometrics, “the study of web-based content with primarily quantitative methods for social science research goals using techniques that are not specific to one field of study,” developed as a new research area in information sciences that has grown out of bibliometric (Thelwall 2009, 6). As the web connects people and organizations, it can host social networks. Therefore, SNA has been used to webometrics. The interconnected web pages can be viewed through SNA as a ‘‘graph’’ in mathematical graph theory where the web domain can be seen as ‘‘nodes’’ and the hyperlinks among pages seen as ‘‘links’’ in the SNA terminology. Previous analysis of webometrics information of development actors operating in the health landscape through SNA is limited. Lang, Gouveia, and Leta (2013a, 2013b) used webometric analyses and techniques, especially interlinks, and SNA map the web presence of the WHO network 222 World Bank Group Support to Health Services | Appendix G health of collaborating centers comprising 190 research institutions from 42 countries. Lang, Gouveia and Leta (2014) identified the web presence of five Brazilian institutions and contributed to understand their role through centrality within the network.1 Coscia, Hausmann and Hidalgo (2013) used webometrics information to study the structure of international aid coordination, creating and mapping a network of donor organizations, recipient countries, and development issues. Hoffman and others (2015) used online network relationships to map the global health system comprising a total of 203 actors. For this evaluation, data was collected by a standard web crawler, implemented in Python using the “scrapy package.”2 The input of the crawler is the list of websites of the international organizations listed in table G.3. The crawler is instructed never to leave the starting domain, so no page outside the selected websites is considered. If a domain has a subdomain (for example, www.worldbank. org contains data.worldbank.org) the subdomain is also crawled. We impose no depth limit, meaning that the domains are fully crawled, without discarding pages that are more than n clicks away from the home page. If the page does not contain any of the specified issues, it is discarded and not used for any of the analysis in this paper. If the page does contain at least one issue, then we store all the keywords – issues, countries, and organizations – mentioned, and all hyperlinks to other websites included in the study. Once the web crawling is complete, the result is a set of pages. A page, p, is defined as a set of entities that co-appear on the page, among which at least one has to be a health issue, otherwise the page is discarded. Every time two entities – say i and j – co-appear in a page of a website w, we increase their co-occurrence counter by one. When we aggregate across all websites w ∈ W, we define their overall “web score” WSij as follows: WSij = X ln(nwij) . w ∈ W We also look at a citation network, connecting organization i to organization j if i cites or links j. The connection strength is still estimated using WSij, replacing nwij with cij. Findings about Citation Network Figure G.7 depicts the organization co-citation network. The first noticeable property of the network is the tendency of organizations to cluster with similar organizations. Table G.3 reports the ten most central organizations in the network, according to betweenness centrality. Betweenness centrality tells us the fraction of paths in the network that would get longer – or disconnected – if the node were to be removed from the network. Independent Evaluation Group | World Bank Group 223 Figure G.7. The Organization Cocitation Network Note: Each node is an international aid organization. The node’s size is relative to its aggregate web score value. The node color reflects the organization type: red = Development Assistance Committee bilateral; blue = United Nations; green = non-DAC bilateral; purple = health initiative; orange = multilateral; yellow = nongovernmental organization; brown = private philanthropy. Edges are sized according to their web space, and colored according to their significance. 10 Central Organizations in the Citation Network and Their Table G.3. Top Incoming Citation Score Rank Organization Betweenness In-WS 1 World Bank 0.193 149.18 2 UNDP 0.115 104.01 3 EU 0.091 147.25 4 FAO 0.065 81.54 5 UNICEF 0.053 126.44 6 ADB 0.039 67.73 7 WFP 0.038 86.91 8 WHO 0.033 146.15 9 IMF 0.027 68.56 10 AfDB 0.026 65.73 Note: ADB = Asian Development Bank; AfDB = African Development Bank; EU = European Union; FAO = Food & Agriculture Organization of the United Nations; IMF = International Monetary Fund; UNDP = United Nations Development Programme; UNICEF = United Nations Children’s Fund; WFP = World Food Program; WHO = World Health Organization; WS = web space. 224 World Bank Group Support to Health Services | Appendix G We estimated the impact of the World Bank in spreading a message through the international aid community borrowing tools from network epidemics (Pastor-Satorras and Vespignani 2001). Susceptible-infected (SI) models have been defined for network data (Karsai et al. 2011). In these models, nodes are assigned to one of two classes: Susceptible if the node can be infected with a disease, and infected if the infection happened. In the simplest model—which we use here—a parameter β is specified: if more than a β fraction of the incoming edge weight of a Susceptible node comes from infected nodes, then the node turns from Susceptible to infected. These models have been successfully used to track the spread of information in social networks (Kwak et al. 2010). Here we assume that a time step zero only the World Bank is “infected” with a message it wants to spread. We run an SI model, recording at each time step the fraction of nodes that are part of the infected pool. Here β represents the share of “infected” cited pages by an organization i necessary to infect i itself. If β = 0.1, then we require that at least 10 percent of the pages cited by i have to come from infected organizations to infect i. If we assume that all influence connections in the network can be used, the World Bank can infect almost the entire network (about 94 percent of the nodes). The leftover nodes have no sufficiently strong incoming connections, and thus cannot be influenced. This is tested across a variety of β values. Figure G.8, panel a, depicts the result of these simulations. The higher the β parameter the harder it is for an infection to spread. However, in the case of the World Bank this only affects the speed of propagation of the information (from three to seven steps), not the final coverage. This means that, according to this model, if the World Bank sends a message to the international aid community, likely 94 percent of organizations will receive it eventually, assuming that the real unobservable epidemics parameter β is equal to or lower than 0.15. However, not all messages are equal. The content of the message likely influences its chances to be passed or not. We can simulate also this case, by creating a multilayer view of the influence network, where each layer only contains citations made from pages containing only a specific keyword. We can run the SI model using exclusively edges coming from a single layer, which will now inform us about the power of the World Bank to influence organizations exclusively about a specific issue. Figure G.8., panel b, depicts the result of these simulations. We also report the result of the simulation using all the layers, for reference. We can see that there are significant differences between issues. The World Bank can reach most nodes in some cases, for instance when talking about public health (the final infected share of nodes is about 88 percent). However, in the case of avian influenza the message finds a bottleneck in the multilayer network, and only reaches a third of the network. Independent Evaluation Group | World Bank Group 225 Infection Dynamics in the Organization-Organization Figure G.8. Different Citation Network a. Infection dynamics considering all edges for different b. Infection dynamics for β = 0.1 for different keywords values of the infection parameter β Note: The lines end when no further infections are recorded in the model. Panel b keywords: purple = overall; green = public health; and blue = avian influenza. How does the World Bank compare to other organizations? Here we choose three comparisons: WHO, UNICEF, and USAID. We run the SI model using all connections from all keywords, and fixing β once again to 0.1. Figure G.9, panel a, depicts the result. We can see that the World Bank is noticeably slower than WHO, which reaches saturation faster. However, the World Bank outperforms UNICEF. When considering all keywords at the same time, the messages coming from USAID are dwarfed and reach a negligible portion of the network. However, as pointed out before, these diffusion patterns are highly dependent on which keyword we are focusing. Figure G.9, panel b, depicts the information spreading results when focusing on a specific one: “Nurse.” In this case, the four organizations are hardly distinguishable, with USAID having an influence potential on par with the World Bank and WHO. Infection Dynamics in the Organization-Organization Figure G.9. Different Citation Network a. Infection dynamics for β = 0.1 for different organiza- b. Same as panel a, but only considering connections tions, aggregated to all keywords. for the topic “Nurse.” Note: The lines end when no further infections are recorded in the model. 226 World Bank Group Support to Health Services | Appendix G Limitations Proper names and acronyms do not always uniquely identify an organization, as some of them are also words can be used, without necessarily referring to the organization that is searched. To avoid the potential biased a procedure was used to exclude organizations using the citation information. Each organization can be cited in two ways: (i) in the text of a page and (ii) by a hyperlink. While the former citation is affected by this problem, the latter is not. Therefore, if an organization is disproportionally cited through mechanism (i), then the organization is likely to introduce noise in the networks and has been excluded from the graph. References Coscia, M., R. Hausmann, and C. Hidalgo. 2012. “The Structure and Dynamics of International Develop- ment Assistance.” Journal of Globalization and Development 3 (2): 1–42. Hoffman, Steven J., Clarke B. Cole, and Mark Pearcey. 2015. Mapping Global Health Architecture to Inform the Future. Centre on Global Health Security. London: Chatham House (Royal Institute of Interna- tional Affairs, U.K). Karsai, M., M. Kivelä, R. K. Pan, K. Kaski, J. Kertész, A. L. Barabási, and J. Saramäki. 2011. “Small but Slow World: How Network Topology and Burstiness Slow Down Spreading.” Physical Review E 83 (2): 025102. Kwak, H., C. Lee, H. Park, and S. Moon. 2010. “What Is Twitter: A Social Network or a News Media?” In Proceedings of the 19th International Conference on World Wide Web, edited by M. Rappa, P. Jones, J. Freire, and S. Chakrabarti, 591–600. New York: Association for Computing Machinery. Lang, P. B., F. C. Gouveia, and J. Leta. 2013a. « Cooperation in Health: “Mapping Collaborative Networks on the Web.” PLOS One 8 (8): e71415. Lang, P. B., F. C. Gouveia, and J. Leta. 2013b. “Cooperation in Health: A Cluster Analysis of 190 Research Institutions.” Journal of Scientometric Research 2 (3): 223–230. Lang, P. B., F. C. Gouveia, and J. Leta. 2014. “Health Research Networks on the Web: An Analysis of the Brazilian Presence.” Cadernos de Saúde Pública 30 (2): 369–378. Pastor-Satorras, R., and A. Vespignani. 2001. “Epidemic Spreading in Scale-Free Networks.” Physical Review Letters 86 (14): 3200. Smoot, M. E., K. Ono, J. Ruscheinski, P. L. Wang, and T. Ideker. 2010. “Cytoscape 2.8: New Features for Data Integration and Network Visualization.” Bioinformatics 27 (3): 431–432. Thelwall, M. 2009. Introduction to Webometrics: Quantitative Web Research for the Social Sciences. Willis- ton, VT: Morgan & Claypool. Independent Evaluation Group | World Bank Group 227 1 Oswaldo Cruz Foundation (FIOCRUZ-BR), Jorge Duprat Figueiredo Foundation for Work Safety and Medicine (FUNDACENTRO-BR), National Cancer Institute (INCA-BR), Nucleus for Studies on Violence, University of São Paulo (NEV.USP-BR), and Institute of Social Medicine, State University of Rio de Janeiro (IMS−BR). 2 https://scrapy.org/ 228 World Bank Group Support to Health Services | Appendix G Appendix H. 2014 Reform Effort Survey Description of the Methodology The 2014 Reform Efforts Survey, was fielded by the College of William and Mary’s Institute for the Theory and Practice of International Relations in partnership with the National Opinion Research Center at the University of Chicago in the summer of 2014 (Custer et al. 2014).1 This first-of-its-kind survey was explicitly designed to provide timely, detailed, and accurate data on the trustworthiness, influence, and performance of 100+ Western and non-Western development partners, as observed and experienced by the in-country counterparts of development partners. A total of 6,731 development policymakers and practitioners from 126 low- and middle-income countries participated in the survey. Figure H.1 shows the number of survey participants by their area of policy expertise. Those individuals with specialization in health accounted for the largest proportion of survey respondents (6.1 percent, N = 356).2 Of these health sector respondents, 199 indicated that they had worked directly with the World Bank, which constitutes the subsample of interest that we will rely on for much of our analysis. Survey Use in the Evaluation The 2014 Reform Efforts Survey, asked to evaluate each of the development partner organizations with which they interacted directly in terms of their performance at different stages of the reform process (for example, agenda-setting, implementation). More specifically, we asked them to indicate: 1) how frequently they found policy advice provided by development partners to be useful (question 14); 2) to what extent those development partners exerted influence on the country’s decision to undertake reforms within their policy domain of expertise (question 21); and lastly, 3) how helpful they were in implementing reform efforts (again within the domain of expertise) (question 25). Of our interest is to identify in which policy areas survey respondents found the World Bank to have provided particularly useful policy advice and/or proved most effective in shaping reform priorities and implementing reforms. To this end, we will produce three different scores based on questions 14, 21, and 25, each of which captures a specific facet of the World Bank’s performance in the health sector reform process: the usefulness of policy advice, agenda-setting influence, and helpfulness in reform implementation. Figures H.2–H.4 show how World Bank’s performance scores vary across different policy domains. In so doing, we identify the specific policy areas in which the World Bank was perceived to have performed well compared with other sectors. The figures below also compare World Bank’s performance scores compared with the average of the other development partners, to examine whether the World Bank’s scores in each policy sector are statistically different from the sector- specific average scores of other major development partner organizations (for example, the United States Agency for International Development, the African Development Bank, and so on). 229 Figure H.1. Number of Survey Participants, by Area of Policy Expertise Source: Custer et al. 2015. Figure H.2. Usefulness of Policy Advice: World Bank versus Other Donors Source: Custer et al. 2015. 230 World Bank Group Support to Health Services | Appendix H Figure H.3. Policy Influence: World Bank versus Other Donors Source: Custer et al. 2015. Figure H.4. Helpfulness in Reform Implementation: World Bank versus Other Donors Source: Custer et al. 2015, Independent Evaluation Group | World Bank Group 231 References Custer, Samantha, Zachary Rice, Takaaki Masaki, Rebecca Latourell, and Bradley Parks. 2015. Listening to Leaders: Which Development Partners Do They Prefer and Why? Williamsburg, VA: AidData. 1 The questionnaire questions are available in the online a of Custer et al. (2015). 2 Individuals who did not have any policy focus are excluded from this figure. 232 World Bank Group Support to Health Services | Appendix H The World Bank 1818 H Street NW Washington, DC 20433