Approach Paper An Evaluation of World Bank Group Support to Health Services September 28, 2016 Background and Context 1. Health services (HS) are crucial for development. HS include all services dealing with the diagnosis and treatment of disease, or the promotion, maintenance and restoration of health.1 They include personal and non-personal health services. Ending extreme poverty and promoting shared prosperity sustainably require, among others, access to social services, including HS. The World Bank Group (WBG)2 works with the public and private sectors, and development partners to improve HS in client countries through finance, knowledge and convening services (World Bank, 2013). DESCRIPTION OF CONTEXT AND ISSUES 2. The global coalition for universal health coverage (UHC) urges government and development partners to accelerate progresses toward the goal that all people receive the quality HS they need, without suffering financial hardship.3 UHC is one of the targets of the Sustainable Development Goals (SDG) of ensuring healthy lives and promoting well-being for all (SDG3).4 Access to HS is closely linked to the Millennium Development Goals (MDGs) and SDG targets of reducing preventable maternal and child mortality; reducing stunting and improving nutrition for infants and children; strengthening health systems; and preventing and treating communicable (e.g. AIDS, Tuberculosis, malaria) and non-communicable diseases (NCDs). The WBG is reporting on its contribution to improved access to essential health, nutrition and population services in its corporate scorecard (World Bank, 2016). 3. The UHC concept provides elements to assess improvement in HS coverage. HS should be available to all people who need them, establishing equity as a central tenet of UHC. Barriers to access take a variety of forms such as distance to the nearest health facility, or overcrowded facilities that impose long waiting times, lack of information on available services, lack of confidence in facilities and staff, and sociocultural barriers including constraints related to gender or age, social norms, beliefs and preferences. In addition, HS should be provided at a level of quality necessary to obtain the desired effect and potential health gains. Finally, the cost of 1 Source http://www.who.int/topics/health_services/en/ 2 The WBG comprises the International Bank for Reconstruction and Development (IBRD), the International Development Association (IDA), the International Finance Corporation (IFC), the Multilateral Investment Guarantee Agency (MIGA) and the International Centre for the Settlement of Investment Disputes (ICSID). 3 See Universal Health Coverage Day http://universalhealthcoverageday.org/partners/ 4 Target 3.8. is “achieve universal health coverage, including financial risk protection, access to quality essential health care services, and access to safe, effective, quality, and affordable essential medicines and vaccines for all.” 1 health services, especially where direct out-of-pocket (OOP) payment is involved, should not push households into, or further into, poverty (Boema et al., 2014; WHO and the World Bank, 2015). 4. Today, more people have access to HS than at any other time in history. The UHC monitoring framework produced by WHO and the World Bank (2015) proposes a set of core, +tracer indicators that are suitable for tracking access to quality HS. Although the ultimate goal of UHC is 100% coverage, the UHC monitoring framework identifies a core subset that all countries would expect to cover. However, despite this progress at least 400 million people are currently not receiving essential health services for achieving millennium development goals (MDG) targets (Figure 1).5 Figure 1. Regional coverage for MDG health services (most recent year, %) Sources: WDI, February 2016; WHO and World Bank Group, 2015 5. Inadequate resources and institutional capacity, poor governance, demographic and epidemiological challenges, global pandemics, failures in health markets and sub-optimal behaviors are among the factors contributing to inadequate use of quality HS among the poor. On the other hand, opportunities are arising. Innovations in health policies, systems, products and technologies, and services and delivery methods improve people’s health and wellbeing.6 Better understanding of human behaviors and how individuals and groups respond to communication, incentives and information, social factors and activities, and psychological factors can improve health results (Flanagan and Tanner, 2016). 6. With almost half of the world’s poor expected to live in fragility and conflicts states (FCS) by 2030, the provision of HS in this context is a priority for the WBG. Reconstruction, rehabilitation and development initiatives start increasingly earlier after crisis, creating potential 5 See Cotlear et al. 2015 for a review of different approaches toward UHC. 6 Source: WHO Health topics – Innovations. http://www.who.int/topics/innovation/en/ accessed on July 8, 2016 2 tensions between development and humanitarian aims (Philips and Derderian, 2015). It is often argued that the provision of HS can contribute to state building and legitimacy even if the degree to which it does so remains uncertain. Limited evidence is also available on how to best deliver HS in poorly governed states and how to meet short-term health needs while developing a coherent system (Jonnalagadda and Rubenstein, 2012). 7. The global health landscape has changed significantly over the past few decades. This evolution has been characterized by greater funding, increasingly complex health challenges, and new – and more diverse – actors operating within the system (Koplan et al., 2009). Development assistance for health (DAH) has increased from US$6 billion in 1990 to almost US$35 billion in 2015.7 Most of this increase has come from official donor country aid. But new sources of global health financing, in particular the Bill and Melinda Gates Foundation (BMGF), have increased significantly. Private funding now accounts for about a quarter of all development aid for health (see Figure 2). The rise in number and scope of global health initiatives makes global health assistance complex and intertwined and could also result in increased transaction costs and fragmented health provision. This creates the demand for coordinated global and country-level interventions. Figure 2. Financing global health 8. A major feature of the health sector is the coexistence of both public and private sectors in the financing and provision of HS. Economic theory indicates three distinct rationales for public intervention in health care markets: (i) to ensure the optimal provision of public goods or services with large externalities (efficiency concern); (ii) to subsidize consumers too poor to buy health insurance or HS out-of-pocket (equity concern); and (iii) to correct failures in the health insurance market (both efficiency and equity concerns) (Musgrove, 1996). The private health sector is a major provider of HS in low and middle income countries (Millis et 7 Source: Institute for Health Metrics and Evaluation Released December 2014 - 2015 University of Washington http://vizhub.healthdata.org/fgh/ accessed on July 8, 2016 3 al, 2002) with square potential to develop inclusive health models that facilitate HS delivery to low-income and to rural populations in developing countries (IFC, 2012). WORLD BANK GROUP POLICIES AND INTERVENTIONS 9. The WBG has been an early supporter of HS working with governments and development partners to ensure all people receive quality, affordable service. The WGB provides a wide range of support to HS. It provides essential financing to HS through IBRD/IDA projects, IFC investment (IFC IS), IFC advisory services (IFC AS) and, potentially, through MIGA guarantees as well as through a number of global/ regional and country level partnerships. Finally, the WBG is also providing a large range of advisory services and analytics (ASA) through different units of the Bank, contributing to Global public knowledge goods as well as local tailor-made solutions. Attachment 3 summarizes the forms of Bank Group support to HS provides details of the portfolio. 10. The World Bank’s policies, strategies, and lending for HS have evolved in phases over the past thirty-five years. During the 1970s, the emphasis was on improving access to family planning services and, to a lesser degree, on nutrition. During the second phase, from 1980-86, the Bank directly financed health infrastructure, with the objective of improving the health of the poor by improving access to low-cost primary health care. During a third “health reform” phase, from 1987-1996, the Bank strived to improve health finance and reform entire health systems. The 1993 World Bank Development report (World Bank, 1993) highlighted the pivotal role of health for development and proposed a three-pronged approach: (i) foster an environment that enables households to improve health; (ii) improve the effectiveness of public health spending; and (iii) promote competition in the delivery of HS. 11. By the late 1990s, the Bank was the largest financier of DAH, and thus very influential in setting priorities in global health. The 1997 Health, Nutrition and Population (HNP) Sector Strategy that would guide the sector for a decade, was issued at the same time that the Bank was reorganized and the Human Development Network (HNP) was formed. It aimed to help client countries: (i) improve the HNP outcomes of the poor and protect the population from the impoverishing effects of illness, malnutrition, and high fertility; (ii) enhance the performance of health systems; and (iii) secure sustainable health financing. 12. IFC created the Health Care Best Practice Group in early 1998 to enhance IFC’s contribution to health investments in developing countries that would benefit particularly the poor.8 In the same year, IFC adopted its frontier country strategy to steer resources towards high-risk and/or low income countries. In 2001 IFC established a dedicated Health and Education Department and the subsequent year presented its health sector strategy.9 The strategy recognized the complementary roles among WBG institutions and clarified the public/private roles in health. The goals for the sector were broadly defined: to improve health outcomes, protect the population from the impoverishing effects of ill health, and enhance performance of health services. 8 IFC. Health Care Best Practice Group, 1999 9 IFC. Investing in Private Health Care: Strategic Directions for IFC, 2002 4 13. The current World Bank Health Strategy embraces many of the same objectives and approaches of the 1997 strategy, while putting greater emphasis on achieving results on the ground (Fair, 2008). It calls for concentrating Bank contributions on its comparative advantages, particularly in health system strengthening, health financing, and economics; for supporting government leadership and international community programs to achieve these results; and for exercising selectivity in engagement with global partners (World Bank, 2007). 14. The 2013 WBG Strategy has important implication for WBG support to HS: work as One Group to strengthen its value proposition toward the twin goals of ending extreme poverty and fostering shared prosperity. The strategy encourages public and private partnerships in order to bring additional resources, experience, and ideas to tackle key development challenges. The strategy also identifies the need to be selective about which activities it takes on, in the context of a right-sized budget, and to consider additional revenue generation measures and sources of financing, such as reimbursable advisory services (RAS) and trust funds to finance knowledge and other non-lending services (World Bank, 2013). The WBG reorganization implemented the following year around global practices (GPs) and cross-cutting solution areas (CCSAs) is intended to help the WBG deliver on its strategy by better connecting global and local expertise within the WBG to better serve its clients. 15. The joint WBG approach to harnessing the private sector focuses on an integrated health system approach that looks for the best solutions, regardless of whether it is public or private. It aims for broader policy reforms and system changes, so that governments can become better stewards of the health systems with the aim to achieve UHC, and recognizes that UHC cannot be achieved without the private sector. It creates a framework for helping WBG clients harness the private sector through “end to end” service offerings (financing and technical assistance), global cross-sectoral expertise and public-private solutions (World Bank Group, 2015). 16. The mission of the health, nutrition and population global practice is to assist countries to accelerate progress towards UHC. The HNPGP priority directions update (HNPGP, 2016) also indicates how the new organizational model structured around regional and functional Practice Managers and seven global solution leads (i.e. Financing, Service Delivery, Population & Development, Nutrition, Health Societies/Public Health, Decision & Delivery Science, Private Sector Engagement) would enhance the lending, knowledge, and convening functions (see Attachment 9 Evolution of the World Bank’s Engagement in Health, Nutrition and Population). PREVIOUS EVALUATIONS 17. This evaluation builds on previous IEG work. Relevant IEG evaluations and recommendations are summarized in Attachment 8. The 2009 evaluation of WBG support in HNP found that while key health outcomes such as infant survival and nutritional stunting have improved over the decade in every developing region, nearly three-quarters of developing countries are either off track or seriously off track for achieving the MDG for reducing under- five mortality. With regards to IFC’s support to the private health sector, the same evaluation found that while the performance of IFC’s health investments, mostly hospitals, has substantially improved, IFC’s health interventions have had limited social impact, although efforts to broaden those impacts are increasing. The Social Safety Nets (SSN) evaluation found that Bank support 5 has largely accomplished its stated short-term objectives and helped countries achieve immediate impacts. But to achieve the longer-term goal of developing country SSNs, short-term objectives need to be better defined, effectively monitored, and anchored in a longer term results framework. Weaknesses in poverty data, program designs, and monitoring indicators need to be addressed to ensure target groups are adequately reached. 18. Past evaluation have shown that WBG support to HS is usually highly relevant, but not always able to achieve its potential. WBG self-evaluations (e.g. implementation completion and results reports (ICRs) of IBRD/IDA projects) and IEG relevant evaluations (see Attachment 8) have already identified a number of lessons that could enhance the achievements of project development objectives, thus accelerating countries toward UHC and, in turn, contribute to attaining the WBG’s twin goals, MDGs and SDGs. Purpose, Objectives and Audience PURPOSE AND OBJECTIVES 19. The purpose of this evaluation is to collect evidence, develop lessons, and propose recommendations that could enhance WBG support to client countries as they move toward UHC. To ensure that the evaluation has a manageable scope, the analysis will focus on those activities that support directly the provision and the demand for HS. Therefore, WBG support that affects HS indirectly (e.g. through improved income, education or the environment) will not be part of this evaluation. 20. The evaluation will cover both learning and accountability aspects of WBG support. With respect to accountability, the study will attempt to determine in what ways and to what extent WBG support to HS has achieved its stated objective, and the extent to which these were aligned with Bank, country, and sector strategies. The learning aspect of the study will focus on drawing lessons from factors associated with successful and unsuccessful interventions. The evaluation will also look at relevant lessons from previous evaluations and, to the extent possible, assess their relevance for this evaluation. Therefore, the evaluation will help the WBG to better support countries towards UHC through HS in the future and to better adapt to the changing global health landscape. 21. This evaluation falls under the IEG Strategic Engagement Area (SEA) Sustained Service Delivery for the Poor.10 Over the FY17-19 this SEA will deliver three sector evaluations: urban transport; water supply and sanitation; and HS. The common framework and analytical tools that IEG developed to analyze and evaluate service delivery (Attachment 6) and behavioral change (Attachment 7) will be applied to the three evaluations. IEG will then produce a “chapeau” or synthesis product that draws upon and contrasts findings from the three sectors, as well as other relevant existing IEG evaluative material. STAKEHOLDERS AND AUDIENCE 22. The primary audiences of this IEG evaluation are the WBG’s Boards of Directors, management, and staff. This evaluation will cover WBG support to HS over FY05-16, thus 10 See: IEG. Work Program and Budget (FY17) and Indicative Plan (FY18-19). IEG, dated May 25, 2016. 6 including WBG activities approved and implemented before and after the 2007 health strategy, the 2013 WBG strategy, the joint WBG approach to health and the updated HNP GP strategic direction. By looking at recent performance in this sector, the evaluation will assess the extent to which the Bank is able to support client countries toward UHC with particular emphasis on the poor and the bottom 40 percent. 23. The global nature of the health landscape and the opportunity for additional use of the evaluative evidence produced would expand the relevant audience of this sector evaluation. Additional stakeholders attentive to this evaluation would also be WBG client governments, multilateral developmental banks, development partners, the private sector, concerned civil society organizations, and the ultimate beneficiaries of HS. Finally, the opportunities of combining evaluative evidence generated from the three IEG evaluations under the SEA sustained service delivery for the poor would also make the result of this evaluation relevant to a much broader audience. Evaluation Questions and Coverage/Scope 24. The overarching question of this evaluation is “what has been the role and contribution of the WBG in supporting HS? And, what should be the role and contribution of the WBG in supporting HS considering its comparative advantages?” 25. The evaluation will focus on the role of the WBG support to HS through the lens of (potential) comparative advantages. Our definition of the concept of comparative advantages comprises the following four dimensions:  The interventions, resources, capacities of the WBG in support of HS;  The needs and priorities in the field of HS of individual countries as well as the other development partners;  The effectiveness of the WBG’s interventions in terms of their contribution to relevant HS-related goals, 11  The roles, activities and resources of the WBG in relation to other institutional actors supporting HS at country and global levels. 26. These dimensions are captured in the four specific evaluation questions presented below which, on the basis of careful reflection and delimitation, reflect selected aspects of the OECD-DAC evaluation criteria:12  Question 1: What has been the nature and extent of WBG support to HS in the last ten years? What have been the WBG’s main modalities and instruments? How has WBG support to HS evolved over time, at country and global levels?  Question 2: What have been the main needs and priorities in the field of HS at global and country levels? How have these evolved over time? How has the WBG’s strategy to support HS evolved over time? 11 It is envisaged that HS-related goals will be categorized according to the following dimensions: HS utilization and quality, efficiency and sustainability, equity and gender aspects. 12 The OECD-DAC evaluation criteria are relevance, effectiveness, efficiency, impact and sustainability. 7  Question 3: To what extent and in what ways has WBG support effectively contributed to the achievement of relevant HS-related goals? What can the existing evidence base tell us about the effectiveness of selected specific service delivery modalities and behavior change interventions supported by the WBG? To what extent has the WBG’s support to HS been informed by evidence on effectiveness?  Question 4: To what extent and in what ways does WBG support to HS distinguish itself from support provided by other institutional actors at country and global levels? What has been the role of the WBG in global partnerships supporting HS? What has been the role of the WBG in country-level partnerships supporting HS? What can we learn about the role13 of WBG supporting HS in the global health landscape? SCOPE 27. The portfolio of WBG support to HS subject to this evaluation includes all activities approved between July 1st 2005 and June 30st 2016 (FY05-FY16).14 The evaluation portfolio includes IBDR/IDA projects, 15 IFC investments (IFC IS) and IFC advisory services (IFC AS), World Bank Advisory services and analytics (ASA),16 as well as partnership programs (PP) and multi-donor trust funds (MDTF) approved between WBG FY2005 and 2016. The relevant portfolio was identified using the WBG’s sector and theme complemented by the manual review of the analyst. The time period of evaluation spans important changes such as the surge in and the increased significance of private funding in DAH, the 2007 health strategy and the 2013 WBG strategies and related WBG reorganization. The details of the portfolio identification strategy and of the identified subsets are presented in Attachment 3 (see Table 1). Table 1. World Bank Group portfolio of activities supporting HS, FY05-16 Type of WBG instrument Number of activities Amount (USD, millions) IBRD/IDA projects 520 projects and 81 additional financing 43,402 World Bank ASA 713 (431 TAs, RAS and IE; 282 ESW) 220 IFC investments 162 2,973 IFC advisory services 78 87.7 Partnership programs 20 n/a Source: IEG Evaluation Design and Evaluability Assessment EVALUATION DESIGN, SAMPLING STRATEGY AND DATA REQUIREMENTS 28. The conceptual framework of this evaluation considers WBG’s support to HS at global and country levels recognizing the linkages between the two. WBG global-level support channeled through global partnerships, knowledge and convening services, and country- level support through partnerships, finance, capacity building and knowledge contribute to 13 For example, it could potentially be complementary, unique, catalytic or duplicative. 14 The portfolio presented includes all activities approved during the FY05-16 period with the exclusion of the last two months. The evaluation will cover the entire FY16 and, when relevant, it will also consider activities that were implemented in the FY05-16 period even if they were approved before July 1st 2005. 15 IBDR/IDA projects comprise: investment Project Financing (IPF), development Policy Financing (DPF) and Program-for-Results (PforR). 16 ASAs include: economic and sector work (EWS), impact Evaluation (IE), technical assistance (TA) and reimbursable advisory services (RAS). 8 improve utilization of quality and affordable HS and to more efficient and sustainable health systems. These, in turn will contribute to the achievement of long-term health improvements. However, WBG support at the global and country levels interact. For example, resource mobilized through global partnerships are channeled through country-level projects, and knowledge generated at country-level contribute to global knowledge (see Figure 3).17 Figure 3. Theory of change for WBG portfolio 29. The evaluation design will be structured around the four specific evaluation questions: 30. Question 1: What has been the nature and extent of WBG support to HS in the last ten years? 18 To address this question and underlying sub questions, the evaluation will explore the use of the following methods and data sources:  Portfolio analysis of relevant WBG instruments (see Table 1).19 The portfolio analysis will identify key characteristics of the WBG portfolio, different funding modalities and types of interventions, volumes and evolution over time, by regions and country-types.  The analysis of the WBG portfolio will also be used to extract relevant equity and gender elements, as well as capture categories of service modalities and behavior change interventions. 17 Attachment 10 depicts how country health systems functions translate into goals and outcomes. 18 This question and underlying sub questions relates to the key activities of WBG support to HS presented in Figure 3. 19 Attachment 3 presents the preliminary portfolio analysis. 9 31. Question 2: What have been the main needs and priorities in the field of HS at country and global levels? 20 To address this question and underlying sub questions, the evaluation will explore the use of the following methods and data sources:  Portfolio review of relevant health priorities addressed at a project level.  Literature review of the policy debate on HS at global level and in selected countries.  Structured review of relevant WBG documents such as health sector strategies, country strategies and corporate strategies.  Data extraction from databases such as the Global Burden of Disease21 (GBD) and the World Development Indicators22 (WDI) to identify global and country-level HS priorities and needs.  Overall WBG portfolio trends will be compared with country-level macroeconomic and health level indicators extracted from WDI.  Semi-structured interviews with WBG staff and relevant stakeholders active at global level and in selected countries. 32. Question 3: To what extent and in what ways has WBG support effectively contributed to the achievement of relevant HS-related goals? 23 To address this question and underlying sub questions, the evaluation will explore the use of the following methods and data sources:  Develop overall and intervention-specific theories of change (TOC).  Portfolio analysis to identify and extract the extent of achievement of results at the project level. 11 In addition, the evaluation will synthesize evidence across the interventions selected for more in-depth analysis around common relevant dimensions (utilization, quality, equity, gender, efficiency and sustainability.  Structured literature reviews: (i) systematic reviews and Gap Maps24 to identify benchmarks and best practices; (ii) literature search protocols of bibliographic databases of academic literature to identify relevant impact evaluations of WBG projects.25  In-depth analysis of selected service delivery and behavior change interventions, global and country-level partnerships.  Descriptive and inferential statistics of WBG portfolio and ancillary data (e.g. macroeconomic and health sector indicators extracted from GBD, WDI and other international databases).26 20 In Figure 3 this question relates to the relationships between WBG support to HS and relevant contextual factors 21 Institute of Health Metrics and Evaluation, University of Washington http://www.healthdata.org/gbd 22 http://databank.worldbank.org/data/reports.aspx?source=world-development-indicators 23 This question and underlying sub questions relates to the achievement of the key outputs and outcomes presented in Figure 3. 24 It is not envisaged that additional systematic reviews of the literature would be required. On the other hand, the evaluation will produce additional user-friendly evidence map gaps. See http://www.3ieimpact.org/en/evaluation/evidence-gap-maps/ 25 See for example Kyu et al. (2013). 26 See for examples Denizer et al., (2013) and Raimondo (2016). 10 33. Question 4: To what extent and in what ways does WBG support to HS distinguish itself from support provided by other institutional actors at country and global levels? The evaluation will distinguish between WBG role in global partnerships and at country-level. To address this question and underlying sub questions, the evaluation will explore the use of the following methods and data sources:  IEG Partnership analyses  Institutional mapping of WBG support to HS at global level and in selected country contexts.  In-depth analysis of WBG partnership role and their effectiveness.  Mapping of institutions that support the health sector in selected countries.  Social network analysis (SNA). 34. The evaluation will combine different data collection and analysis methods to answer the proposed evaluation questions. The specific methods and sources of data that will be used in the evaluation can be classified under the following main methodological approaches: Theories of change  The overall TOC presented in Figure 3 will be further refined during the course of the evaluation.  The overall portfolio analysis and consultations with WBG staff will provide inputs to define the set of interventions that will be subject to a deeper dive using the service delivery and behavior change tools. Specific TOCs will be developed for these types of interventions. Portfolio analyses  An overall portfolio analysis performed to identify trends and to categorize objectives, components, and key elements WBG projects, among which: (i) the type of HS supported; (ii) the service delivery modalities and behavior change interventions; (iii) the desired results and their achievements;27 (iv) the lessons learned; and (v) relevant contextual factors (e.g. country level of development, FCS status). Among other things, this information will be used for clustering projects to facilitate comparative analysis. .  The overall portfolio review will also extract from relevant project documents information and data on the relationships between project objectives and WBG country partnership frameworks (CPF) and country assistance strategies (CAS) assistance.  Portfolio analyses on subsets of the overall portfolio will be conducted as a part of in- depth analyses of different intervention types.  The overall and more focused portfolio analyses will cover IBRD/IDA and IFC interventions. ASAs will be analyzed and, to the extent feasible, classified and assessed using a more simplified coding and text analytics tools. Sources of data include PADs, Board Reports, ICRs, ICRRs, XPSRs, PCRs and Evaluation Notes. 27 It is envisaged that results will be categorized according to the following dimensions: HS utilization and quality, efficiency and sustainability, equity and gender aspects. 11 Literature reviews  The literature will be reviewed to identify key HS trends and priorities at global level and in selected countries. This information will be compared with WBG health strategies and other relevant HS policies to understand how WBG support relates with global and country priorities. A review of the effectiveness literature (e.g. including repositories of knowledge from the Cochrane collaboration, Campbell collaboration, 3ie databases), covering mainly systematic reviews and impact evaluations of select health sector interventions (focus on behavior change interventions and service delivery models) will be conducted to create a framework for reference and benchmarking of WBG health sector interventions. The state of the evidence and related gaps will be presented in Evidence Gap Maps28. Secondary data analyses  GBD and WDI data will be used to identify country health needs against which WBG country-level support to HS will be evaluated. The evaluation will also explore relations among WBG portfolio and ancillary data using descriptive and inferential statistics.29 Semi-structured interviews (and surveys)  Semi-structured interviews will be conducted with WBG staff, global stakeholders and WBG country-level partners, as relevant.  Purposive and snowball sampling techniques will be used to cover relevant stakeholder groups relating to the different (in-depth) analyses (see below) included in the evaluation. Criteria driving the sampling include the following: the overall diversity of WBG support in terms of: geographical distribution; country characteristics (e.g. FCS versus non-FCS); institutional diversity within countries; and other issues arising during the evaluation, including practical considerations.  Where needed and feasible (weighing cost against utility considerations), surveys targeting specific respondent groups will be conducted. In-depth analysis  Three different in-depth analyses will be conducted to cover intervention types, global partnerships and country-level partnerships. Each of the three types of in-depth analysis will rely on the methods discussed in previous paragraphs (theories of change, portfolio analyses, literature reviews, secondary data analyses, semi-structured interviews and surveys).  The evaluation will include illustrative in-depth analyses of global and country partnerships in health. In addition, a limited number of intervention types will be selected for in-depth analysis. The selection of intervention types will be informed by the overall portfolio analysis and is guided by the following criteria: frequency of occurrence of particular intervention types in the portfolio, geographical distribution, and the innovative 28 For a definition and examples of EGMs, see http://www.3ieimpact.org/en/evaluation/evidence-gap- maps/ 29 See for examples Denizer et al., (2013) and Raimondo (2016). 12 and/or emerging nature of particular types of intervention. The final selection will be made in consultation with WBG colleagues. In addition, the in-depth analyses will also adopt “equity” and “gender” lenses to assess how and to what extent WBG support to HS has improved equity and gender specific outcomes. Country visits  To support the in-depth analyses of intervention types and country-level partnerships, the evaluation will conduct a number of country missions. The selection of the countries (including the number of countries) will be informed by the sampling processes discussed above and, inter alia, will reflect identified questions or issues that require field-work for further exploration. In addition, for efficiency reasons, the analyses of different interventions (under each type) as well as partnerships will be clustered as much as possible to reduce the number of countries to be covered by the evaluation teams. Social network analysis and institutional mapping  Social network analysis and institutional mapping will be used to analyze WBG role in global and country-level partnerships. SNA is a powerful tool to produce evidence on elements that are not directly measurable, such as knowledge flow and influence. The relationships among actors are visualized in network maps, which enable audience to understand the feature of the network intuitively. SNA applied to webometrics of a subset of HS global partnerships will be used to assess the partners’ relative position (e.g. centrality) and roles, including the WBG (Coscia, 2012; Hoffman et al., 2015). In addition SNA applied to bibliographic data to analyze knowledge creation and diffusion (Carrington, et al., 2005). Institutional mapping is another tool used to identify and represent perceptions of key institutions (formal and informal) and individuals inside and outside a community and their relationships and importance to different social groups. Institutional mapping in selected country levels will be based on review of data and information available from previous evaluations30 and new information from the joint analytical work, joint assessment reviews, joint work, and joint monitoring frameworks, conducted under the SWAPs and semi-structured interviews. SNA and institutional mapping will complement the IEG’s partnership mainstreaming guidance and partnership evaluation tools.31 DESIGN STRENGTHS AND LIMITATIONS 35. The team adopted various strategies to strengthen the evaluation design. First, the evaluation is using a logical approach to address the main questions regarding the role and comparative advantages by looking at four dimensions. Second, the evaluation is strengthening the link between conceptual and methodological approaches by developing specific TOC to 30 See Vaillancourt (2009; 2012) and Vaillancourt and Pokhrel (2012). 31 The evaluation is exploring the possibility of using the 2014 AidData Reform Efforts Survey that tracked a large portion of DAH and collected information and opinion from representative sample of development partners and recipient governments on their use and impact. See Custer et al., (2015) and http://aiddata.org/ 13 guide the specific methodological framework. Third, the evaluation is adopting explicit strategies to maximize depth and breadth of the evaluable material in a cost-effective manner. This principle has led to prefer the use of methods, such as SNA applied to open data (e.g. webometrics and bibliographic databases), the use of desk-based in-depth analysis and the use of existing data sources such as WDI, GBD and the AidData survey. 36. However, the variety of interventions, country contexts and institutional landscapes covered by the evaluation pose challenges. The evaluation team will have to be selective in analyzing a limited number of interventions, capturing some variety of modalities, countries and institutional contexts. A particular challenge is the limited capacity of the monitoring and evaluation systems in FCS, as well as potential security limitations imposed on the IEG evaluation team to collect additional information. To overcome the limitations and challenges, the evaluation team will first do a desk review of the relevant portfolio to better identify the issues that require more in-depth exploration. Based on this, and in consultation with the IEG FCS community of practice and the WBG FCS CCSA, the team will identify the countries and specific projects that are more likely to generate quality information and data. The TOCs provide a simplified and intervention specific framework of (intended) causal change. While such frameworks can be very helpful to support data collection and (causal) analysis they are also intrinsically biased. Consequently, the evaluation will explore the use of system perspectives that model WBG support, such as SNA. Quality Assurance Process 37. The evaluation will be subject to various quality controls. First, the Approach Paper would go through IEG’s management and external peer reviewers control to ensure relevance of evaluation questions and issues covered, adequacy of scope of the evaluation and appropriateness of methodology. External peer reviewers are Mead Over, Senior Fellow at the Center for Global Development and former Lead Health Economist in the Development Research Group of the WBG; Leslie Faye Stone, Lead Economics Specialist at the Office of Evaluation & Oversight of the Inter-American Development Bank; Pedro Pita Barros, Professor of Economics at Universidade Nova de Lisboa where he teaches industrial organization and health economics; and William Savedoff Senior Fellow at the Center for Global Development where he works on issues of aid effectiveness and health policy. The methodologies of the evaluation will be further developed with the support of IEG Method Advisor. EXPECTED OUTPUTS, OUTREACH AND TRACKING 38. Planned Reporting Vehicle. The primary output of the evaluation will be the report to the Board’s Committee on Development Effectiveness (CODE), which will contain the main findings and recommendations (see Attachment 9). The finished evaluation will be published and disseminated both internally and externally. IEG will develop additional dissemination products, such as working papers, presentations, blogs, and videos, as appropriate to enhance the dissemination of the key findings. Finally, the findings will contribute to a chapeau product distilling and contrasting the lessons learnt related to service delivery and behavior change across the three service-sectors evaluated in the SEA. 14 39. Regular stakeholder interaction will be sought to enhance the evaluation process. This will include consultation while the evaluation is under way and dissemination and outreach once the study is complete. The use of PROACT and REACT workshops and a virtual collaboration space to share the portfolio will be considered. During evaluation preparation, the team will solicit feedback and comments from stakeholders, in particular WBG management and staff, health practitioners in global and government agencies in client countries, to improve the evaluation’s accuracy and relevance. Such stakeholder interaction will contribute important information and qualitative data to supplement data, interviews, in-depth analysis, and other research. Social media will be used to reach out to the broader development community and concerned stakeholders, potentially including beneficiaries of HS. Consultations will also be held during field missions with stakeholders including government counterparts, bank staff, NGOs and other donors, private sector and beneficiaries. 40. Outreach strategy. In addition to outreach during the evaluation process, IEG will implement an outreach plan once the evaluation is completed. IEG will launch the report both in Washington, DC, and at a major international conference. The efforts will target key stakeholders, including staff at headquarters and country offices, other multilateral development banks and donors, government authorities, civil society organizations, and counterpart officials. Through these means and relevant international fora, the team will seek to maximize awareness and the value and use of findings and recommendations to strengthen development outcomes. A more detailed plan will be developed closer to completion, once the type of messages emerging is clearer. Resources 41. Timeline and budget. The evaluation will be submitted to CODE by the end of Q1 FY18. The budget for the study is estimated at $899,395 (see Attachment 4 for details). 42. Team and Skills Mix. The skills mix required to complete this evaluation includes expertise in health, evaluation experience and knowledge of IEG methods, including SNA, descriptive and inferential statistical, and portfolio analysis; familiarity with the policies, procedures and operations of IFC, MIGA, and the World Bank; and knowledge of relevant development partners activities. The evaluation will be led by Antonio Giuffrida, Lead Evaluation Officer, TTL with Hiroyuki Hatashima, Senior Evaluation Officer co-TTL until Approach Paper approval and Maria Elena Pinglo, Evaluation Officer, co-TTL after Approach Paper approval. The current task team comprises Aline Dukuze, Anna Aghumian, Ann Flanagan, Ayse Boybeyi, Catherine Seya, Denise Vaillancourt, Disha Zaidi, Eduardo Maldonado, Gisela Garcia, Jeffery Tanner, Katsumasa Hamaguchi, Mercedes Vellez and Susan Caceres. . The report will be prepared under the direction of Marie Gaarder, Manager, IEGHC; and Nicholas David York, Director, IEGHE. 15 Attachment 1. References Abbasia A, Jörn Altmannb, Liaquat Hossaina. Identifying the effects of co-authorship networks on the performance of scholars: A correlation and regression analysis of performance measures and social network analysis measures. Journal of Informetrics, Vol. 5(4):594–607, 2011 Andersen RM. Revisiting the Behavioral Model and Access to Medical Care: Does it Matter? Journal of Health and Social Behavior, Vol. (36)1:1-10, 1995 Bamberger, Michael, Jos Vaessen, and Estelle Raimondo, eds. Dealing with complexity in development evaluation: a practical approach. Sage Publications, 2015 Boerma T, Eozenou P, Evans D, Evans T, Kieny M-P, Wagstaff A. Monitoring Progress towards Universal Health Coverage at Country and Global Levels. PLoS Med 11(9): e1001731. doi:10.1371/journal.pmed.1001731, 2014 Coscia, M., Hausmann, R. and Hidalgo, C., The Structure and Dynamics of International Development Assistance. Journal of Globalization and Development, Vol. 3(2):1-42, 2012 Cotlear Daniel, Somil Nagpal, Owen Smith, Ajay Tandon, and Rafael Cortez. Going Universal: How 24 Developing Countries Are Implementing Universal Health Coverage Reforms from the Bottom Up. Washington DC: The World Bank Group, 2015 Caceres Susan A., Robert Yin, Anthony Tyrrell, Judy Hahn Gaubatz, and Sian Williams. Introducing a Framework for Evaluating Services Delivery in Sector Evaluations (Urban Transport, Water and Sanitation, and Nutrition). IEG Working Paper. Washington DC: The World Bank Group, 2016 Carrington, Peter J., John Scott, and Stanley Wasserman, eds. Models and methods in social network analysis. Cambridge University Press, 2005 Custer, Samantha, Zachary Rice, Takaaki Masaki, Rebecca Latourell and Bradley Parks. 2015. Listening to Leaders: Which Development Partners Do They Prefer and Why? Williamsburg, VA: AidData, 2015 Denizer C, Kaufmann D, Kraay A. Good countries or good projects? Macro and micro correlates of World Bank project performance. Journal of Development Economics Vol. 105:288–302, 2013 Fair M. From Population Lending to HNP Results: The Evolution of the World Bank’s Strategies in Health, Nutrition and Population. IEG Working Paper 2008/3. Washington DC: The World Bank Group, 2008 Flanagan Ann Elizabeth and Jeffery C. Tanner. A Framework for Evaluating Behavior Change in International Development Operations. IEG Working Paper. Washington DC: The World Bank Group, 2016 Gwatkin D, Wagstaff A, Yazbeck A, editors. Reaching the Poor with Health, Nutrition and Population Services: What Works, What Doesn't, and Why. Washington DC: World Bank, 2005 HNPGP. Updates to the Priority Directions for the Health, Nutrition and Population Global Practice 2016-2020. Washington DC: The World Bank Group, June 2016 Hoffman Steven J., Clarke B. Cole, Mark Pearcey. Mapping Global Health Architecture to Inform the Future. Centre on Global Health Security, January 2015 IEG. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population: An Evaluation of World Bank Group Support Since 1997. IEG. Washington DC: The World Bank Group, 2009 IEG. Social Safety Nets: An Evaluation of World Bank Group Support Since, 2000–2010. IEG. Washington DC: The World Bank Group, 2011. IEG. World Bank Support to Health Financing – An Independent Evaluation. IEG. Washington DC: The World Bank Group, 2014 IFC. Health Care Best Practice Group. Washington DC: The World Bank Group, 1999 IFC. Investing in Private Health Care: Strategic Directions for IFC. Washington DC: The World Bank Group, 2002 16 IFC. The Role of the Private Sector in Expanding Health Access to the Base of the Pyramid. Washington DC: The World Bank Group, 2012 IEG. Partnership Assessment Mainstreaming and Process Guidance. IEG mimeo. May 16, 2016 Kyu HH, Georgiades K, Shannon HS. et al. Evaluation of the association between long-lasting insecticidal nets mass distribution campaigns and child malaria in Nigeria. Malaria Journal, 12:14, 2013 Koplan Jeffrey P, T Christopher Bond, Michael H Merson, K Srinath Reddy, Mario Henry Rodriguez, Nelson K Sewankambo, Judith N Wasserheit. Towards a common definition of global health. Lancet, Vol. 373:1993– 95, 2009 Kutzin J. Health financing policy: a guide for decision-makers. Health financing policy paper. Copenhagen, WHO Regional Office for Europe, 2008 Jonnalagadda Haar R. Rubenstein LS. Health in Postconflict and Fragile States. United States Institute of Peace. Special Report 301, January 2012 Mills A, Brugha R, Hanson K, McPake B. What can be done about the private health sector in low-income countries? Bulletin of the World Health Organization 2002;80:325-330 Musgrove P. Public and Private Roles in Health: Theory and Financing Patterns. Health, Nutrition and Population Discussion Paper. Washington DC: The World Bank Group, 1996 Philips M, and K. Derderian. Health in the service of state-building in fragile and conflict affected contexts: an additional challenge in the medical-humanitarian environment. Conflict and Health, Vol9:13 DOI 10.1186/s13031-015-0039-4, 2015 Raimondo E. What Difference Does Good Monitoring & Evaluation Make to World Bank Project Performance? Policy Research Working Paper 7726. Washington DC: The World Bank Group, 2016 Vaillancourt D. Do Health Sector-Wide Approaches Achieve Results? Emerging Evidence and Lessons from Six Countries Bangladesh, Ghana, Kyrgyz Republic, Malawi, Nepal, Tanzania. IEG Working Paper 2009/4. Washington DC: The World Bank Group, 2009 Vaillancourt D, and Sudip Pokhrel. Aid Effectiveness in Nepal’s Health Sector: Accomplishments to Date and Measurement Challenges. International Health Partnership (IHP+), February 1, 2012 Vaillancourt D. In Sweet Harmony? A Review of Health and Education Sectorwide Approaches (SWAps) in the South Pacific Phase 1 of a Joint Learning Initiative. ADB, AusAID, NZ-MFAT and The World Bank, May 3, 2012 The World Bank. World Development Report 1993: Investing in Health. Washington DC: The World Bank Group, 1993 The World Bank. Healthy Development—The World Bank Strategy for Health, Nutrition, & Population Results. Washington DC: The World Bank Group, 2007 The World Bank. World Development Report 2004: Making Services Work for Poor People. Washington DC: The World Bank Group, 2004 The World Bank Group. Technical Briefing to the Board: Joint WBG Approach to Harnessing the Private Sector in Health. Washington DC: The World Bank Group, April 24, 2015 The World Bank Group. Corporate Scorecards. Washington DC: The World Bank Group, 2016 The World Health Organization and The World Bank Group. Tracking universal health coverage: first global monitoring report. Geneva: World Health Organization and The World Bank Group, 2015 17 Attachment 2. Detailed Evaluation Design Matrix Data collection Data analysis Evaluation questions Information required Information sources methods methods Limitations Overarching question: “what has been the role and contribution of the WBG in supporting HS? What should be the role and contribution of the WBG in supporting HS considering its comparative advantages?” Dimension 1: The interventions, resources, capacities of the WBG in support of HS What has been the nature and extent Basic data of all WBG of WBG support to HS in the last ten interventions to support years? HS32 approved in the Descriptive statistics of FY05-16 period33 (e.g. portfolio: internal What have been the WBG’s main Data extraction from date of approval, WBG portfolio analysis benchmarking by Descriptive analysis modalities and instruments? WBG portfolio commitments, regions, sub-periods, How has WBG support to HS disbursements, region, WBG instruments evolved over time, at country and PDO, themes and sector global levels? codes) Dimension 2: The needs and priorities in the field of HS of individual countries as well as the other development partners 1. Data extraction 1. Portfolio review 1. Health priorities from portfolio 1. Descriptive statistics What have been the main needs and addressed at a project 2. Reviews of global and of portfolio priorities in the field of HS at global level. 2. Literature review Uniform application of national (selected countries) and country levels? of the policy debate 2. Content analysis coding and text analytics health policy literature 2. Policy debate on HS at on HS at global level to WBG portfolio How have these evolved over time? global level and in selected 3. Review of relevant WBG and in selected 3. Content analysis countries. documents and strategies countries. Matching data from How has the WBG’s strategy to 4. Qualitative analysis, different sources. support HS evolved over time? 3. WBG priorities and 4. Databases (e.g. GBD and 3. Data extraction and descriptive strategies WDI); WBG staff and from documents and statistical analysis. relevant stakeholders active strategies 32 IBRD/IDA projects, World Bank ASA, IFC investments, IFC advisory services, PPs and MDTFs 33 The evaluation will attempt to include also projects active in the FY05-16 period even if they were approved before FY05 and will include also the latest FY16 approvals 18 Data collection Data analysis Evaluation questions Information required Information sources methods methods Limitations 4. Global and country-level at global level and in selected 4. Data extraction HS priorities and needs. countries. from databases (GBD and WDI); Semi-structured interviews Dimension 3: The effectiveness of the WBG’s interventions in terms of their contribution to relevant HS-related goals34 1. Overall and intervention-specific TOCs 2. Achievement of results 1. Literature reviews; semi- To what extent and in what ways has at the project level structured interviews with WBG support effectively contributed 1. Content analysis to the achievement of relevant HS- 3. (i) Benchmarks and best WBG staff and partners 1. Theories of change related goals? practices; (ii) impact 2. Data extraction 2. Portfolio analysis 2. Portfolio analysis Uniform application of evaluations of WBG from portfolio What can the existing evidence base coding and text analytics projects 3. (i) systematic reviews and 3. Structured literature tell us about the effectiveness of 3. Data extraction to WBG portfolio Gap Maps; (ii) literature reviews selected specific service delivery 4. Effectiveness of selected search protocols of from literature Identification strategies modalities and behavior change service delivery and reviews 4. In-depth analysis bibliographic databases of interventions supported by the behavior change Variety of interventions, academic literature 4. In-depth analysis 5. Descriptive and WBG? interventions, global and and country contexts country-level partnerships. 4. In-depth analysis inferential statistics of To what extent has the WBG’s 5. Data extraction ancillary data support to HS been informed by from GBD and WDI 5. Relationships between 5. WBG portfolio, GBD and evidence on effectiveness? WBG portfolio and WDI ancillary data (e.g. macroeconomic and health sector indicators) Dimension 4: The roles, activities and resources of the WBG in relation to other institutional actors supporting HS at country and global levels 34 It is envisaged that HS-related goals will be categorized according to the following dimensions: HS utilization and quality, efficiency and sustainability, equity and gender aspects. 19 Data collection Data analysis Evaluation questions Information required Information sources methods methods Limitations 1. Portfolio analysis; data extraction from external databases (e.g. OECD/DAC, AidData) 2. Data extraction To what extent and in what ways from world-wide- does WBG support to HS 1. WBG Portfolio; data web (WWW); distinguish itself from support sources for DAH: bibliographic provided by other institutional OECD/DAC, AidData databases; semi- 1. IEG Partnership actors at country and global levels? structured interviews analyses; statistical 1. Partnerships data and 2. Webometrics about HS Institutional Difficulties in extracting information; DAH data global development partners; analysis of DAH data data from WWW and What has been the role of the WBG mapping of WBG bibliographic data about support to HS at 2. SNA; institutional AidData in global partnerships supporting 2. Role of WBG and development partners; WBG global level and in mapping; In-depth HS? partners supporting HS at Feasibility and robustness staff and development selected country analysis of partnerships global level of the statistical analysis partners contexts. What has been the role of the WBG 3. Role of WBG and 3. SNA; institutional Quality of responses in in country-level partnerships 3. Webometrics about HS 3. Data extraction mapping; In-depth partners supporting HS in semi-structured supporting HS? global development partners; from world-wide- analysis of partnerships selected country contexts. interviews bibliographic data about web (WWW); What can we learn about the role of development partners; WBG bibliographic WBG supporting HS in the global staff and development databases; semi- health landscape? partners structured interviews Institutional mapping of WBG support to HS at global level and in selected country contexts. 20 Attachment 3. Preliminary Portfolio Review INSTRUMENTS OF WBG SUPPORT TO HS Partnership programs (PP) and multi-donors trust funds (MDTFs): The WBG is engaged in a number of global and country-level partnerships aimed at improving access to affordable and quality HS. While the WBG enters in many different partnerships, the evaluation will look at partnership programs and large MDTFs that operate at global, regional and county level.35 Attachment 5 describes the WBG’s global partnerships most relevant to the objective of this review. IBRD/IDA projects. The WBG provides IBRD loans, IDA credit/grants and guarantee financing to governments to improve affordable access to quality HS through the following instruments:36 - Investment Project Financing (IPF) provides financing to governments for activities, and physical and social infrastructure. - Development Policy Financing (DPF) provides budget support to governments for a program of policy and institutional actions. - Program-for-Results (PforR) that links disbursement of funds directly to the delivery of defined results, helping countries improve the design and implementation of their own development programs. IFC investments (IFC IS): IFC finances projects and companies that improve affordable access to quality HS through the provision of loans and equity investments. IFC investments enable companies to manage risk and broaden their access to foreign and domestic capital markets.37 IFC investments support networks and specialty hospitals, as well as pharmaceuticals and other medical product manufacturers. Either directly through equity, loans and guarantees or indirectly through wholesaling by rolling out funds. MIGA guarantees. MIGA provides political risk insurance (guarantees) for projects in a broad range of sectors in developing member countries, covering all regions of the world. So far MIGA has provided “very few” guarantees to HS projects, thus MIGA operations will not be included in the evaluation. 38 35 Global and Regional Partnership Programs are programmatic partnerships in which a) the partners dedicate resources towards achieving agreed objectives over time, b) conduct activities that are global, regional, or multi-country in scope, 3) partners establish a new organization with shared governance and management unit to deliver these activities. While multi-donor TFs are very similar but they don’t have a governing body and the program manager reports only to his/her line manager, and ultimately to the board of the host organization (IEG, 2016). 36 Source: http://www.worldbank.org/en/projects-operations/products-and-services 37 Source: http://www.ifc.org/wps/wcm/connect/CORP_EXT_Content/IFC_External_Corporate_Site/Solutions/Prod ucts+and+Services 38 Source: https://www.miga.org/investment-guarantees 21 Advisory services and analytics (ASA). The Bank has supported HS through different ASA.39  Technical assistance (TA). TA activities assist clients building capacities or strengthening institutions. There are two TA output types: "event proceeding document" and "advisory services document".40  Economic and sector work (EWS). ESW involves diagnostic and analytical work aiming to influence policy choices and programs. Final outputs are reports shared with the clients which can also inform lending work.  Impact Evaluation (IE). IE establishes the causal link between the change in outcomes and specific policy actions. By measuring cause-effect relationships.  Reimbursable advisory services (RAS). RAS (formerly called Fee-Based Services or Reimbursable Technical Assistance) meet emerging client demand through the provision of customized advisory services. They are a key feature in the Bank’s Knowledge Agenda and of significant importance for the Bank’s engagement with middle income countries (MICs) and high income countries (HICs), including non-borrowing members. IFC advisory services (IFC AS). IFC AS combine IFC’s knowledge, expertise, and tools to unlock investment opportunities in different markets and strengthen the performance and impact of private sector clients across industries. IFC AS support to HS has a strong emphasis on public-private partnerships. The PPP Transaction Advisory Group (C3P) helps to identify, structure, and launch sustainable infrastructure projects, which leverage private sector expertise and capital, and achieve public development objectives. Other IFC AS health initiatives support the improvement of the business environment for the private health sector, enhance competition and creates awareness. 41 IDENTIFICATION OF HEALTH SERVICES PROJECTS ACROSS THE BANK GROUP – FRAMEWORK World Bank Health Services (HS)-relevant projects are classified in three major categories: projects that contribute to the direct provision of HS, projects that focus on behavioral interventions and health system strengthening to improve HS. The large majority of projects classified in the first category use the following lending financing instruments: investment project financing; development policy financing; program-for-results, trust funds and grants. Projects constituting the second category include some of the lending instruments above- mentioned as well as advisory services, and analytics, or ASAs. Projects classified in the first category, which aim to directly impact HS concerns, have as their objectives, to improve the quality of, and access to HS. Projects classified in the second category tend to focus on 1- strengthening the institutional environment of the health sector and improve capacity building at the national level, and 2- demand-side interventions to increase access to HS. All non-lending 39 ESW, TA, IE, TE and RAS codes are used to create new tasks only until July 11, 2016. After this date the new ASA portal will be released, merging and replacing the current product lines (ESW, TA, IE, TE and RAS) into one single product line governed by a single directive/procedure. Source: http://go.worldbank.org/P6CHNWJXH0 40 Source: http://go.worldbank.org/E0ZF9BKFN0 41 These have been mapped to Health, Nutrition and Population and Trade and Competitiveness global practices. 22 operations, or advisory services and analytics, are automatically classified into the second category. Furthermore, all HS-relevant lending projects can be classified into three subgroups based on their Development Objectives (PDOs). The three subgroups are: (i) PDOs that aim to improve access to HS, (ii) PDOs that aim to improve the quality of HS, and (iii) PDOs that aim to strengthen the institutional and regulatory framework. IDENTIFICATION OF HEALTH SERVICES PROJECTS ACROSS THE BANK GROUP – METHODOLOGY IEG’s identification methodology leveraged the Bank Group’s industry coding and system-based flags together with text analytics strategies to systematically capture and categorize the relevant portfolio subsets. In addition to consultations with relevant stakeholders, IEG employed the following steps in order to identify the evaluation’s portfolio of projects: (i) identify relevant system flags (e.g. sector and theme codes), (ii) for projects that do not contain at least one of the relevant system flags, perform a targeted keywords search, and (iii) manually review the projects identified in steps (i) and (ii) as a quality check and to remove false positives and systematically categorize them in order to have a more unified portfolio view. For the World Bank-lending projects, IEG identified 4 sector codes and 9 theme codes as key to the evaluation. The four sector codes are: compulsory health finance, public administration – health, non-compulsory health finance, and health. The nine theme codes are: child health, health system performance, HIV/AIDS, Malaria, other communicable diseases, nutrition and health security, population and reproductive health, non-communicable diseases and injury, and tuberculosis. Projects were selected for review if they contained more than zero percent of at least one of the above theme or sector codes. As a second step, IEG also performed a targeted keyword search of all the preliminary portfolio’s PDOs and Components. This first selection process, step (i), resulted in a list of 1049 projects. After the manual review, 601 World Bank- lending projects, of which 81 were additional financing, were judged relevant the rest were not (Figure 2). For the World Bank non-lending projects, or ASA, the same 4 sector and 9 theme codes were identified as key to the evaluation. Projects were selected for review if they contained more than zero percent of at least one of the above theme or sector codes. Using the first step, the sector and theme codes system flag mentioned hereinabove, IEG identified a list of approximately 751 Technical Assistance (TA) projects and 1031 Economic and Sector Work (ESW) services approved between fiscal years 2005 and 2016. IEG performed a manual review of this preliminary set of 1782 ASAs to eliminate all false positives. 713 ASAs were selected following the manual review of which 282 were ESWs and 431 were TAs. The International Finance Corporation (IFC) also supported HS delivery through investment projects and advisory services. The methodology for the identification of an HS-relevant subset across the IFC investment portfolio differs from the one used for the World Bank lending and non-lending subsets. The selection of IFC HS-relevant investment projects, for both investments and advisory services, are based on IFC’s own classification of “Health” projects, which includes health care, life science (pharmaceuticals) and other services directly linked to the health sector 23 (e.g., medical education, health-related). Some projects in Finance and Insurance are deemed HS-relevant because they contribute to HS delivery. A few investment funds categorically targeted health/pharmaceuticals and are therefore included in IEG’s relevant subset. IEG identified 162 HS-relevant investment projects and 78 advisory services approved between fiscal years 2005 and 2016. These 162 investments totaled slightly above US$ 2.9 billion. The 78 ASAs had a total original commitment of approximately US$ 87.7 million. Table 1: World Bank Group Instruments to Support HS Delivery World Bank Group No. Projects Amounts (US$, millions) Instruments World Bank Lending 520 + 81 Add. Financing 43,402 World Bank ASAs 713 (431 TAs, RAS and IE; 282 ESW) 220 IFC Investments 162 2,973 IFC ASAs 78 87.7 Partnerships 20 n/a Source: IEG DESCRIPTION OF THE IDENTIFIED PORTFOLIO OF HS-RELEVANT SUBSET As mentioned hereinabove, the Bank Group has supported HS through a wide range of relevant instruments, approaches and services over the targeted period FY 2005-2016. The Bank Group’s entire portfolio is expansive both in terms of numbers of projects as well as financial commitments; it spans mainly two out of the three institutions: World Bank and IFC, and multiple sectors. A little over 6500 World Bank lending projects were identified during fiscal years 2005-2016 accounting for over US$476 billion. In this broad portfolio, the targeted HS lending subset account for 601 (16 percent) projects with activities accounting for slightly over US$43 billion. Two major subsets constitute the HS-relevant lending portfolio: projects that contribute to the direct provision of HS, and projects that focus on behavioral interventions and health system strengthening to improve HS. Of the 601 World Bank lending HS-relevant projects, 384 were categorized Direct HS Provision, and 217 Non Direct HS (refer to figure 2). Further, 292 PDOs sought to improve quality, 462 sought to improve access and 298 sought to strengthen the health sector’s institutional environment. (Refer to figure 3). 24 Figure 2: World Bank HS-Relevant HC-Relevant Lending Subset by No. Projects and Commitments Note: 81 of the 601 projects that constitute the Relevant subset were additional financing investments. Note: Commitments do not always reflect the final amount disbursed Source: IEG 25 Figure 3: Sub-classification of World Bank Lending HC-Relevant Projects by PDOs Improve Access Improve Quality Strenghten Institutional Environment Note: The above three categories are non-mutually exclusive. The classification was based Project Development Objectives found in the Project Appraisal or Information Documents. Source: IEG 26 Table 2: World Bank HS Relevant Lending Projects (Summary Table) Number of Projects Amount HS-Relevant WB lending (US$, millions) IBRD IDA OTHER TOTAL IBRD IDA OTHER TOTAL HNP 53 212 120 385 7432 12032 968 20432 Sector Non HNP 54 95 67 216 14803 6363 369 21535 Board Active 30 110 48 188 5879 8707 611 15257 Project Inactive 78 198 137 413 17769 9694 682 28145 Status DPO 29 41 10 71 12215 2208 1 14424 Lending Investment 76 263 150 489 10810 15362 1329 27501 Instrument P4R 3 4 0 7 624 831 0 1455 N/Assigned - - 34 34 - - 22 22 AFR 2 168 70 240 70 10380 402 10852 EAP 10 29 38 77 1967 1465 361 3793 Region ECA 30 40 19 89 9762 849 40 10651 LAC 62 14 11 87 11060 284 16 11360 MNA 4 8 29 41 675 124 204 1003 SAR - 49 12 61 - 5414 304 5718 Other - - 6 6 - - - 26 26 Note: The projects labeled Not Assigned (N/Assigned) were implemented on either Institutional Development Fund, or Highly Indebted Poor Countries Agreements. Source: IEG In recent years, World Bank approvals for lending projects has declined compared to the early years of the evaluation period. 113 projects were approved between fiscal years 2005 and 2006, while only 65 projects were approved in the later period between fiscal years 2015 and 2016 representing a decline of over 40 percent between the early and later periods. In terms of financial commitments, however, the trend is different. The level of commitment was slightly higher in the later period revealing larger investments per project on average. Approximately US$5.2 billion were committed during fiscal years 2005 and 2006, and US$6.2 billion were committed during fiscal years 2015 and 2016. Commitments stretch to a high peak from 2009 to 2011 reaching approximately 8.3 billion in 2010. Note: Several DPLs had health as one of several components. The commitment figures reflect the entire DPL amounts, as opposed to the partial amounts. Therefore, the below commitment figures represent an upper bound estimation. 27 Figure 4: World Bank Lending HS-Relevant Subset Trends by No. Projects and Commitments Source: IEG World Bank non-lending project are Technical Assistance (TA) and Economic and Sector Work (ESW). IEG identified 5857 TAs and 5508 ESW that were delivered between fiscal years 2005- 2016. Using step (i) of the methodology detailed hereinabove, 1782 ASAs (1031 ESW, 751 TA) were identified for manual review. AS a next step, IEG’s will review this subset to rule out false positives and identify a final HS relevant ASA portfolio. Figure 5: World Bank HS-Relevant Non-lending Subset by No. Services and Commitment Note: Of the Relevant subset, 431 were TAs and 282 were ESWs. 28 Note: Of the Relevant subset, US$ 88M were committed to TAs and about US$131 M to ESWs. The amounts in this figure do not include the commitment amounts of several ESWs delivered in FY 16’. Source: IEG The number of HS-relevant ASAs delivered in recent years has noticeably increased from an average of approximately 49 services per year during the earlier period fiscal years 2005 to 2007, to 80 per year during the later period 2015 to 2016 with a peak reaching about 101 services delivered in fiscal year 2016. Fiscal year 2013 recorded the lowest number of ASAs delivered (27 ASAs). On a commitment basis, the total cumulative cost of delivered tasks has been mostly upward between fiscal years 2005 and 2016 except for the sharp recorded drop from US$ 16.3 million in 2012 to $US4.04million in 2013 Figure 6: World Bank ASAs HS Subset Trends by No. Projects and commitments Note: The commitments reflect the sum of ESWs and TAs. Source: IEG 29 162 IFC health investments were identified during fiscal years 2005-2016 accounting for over US$ 2.9 billion. Total original committed amounts increase, on average, between the early and later years of the period (fiscal years 2005 to 2014) and two sharp drops in net commitments are recorded following fiscal years 2012 and after 2015. The total number of projects per year also increases, although not steadily between fiscal years 2005 and 2015. Three sharp drops in number of projects approved are recorded. The first takes place between fiscal years 2008 and 2009 from 18 to 11 respectively; the second takes place soon after fiscal year 2011; and the third is recorded in fiscal year 2015. The 78 HS-relevant ASAs targeted accounted for US$87.7 million. The commitment and number of projects trends are well aligned. The amount of money IFC committed to ASs has increased between fiscal years 2005 and 2016 reaching about US$6.5 million in fiscal year 2016 from US$0.38 million in 2005. Fiscal year 2012 observed the highest recorded total committed funding at approximately US$17.4 million. Figure 7: IFC HS-Relevant Investment Subset by No. Projects and Commitments Source: IEG 30 Figure 8: IFC HS-Relevant Investments by No. Projects and Commitments Note: 13 of the 162 selected investments are not included in this graph because their original commitments as well as the year of their respective approvals were unfound in the database used by IEG Source: IEG Figure 9: IFC HS-Relevant Advisory Services by No. Projects and Commitments Source: IEG 31 The World Bank’s HS-relevant lending subset is mainly concentrated in three Global Practices (Health, Nutrition and Population; Social Protection and Labor and Social, Urban and Rural Resilience) and focuses on improving access to, and the quality of HS. The Non-lending services, however, span mainly across Health, Nutrition and Population, Governance and Social Protection and Labor. Investment lending is the most often utilized lending instrument (83.4 percent). Development policy loans are the second most utilized lending instruments (8 percent) and program for result loans make up a very small proportion of all lending projects (1.4 percent). The lending instruments used for the remaining projects are unspecified. For the non- lending subset, TAs are the most utilized instrument. Figure 8: World Bank HS-Relevant Portfolio by Distributed Global Practice Source: IEG Source: IEG Portfolio Review – preliminary results Note1: Other includes POPs, ODS Note2: Projects may contain more than one intervention, thus the numbers above may be greater than the number of direct pollution projects 32 Attachment 4. Detailed Timeline Attachment 4. Outline of Evaluation Report 1. Executive Summary 2. Introduction: Why HS matter, UHC, SDGs, MDGs and WBG goals. 3. How coverage of quality and affordable HS has evolved - main challenges and opportunities 4. How WBG role in supporting HS has evolved both at global and country-level 5. How relevant and effective is WBG support to HS a. At global level: global partnerships, knowledge and convening services b. At country level: as part of country-level partnerships, as a One WBG c. What can be learned from specific service delivery and behaviors change interventions. 6. Moving forward: Evidence-based lessons to enhance the WBG support to HS 33 Attachment 5. Global Partnership Currently, the World Bank Group is engaged in more than 20 large and medium size partnership programs and initiatives that in one way or another focus of provision of HS and complement the WBG’s work. Most of these programs provide technical assistance, contribute to knowledge generation and sharing, and a few (such as the Global fund and GAVI, and the GFF) also finance country level investments. The Bank plays many roles in these partnerships--- helps mobilize resources through building and managing innovative financing mechanisms (e.g. GFF, IFIM and AMC for –GAVI which helped to secure 30% of GAVI’s funds from 2000 to 2010) acts as a convener in the area (e.g. IHP+). The WBG has been one of the leading financiers of the Ebola response, providing treatment and care, containing and preventing the spread of infections, helping communities cope with the economic impact of the crisis, and improving public health systems. To ensure that the world is better prepared and respond much more quickly to future disease outbreaks, the World Bank Group, the World Health Organization, and other partners, are developing a plan for a new Pandemic Emergency Facility that would enable resources to flow quickly when outbreaks occur. The WBG also has established an Ebola Recovery and Reconstruction Trust Fund to address the urgent and growing economic and social impact of the crisis in the region.42 The Avian and Human Influenza Facility (AHIF) is a Multi-Donor Trust Fund (MDTF) created in 2006 to help developing countries minimize the risk and socioeconomic impact of avian influenza and other zoonoses and of possible human pandemic influenza. The AHIF is led by the European Commission and supported by the governments of Australia, China, Estonia, Iceland, India, Korea, Russian Federation, Slovenia and the United Kingdom. The MDTF is administered by the World Bank. GAVI, The Vaccine Alliance is an international organization created in 2000 that brings together public and private sectors with the shared goal of creating equal access to new and underused vaccines for children living in the world’s poorest countries.43 Global Alliance for Improved Nutrition.44 Global Financing Facility (GFF) is a multi-stakeholder partnership that supports country-led efforts to improve the health of women, children and adolescents by: (i) acting as an innovative financing pathfinder to accelerate the efforts to reach the 2030 goals for women, children’s and adolescents’ health; (ii) financing high impact, evidence- and rights- based interventions to achieve measurable and equitable results; (iii) building inclusive, resilient systems and increasing domestic financing over time to sustain the gains and ensure that all women, children and adolescents have access to essential health care, contributing to universal health coverage; and 42 Source: http://www.worldbank.org/en/topic/health/brief/world-bank-group-ebola-fact-sheet accessed on July 8, 2016. 43 Source: http://www.gavi.org/ accessed on July 8, 2016. 44 http://www.gainhealth.org/ 34 (iv) filling the financing gap by mobilizing additional resources from public and private sources, both domestic and international, and making more efficient use of existing resources.45 The Global Fund to Fight AIDS, Tuberculosis and Malaria is a partnership between governments, civil society, the private sector and people affected by the diseases. The Global Fund raises and invests nearly US$4 billion a year to support programs run by local experts in countries and communities most in need.46 The Global Partnership on Output-Based Aid (GPOBA) is dedicated to making sure the poor and marginalized have access to electricity, water, sanitation, health care, education and other basic services necessary for growth and opportunity. GPOBA funds, designs, demonstrates and documents output-based aid approaches to improve the delivery of basic services in developing countries through a diverse portfolio of projects.47 The Health in Africa Initiative (HiA) was launched by the World Bank Group in 2009 with support from the Bill and Melinda Gates Foundation. The HIA strategy is to better engage the private health sector by mobilizing up to US$1 billion over five years in investment and advisory services to boost socially responsible health care in Sub-Saharan Africa.48 The Health Results Innovation Trust Fund (HRITF) was created in 2007 to support results-based financing (RBF) approaches in the health sector. Through RBF, the HRITF aims to improve maternal and child health around the world. HRITF is supported by the Governments of Norway through NORAD and the United Kingdom through the Department for International Development (DFID). It is administered by The World Bank.49 International AIDS Vaccine Initiative.50 The Joint Learning Network is an innovative, country-driven network of practitioners and policymakers from around the globe who co-develop global knowledge products that help bridge the gap between theory and practice to extend coverage to more than 3 billion people.51 Mectizan Donation Programme. 52 Partnership for Maternal, Newborn and Child Health. 53 45 Source: http://www.worldbank.org/en/topic/health/brief/global-financing-facility-in-support-of-every- woman-every-child accessed on July 3, 2016. 46 Source: http://www.theglobalfund.org/en/overview/ accessed on July 8, 2016. 47 Source: https://www.gpoba.org/ accessed on July 8, 2016. 48 Source: http://www.hanshep.org/resources/links/world-bank-group-health-in-africa-initiative accessed on July 3, 2016. 49 Source: https://www.rbfhealth.org/mission accessed on July 3, 2016. 50 http://www.iavi.org/ 51 Sources: http://www.jointlearningnetwork.org/what-we-do accessed on July 8, 2016. 52 http://www.mectizan.org/ 53 http://www.who.int/pmnch/en/ 35 The Poverty and Social Impact Analysis (PSIA) is a Multi-Donor Trust Fund (MDTF) that was established in 2010 in the WBG. The MDTF has financed more than 240 grants in over 85 countries to assess the distributional and social impacts of policy reforms on the well-being of different groups of the population, particularly on the poor and most vulnerable.54 Power of Nutrition is a partnership of like-minded investors and implementers committed to helping children grow to their full potential, ending the cycle of undernutrition, and enabling countries to build strong and prosperous communities.55 Primary Health Care Performance Initiative (PHCPI) supports countries to strengthen monitoring, tracking and sharing of key performance indicators for primary health care. While many countries have identified primary health care as an urgent priority, they lack the data needed to pinpoint weaknesses, understand their causes and drive improvements.56 Roll Back Malaria.57 The Stop Tuberculosis (TB) partnership was founded in 2001 with the mission to serve every person who is vulnerable to TB and ensure that high-quality diagnosis, treatment and care is available to all who need it.58 The Strategic Impact Evaluation Fund (SIEF) supported researchers are working with health ministries and non-governmental organizations to measure the effectiveness of new and existing initiatives to strengthen access to and quality of care, including better services for women and girls.59 The WB’s Tobacco Control Program is implemented through a multi-donor trust fund financed by the Bill & Melinda Gates Foundation and the Bloomberg Philanthropies. These donors take part in governance of the trust fund and participate in the selection of priority countries included for support under the program.60 54 Sources: http://www.worldbank.org/en/topic/poverty/brief/poverty-and-social-impact-analysis-psia accessed on July 8, 2016. 55 Sources: http://www.powerofnutrition.org/ accessed on July 8, 2016. 56 Source: http://www.worldbank.org/en/news/press-release/2015/09/23/new-partnership-to-help- countries-close-gaps-in-primary-health-care accessed on July 3, 2016. 57 http://www.rollbackmalaria.org/ 58 Sources: http://stoptb.org/ accessed on July 8, 2016. 59 Sources: http://www.worldbank.org/en/programs/sief-trust-fund#3 accessed on July 8, 2016. 60 Sources: http://www.worldbank.org/en/topic/health/brief/tobacco accessed on July 8, 2016. 36 Attachment 6. A Framework for Evaluating Service Delivery in Sector Evaluations61 The Bank has expanded the sectors beyond those initially included within Making Services Work for the Poor (WDR 2004) – health, education, water, sanitation, and electricity access – to include social protection, information, transport, financial services and credit markets. While Making Services Work for the Poor (WDR 2004), advanced the concept of “eight sizes fits all”, this Framework builds upon that foundation established, but contains others found in the literature and supported in WBG projects (i.e. central provision or contracting, decentral provision or contracting, hybrid between central and decentral provision or contracting, private- public contract, private sector, citizen-directed provision such as voucher or community driven development), as well as leaves the possibility of advancing other innovative provisions. The IEG framework and the analytical protocol to evaluate service delivery comprises four elements (see Figure 4):  Enabling conditions refer to the broader government context that enables (or constrains) delivery. The framework contains the assessment of enabling conditions such as political economy analysis, leadership, and development of: budget, policy, regulatory, legal, capacity, or data systems.  Inputs emphasize the preparation of actions specific to the design of service delivery, including the typical system inputs such as human resources, financial resources and technology. Recognizing the heterogeneity of citizen beneficiaries, the IEG protocol would examine the extent to which that project interventions are adapted to population groups and that the monitoring and evaluation (M&E) system collect disaggregated data.  Service delivery implementation refers to the different models of delivering services: centralized versus decentralized provision; private-public interface; citizen, community, citizens and users’ engagement, etc.  Service outputs and outcomes. The framework will evaluate the way outputs are monitored in term of quality and quality control the outputs, including the role of beneficiaries’ feedbacks. 61 Source: Caceres, et al, 2016. 37 Figure 4. Framework for Evaluating Service Delivery 38 Attachment 7. A Framework for Evaluating Behavior Change in International Development Operations62 The behavior change framework builds on economic and psychosocial theories of human behavior. The behavior change protocol categorizes “levers” or “influences” drawn from these theories that can help policy makers design behaviorally conscious interventions (e.g., financial incentives, activating social norms). The protocol allows for an assessment of the prevalence, integration, and effectiveness of behavior change concepts into the life cycle of an intervention from the diagnostic phase through monitoring and evaluation. The behavior change evaluative framework -- CrI2SP -- utilizes five basic categories to characterize both the barriers and interventions to motivate behavior change: communication, resources, incentives and information, social factors and activities, and psychological factors and activities (see Figure 5). Figure 5. The CrI2SP framework of barriers and intervention types that can motivate behavior change: Communication, resources, Information and Incentives, Social Factors, and Psychological Factors Source: IEG Communication serves as the backdrop to interventions intent on changing behavior. This element encompasses how an individual comes to know about what the intervention is and what the desired behavior change is. A lack of information, misinformation, or incomplete or mistrusted information about an intervention can inhibit service take-up. A well-designed communication strategy can reduce informational frictions and encourage particular behaviors. Incentives and Information describe the motivators of a “rational actor”. Incentives may be financial or temporal. They may be positive or negative, as in shortening the time of travel, or providing a subsidy for mass transit use, or a tax on fuel. Information may be complete, incomplete (subject to uncertainty) or asymmetric. Information is translated into two types of 62 Source: Flanagan and Tanner, 2016 39 knowledge. Knowledge that includes information on expected benefits or costs, such as the benefits of washing vegetables, while knowledge how includes information on the skills and techniques needed to accomplish a specific task, often a behavior, such as water purification. Social Factors influence people’s behavior through concerns about how they may be perceived by others. These often include social norms, which are broadly shared beliefs about what group members should and are likely to do. Moral norms, such as appeals to fairness, equity, and responsibility, also figure here. Interventions might engage social norms through a soap opera that destigmatizes public transportation, or encourage a group identity that has socially desirable attributes leading to better outcomes. Psychological Factors account for ways in which individuals’ behavior is influenced by their non-rational or bounded-rational perception of the world around them. This category also includes issues of cognitive stress and hassle. Mental models enter here (including stereotypes, causal narratives or heuristics) as do cognitive biases and limitations. 40 Attachment 8. Relevant IEG sector evaluations Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population (HNP): An Evaluation of World Bank Group Support Since 1997 (IEG, 2009). The evaluation of the HNP support of the World Bank focuses on the effectiveness of policy dialogue, analytic work, and lending at the country level, while that of IFC focuses on the performance of health investments and advisory services before and after its 2002 health strategy. The evaluation identified the following recommendations: (i) Intensify efforts to improve the performance of the World Bank’s support for health, nutrition, and population; (ii) Renew the commitment to health, nutrition, and population outcomes among the poor; (iii) Strengthen the World Bank Group’s ability to help countries to improve the efficiency of health systems; (iv) Enhance the contribution of support from other sectors to health, nutrition, and population outcomes; (v) Implement the results agenda and improve governance by boosting investment in and incentives for evaluation. Social Safety Nets (SSNs): An Evaluation of World Bank Group Support Since, 2000–2010 (IEG, 2011). This evaluation assessed how effective and relevant the Bank has been in its support for SSNs and to draw lessons that can be applied to future support. The key recommendations were: (i) Engage during stable times to build SSNs that can help countries respond effectively to shocks; (ii) Support the development of SSN institutions and systems; (iii) Increase SSN engagement in LICs; (iv) Improve the results frameworks of Bank supported SSN projects; (v) Clearly define objectives and assess benefits, costs, and feasibility of policy alternatives to ensure the most appropriate use of SSNs; and (vi) Improve internal coordination of SSNs. World Bank Support to Health Financing – An Independent Evaluation IEG, 2014. The evaluation examines WBG support to revenue collection for health, pooling health funds and risks, purchasing and factors in successful Bank Group support to health financing reforms. The evaluation develops the following recommendations: (i) Support government commitment and build technical and information capacity to be able to inform health priorities and spending; (ii) Address health financing as a crosscutting issue at the country level; () Have Global Practices focus on health financing as a core comparative advantage of the Bank; Integrate all health financing functions; Strengthen M&E in Bank and IFC projects. World Bank Support to Early Childhood Development (ECD) – An Independent Evaluation IEG, 2015. The evaluation examined the Bank’s design of projects supporting ECD interventions to see if they are consistent with the growing body of research around efforts that are likely to lead to changes in children’s development and improve their readiness for school. Key recommendation include: (i) enhance organizational arrangements for ECD; (ii) adopt the practice of using diagnostics in the preparation of systematic country diagnostics to determine ECD need; (iii) Increase knowledge to address key ECD operational challenges; (iv) improve monitoring and evaluation of ECD interventions. 41 Attachment 9. Evolution of the World Bank’s Engagement in Health, Nutrition and Population Source: HNPGP, June 2016 42 Attachment 10. A model of health system functions and outcomes Source: Adapted from Kutzin 2008 43