Approach Paper Evaluation of the World Bank’s Support to Improving Child Undernutrition and Its Determinants February 11, 2020 Highlights • Global reports on indicators of child undernutrition show mixed progress in reducing the stunting (impaired growth and development) of children under five, with Africa and South Asia most severely affected. • There are many determinants of child undernutrition, which makes the challenge of improving outcomes multidimensional, requiring interventions in areas of health; agriculture; water, sanitation, and hygiene; social protection; education; and governance, depending on the country context. • The objectives of this evaluation are to assess the contribution of the World Bank to improving outcomes related to child undernutrition and its determinants in countries affected by undernutrition, and to provide lessons and recommendations to inform the design of the World Bank’s future multidimensional nutrition support. • The evaluation aims to engage stakeholders within the World Bank to develop relevant learning in child undernutrition. 1. Background and Context 1.1 Child undernutrition negatively affects the health, physical growth, and cognitive development of a child. Undernutrition develops from insufficient intake or absorption of nutrients, which are affected by inadequate feeding, care practices, health services, and water, sanitation, and hygiene (WASH). It is a cumulative condition, starting from the nutrition and health status of the future mother, which affects the growth and development of the fetus and impacts birth outcomes leading to irreversible effects through early childhood and beyond. 1.2 Improving child undernutrition is essential for enhancing human capital accumulation, boosting economic growth, and reducing poverty. The consequences for young children last through adulthood and reduce their potential to learn and contribute to society. These consequences are also often intergenerational, affecting future children. In terms of enhancing the potential of a child to grow and contribute to the economy, the average income penalty per person from stunting (impaired growth and development) is estimated at approximately 7 percent (Galasso and Wagstaff 2018). 1 1.3 Improving child undernutrition requires making improvements at each stage of the life cycle of mother and child (figure 1.1) and addressing the determinants of undernutrition within this life cycle. Malnourished pregnant women may deliver newborns with low birth weight (LBW), and mothers with low body weights or micronutrient deficiencies may struggle to sustain exclusive breastfeeding or feed and care for their babies. Children with low or inadequate nutritional status are more prone to childhood infections, which further aggravate the child’s capacity to absorb nutrients , and to have slower growth and impaired cognitive capacity (Maternal and Child Nutrition Study Group 2013). 1.4 The evaluation of child undernutrition will assess outcomes at different points in the life cycle of mother and child, with a focus on the early years of life. It will assess the contribution to outcomes of undernutrition, including stunting, wasting, underweight, LBW, and micronutrient deficiencies. In addition, the evaluation will assess intermediate outcomes and outputs related to the determinants of undernutrition and results of nutrition interventions in various countries. Within the life cycle, the time when mother and child are most sensitive to the consequences of undernutrition is from preconception, through pregnancy, until the child is approximately two years of age. Children, mothers, and future mothers, including adolescents, are the target of nutrition interventions during this period. Figure 1.1. Nutrition in the Life Cycle of Mother and Child Reduced capacity to care for baby Household caregiver Inadequate malnourished LBW baby breastfeeding, weaning Frequent Inadequate infections catch-up in Inadequate Inadequate food, growth, fetal nutrition care, health, WASH delayed Mother milestones Child malnourished, low stunted energy Pregnancy Reduced cognitive low weight gain capacity Inadequate food, Adolescent learning, health, stunted WASH Higher neonatal Inadequate food, care, mortality, anemia Reduced capacity to contribute health, WASH socially and economically Source: Adapted from United Nations Administrative Committee on Coordination Sub-Committee on Nutrition 2000 and United Nations Commission on the Nutrition Challenges of the 21st Century 2000. Note: LBW = low birth weight; WASH = water, sanitation, and hygiene. 2 1.5 Global reporting on child undernutrition shows mixed progress in reducing the stunting of children under five, with Africa and South Asia most severely affected. Globally, more than 150 million children (22 percent) were stunted in 2018, compared with 198 million (33 percent) in 2000. All affected regions have reported some reductions in stunting since 2000: South Asia (from 51 to 34 percent), East Asia and Pacific (from 25 to 12 percent), Middle East and North Africa (from 23 to 15 percent), and Latin America and the Caribbean (from 17 to 9 percent). In Africa, although there has been an overall improvement in stunting rates since 2000 (from 43 to 34 percent), the undernutrition situation is worsening while the population is growing (from 50.6 to 58.7 million), so the total number of stunted children is increasing (Development Initiatives 2018; United Nations Children's Fund [UNICEF], World Health Organization [WHO], and World Bank 2019). 1.6 Progress varies on other global nutrition 2025 targets for women and children, such as reduced rates of anemia and LBW and increased exclusive breastfeeding. However, individual countries may have more significant improvements. The latest figures show that the prevalence of anemia in girls and women of reproductive age appears to have stagnated at 33 percent, from 34 percent in 2000. Approximately 20 million babies are LBW globally, compared with 22.9 million in 2000. As with stunting, lower-income countries are most affected, especially Africa (5.7 million) and South Asia (9.8 million). In these countries, only 41 percent of children aged six months or younger are exclusively breastfed (43 percent in Africa; 54 percent in South Asia) (Development Initiatives 2018; UNICEF, WHO, and World Bank 2019). Evolution of the Global Nutrition Agenda 1.7 Building human capital requires investment to improve child undernutrition and its determinants, particularly in countries where it is a high burden. Therefore, improving undernutrition has become central to the global development agenda in the past decade. This is accompanied by political commitment from governments and financing resources to support nutrition. 1.8 The World Bank’s nutrition agenda has evolved from its early focus on food into a more multidimensional and multisector collaborative agenda (figure 1.2). In the 1970s, nutrition was integrated into poverty reduction through multisectoral rural development projects. Government commitments to implement these projects were weak, and implementation arrangements were complex (MacNally 1983; World Bank 2014). Later projects either focused mainly on the health sector (Berg 1987), where the challenge became how to meaningfully integrate nutrition interventions into one component of the project and scale up interventions in health services, or on interventions that were confined to small geographical areas. 3 Figure 1.2. Evolution of Global Nutrition Agenda Agriculture sector Rome starts tracking Declaration on nutrition-sensitive Nutrition approaches in Nutrition and rural projects, following Increasing role of development World Bank Adoption of SDG Governance sector in integrated in poverty Agriculture Action N4G raises nutrition target nutrition financing Institutional developments reduction; focus on Plan commitment to and stunting food intake nutrition indicator Human Capital Project increases Nutrition-sensitive World Health Assembly UN Decade of Launch of Early focus on nutrition WASH Approaches Comprehensive Implementation Action on MDGs childhood Nutrition (2016- Plan on Nutrition Nutrition-smart Review of nutrition nutrition 2025) in first World Bank SecureNutrition part of agriculture projects; First Knowledge Platform World Bank Early Years innovative approach multisectoral shares knowledge response to Initiative projects had limited shared increases focus on success Launch of the SUN movement prosperity nutrition 1970s-90s 2000 2006 2008 2010 2012 2013 2014 2015 2016 2017 2018 2019 Malnutrition What Second First Global Lancet An Investment Can be Done? – Repositioning First Lancet Scaling Up Improving Nutrition series Framework for Key publications shifted focus to Nutrition as series, providing Nutrition: What Nutrition Report Nutrition health sector Central to evidence on Will It Cost?, through Development – interventions first estimate of Multisectoral All Hands on Deck – Combating galvanizing financing needs Approaches, Future of Food--Food multisectoral Malnutrition, leadership framed multi- Incentivizing system and nutrition convergence Time to Act –- dimensional Nutrition outcomes UNICEF actions collaboration Scope of evaluation: FY2008-FY19 Black: Actions by the World Bank; Blue: Multipartner actions Source: Adapted from Rokx 2006 and Shekar et al. 2017. Note: MDG = Millennium Development Goal; N4G = Nutrition for Growth; SDG = Sustainable Development Goal; SUN = Scaling Up Nutrition; UN = United Nations; UNICEF = United Nations Children’s Fund; WASH = water, sanitation, and hygiene. 4 1.9 Development partners, including the United Nations Children's Fund (UNICEF) framework of the determinants of child undernutrition (UNICEF 1990, 2015) and United Nations Commission on Nutrition Challenges of the 21st Century (UNCNC21 2000), highlighted the need to address nutrition’s multidimensional determinants, including access to food, caregiving, health services, and WASH throughout the life cycle of mother and child. These publications initiated a series of efforts to collaborate with development partners (including the World Bank), strengthen country commitment, and galvanize leadership to reposition nutrition in the development agenda (Gillespie, McLachlan, and Shrimpton 2003; Heaver 2005; World Bank 2006). For example, the Scaling Up Nutrition (SUN) Movement, initiated in 2010, has brought together countries, sectors, and development partners to act on nutrition (SUN Movement 2010). In some countries, the movement has initiated reforms in policies and institutional arrangements to coordinate, plan, measure, and implement nutrition interventions and find solutions to overcome previous challenges relating to the countries’ ownership and delivery of the agenda; that is, nutrition does not fall within the mandate of any one sector (SUN Movement 2019). Within the World Bank, the commitment to the SUN Movement has renewed the engagement of sectors (agriculture, social protection, health, water, and so on) to address nutrition in country programs (Alderman 2016; Hawkes and Ruel 2008; World Bank 2013a, 2014). 1.10 In 2008, the first of several Lancet series on nutrition began consolidating the knowledge and evidence on interventions effective in improving nutritional outcomes.1 In 2010, the World Bank published the first estimates for financing nutrition interventions in countries; these estimates were later detailed in country level investment cases (Horton et al. 2010) and became the basis for mobilizing financing for nutrition and political commitment through Nutrition for Growth from 2013. Multisectoral knowledge sharing was also supported through the SecureNutrition Knowledge Platform (World Bank 2017). 1.11 In 2016, the Sustainable Development Goals (SDGs) committed to improve undernutrition, and the United Nations declared its Decade of Action on Nutrition (2016–25). The Millennium Development Goals focused on halving the prevalence of underweight children under five by 2015, which did not fully address the importance of nutrition to healthy growth and child development. SDG 2 is to end hunger, achieve food security and improved nutrition, and promote sustainable agriculture. Global nutrition targets set by the World Health Assembly for 2025 include a 40 percent 1See Breastfeeding Series Group (2016), Early Childhood Development Series Steering Committee (2016), Maternal and Child Nutrition Study Group (2013), and Maternal and Child Undernutrition Study Group (2008). 5 reduction in stunting, a 50 percent reduction in anemia in women, a 30 percent reduction in LBW newborns, and an achievement of at least 50 percent for exclusive breastfeeding (WHO 2014). The focus on stunting in the SDGs was influenced by the World Bank’s new strategy of reducing extreme poverty and promoting shared prosperity and its emphasis on inequalities in early childhood development and nutrition (Denboba et al. 2014; World Bank 2013c). Since the creation of the SDGs, the World Bank has supported nutrition investments in countries and the development of an investment framework to achieve the global nutrition targets (Laviolette et al. 2016; Shekar et al. 2017; WHO 2014). 1.12 The launch of the World Bank’s Human Capital Project in 2018 further reinforced the importance of improving child undernutrition and of implementing a package of multidimensional interventions to achieve results. The percentage of children under five who are not stunted is now used as a proxy for healthy child growth in the Human Capital Index (World Bank 2018). The human capital agenda has led to (i) efforts to improve data on nutrition indicators and determinants and (ii) analysis to understand aspects of multidimensionality relating to how interventions from different sectors can be prioritized in a country and be assembled in a package that does not necessarily require complex multisectoral coordination (UNICEF, WHO, and World Bank 2019; World Bank 2019a). Countries are now collecting data for the first time on the economic costs of stunting in children (Galasso and Wagstaff 2018). 1.13 Recent World Bank efforts engage the health, agriculture, WASH, social protection, education, social development, urban, macroeconomic, and governance sectors. Sectors are tracking and defining nutrition interventions related to health, food systems, and WASH, among other indicators (Chase et al. 2019; Jaffee et al. 2019; Shekar 2019; World Bank 2016, 2019b). Other recent experiences optimized investments and converged multisectoral approaches to benefit targeted communities or households burdened by undernutrition in Indonesia, Peru, and Senegal (Pearson et al. 2018). Multisectoral support efforts often focus on national coordination of interventions, synergistic geographic coverage, or sequenced timing of interventions. The current evaluation is well-timed to coincide with the Global Nutrition Summit in Japan in 2020. Value Added of the Evaluation 1.14 The evaluation is unique because it seeks to learn from the multisectoral portfolio of World Bank support to child undernutrition across the health, agriculture, WASH, social protection, education, urban, social development, macroeconomic, and governance sectors. This will add to previous efforts of the Independent Evaluation Group (IEG) to learn from nutrition-related investments: (i) What Can We Learn from Nutrition Impact Evaluations? Lessons from a Review of Interventions to Reduce Child 6 Malnutrition in Developing Countries (World Bank 2010), (ii) The World Bank Group and the Global Food Crisis: An Evaluation of the World Bank Group Response (World Bank 2013b), (iii) World Bank Support to Early Childhood Development (World Bank 2015), and (iv) Later Impacts of Early Childhood Interventions: A Systematic Review (Tanner, Candland, and Odden 2015). 1.15 Through a review of the literature and consultations with World Bank staff engaged in nutrition across Global Practices, the team has identified areas in which the evaluation findings can contribute to learning for future operations. The following areas have framed the design of this evaluation: • Learning related to evidence-based interventions that can improve child undernutrition outcomes and its determinants in different country situations, including the targeting and design of these interventions; • Multisectoral coordination and clarity of operational approaches, particularly in terms of addressing multidimensional determinants of undernutrition; • Tracking and measuring of results from investments in nutrition across a range of projects, considering nutrition is often a small proportion of a project; • Strengthening of institutions in countries (horizontal and vertical arrangements, reaching communities) to ensure that a prioritized package of quality nutrition- related interventions is delivered to beneficiaries in a sustainable manner; and • Behavioral changes, their sustainability, and their ability to influence determinants of undernutrition in countries. 2. Conceptual Framework 2.1 The conceptual framework underpinning this evaluation is adapted from the UNICEF framework of the determinants of child undernutrition (Maternal and Child Nutrition Study Group 2013; UNCNC21 2000; UNICEF 1990, 2015). The framework models interlinked building blocks to sustainably address child undernutrition in a country context. These building blocks are immediate and underlying determinants of undernutrition, social norms and behaviors, multidimensional interventions to influence outcomes, enabling environment interventions, and factors within the country used to prioritize and target interventions (figure 2.1). Improving nutrition at different points in the lifecycle of mother and child provides the foundation for these building blocks. 7 Figure 2.1. Conceptual Framework for Evaluation, Focused on Undernutrition of Women and Children Human capital benefits during the life course Morbidity and mortality in Cognitive, motor, socio- School performance and Work capacity and Economic growth childhood emotional development learning capacity productivity life cycle of mother Outcomes in the Outcomes for future mothers, mothers, children under 5 Improved undernutrition (stunting, wasting, underweight, low birth and child weight, breastfeeding, micronutrient deficiencies) Ag Nutrition-specific Nutrition-sensitive Preconception through early childhood (first 1,000 days) Improved Improved feeding Improved health of SP interventions (Examples) interventions (Examples) nutrient intake/ and caregiving mother and child Interventions throughout the life course Determinants of undernutrition; and diet diversity intermediate outcomes and outputs • Adolescent nutrition • Agriculture and food • Maternal nutrition systems, safety, Edu • Breastfeeding, child security approaches feeding and stimulation • Social safety nets for • Dietary support and households with micronutrient Maternal and Access to health children Access to nutrient- Water childcare services and • Complete early child Lending supplementation or rich food Lending resources WASH development (such as fortification for children Health • Treatment of stimulation, learning, malnutrition care, nutritious food, Urban Analytical Analytical work • Childhood disease Social norms and behaviors safe environment) work prevention and (service providers, community, household, mothers and caregivers) • Women and girls’ management empowerment (literacy, etc.) Social • WASH approaches Coordination (such as access to clean Environment to enable outcomes Coordination with partners water, hygiene • Nutrition policies, financing and coordination with partners Gov promotion) • Arrangements to deliver interventions (Ex. strategies, targeting, • Maternal and child capabilities, coverage of difficult to reach communities, health and family partnerships, M&E and learning) planning approaches Macro • Engagement and ownership (Ex. Commitment of leaders, community participation and engagement of citizens, demand for accountability, transparent information) Lending Lending Health Institutional strengthening interventions Analytical Analytical work work Country context: inequalities in the distribution of outcomes; poverty; health status; demographics; status of women; fragility and conflict; politics; environment Source: Adapted from Maternal and Child Nutrition Study Group 2013 and UNICEF 1990. Note: The assessment of the contribution of the World Bank’s nutrition support to human capital benefits is outside of the scope of the evaluation. Ag = agriculture; Edu = education; Gov = governance; LBW = low birth weight; Social = social development; Macro = Macroeconomic; M&E = monitoring and evaluation; SP = social protection; WASH = water, sanitation, and hygiene; Ex. = examples. 8 2.2 The framework premises that stunting, LBW, and micronutrient deficiencies are lower among women and children with adequate underlying nutritional determinants. Immediate determinants of child undernutrition relate to caregiving practices, inadequate dietary intake or diversity, and the health status of the mother and child. These factors cannot be overcome when communities lack adequate access to underlying determinants of undernutrition, including (i) nutrient-rich food, (ii) caregiving resources, (iii) health services, and (iv) WASH. Improvements in these underlying determinants are interdependent; that is, access to food is not enough without adequate feeding, proper care, adequate and accessible health services, and clean water. 2.3 Successfully addressing both the immediate and underlying determinants of child undernutrition requires transforming social and behavioral norms relating to feeding, caregiving, health, and WASH practices, including those related to gender relations and practices. Behavioral interventions are central to the framework and can target women, caregivers, children, and other agents of change who can influence the prevailing social norms at both the community and household level (for example, household members and community leaders). 2.4 The framework suggests that nutrition interventions within a country need to be multidimensional to address both the immediate and underlying determinants of undernutrition in their context, and this may require integrating interventions related to multiple sectors. Nutrition-specific interventions, such as adolescent nutrition, maternal nutrition, breastfeeding support, micronutrient supplementation, child disease prevention and management, and treatment of undernutrition, are expected to influence immediate determinants of undernutrition. However, nutrition-sensitive interventions, such as cash transfers, WASH approaches, girls’ education, and food system improvements, are expected to address underlying determinants of undernutrition. Nutrition-specific interventions are often delivered by the health system and target women and children, whereas nutrition-sensitive interventions may be delivered by various sectors and target households and communities or geographies with inadequate determinants of undernutrition (access to nutritious food, caregiving resources, health services, and WASH). 2.5 The country-specific situation, including the distribution of outcomes, frames the context in which to prioritize and target interventions. The enabling environment frames interventions to strengthen institutional capacities at national and subnational levels over time in a country to support outcomes. Factors of fragility and distributional factors related to inequalities in nutritional outcomes, health and education status, and poverty can create different country scenarios in which to prioritize and target interventions to improve undernutrition. Moreover, the distribution of determinants in a population — that is, access to nutritious foods, caregiving resources, health services, and WASH—can 9 provide information on investment needs. The institutional capacities in the enabling environment at the national and subnational levels can frame priorities for interventions to improve the delivery of services and programs, the engagement of communities, and the implementation of policies to address nutrition in countries. 3. Objectives and Audience 3.1 The objectives of the evaluation are to assess the contribution of the World Bank to improve outcomes related to child undernutrition and its determinants in countries burdened by undernutrition and to provide lessons and recommendations to inform the design of the World Bank’s future multidimensional nutrition support. Because of the breadth of the nutrition portfolio, the evaluation is expected to help fill an accountability gap by providing evidence on results across sectors and lessons from operational experience to feed into country strategies and project design in countries, particularly those countries where the prevalence of childhood stunting is high and an important factor inhibiting the healthy growth of children and the accumulation of human capital. The intended audiences for the evaluation include (i) management of World Bank country and sector programs, (ii) task teams in sectors with roles that address nutrition, (iii) client countries, and (iv) development partners and practitioners that engage with the World Bank in countries where undernutrition is a priority. 4. Evaluation Questions and Scope 4.1 The evaluation questions and scope are informed by two main sources of information: (i) consultations with World Bank staff engaged with nutrition and (ii) a review of the literature. Evaluation Questions 4.2 Motivating this evaluation is this overarching question: What has been the contribution of World Bank support to improve outcomes and intermediate outcomes of child undernutrition and its determinants in countries burdened by undernutrition? Underlying this question are three main lines of inquiry: • EQ1. To what extent is the World Bank supporting relevant interventions to improve outcomes and intermediate outcomes of child undernutrition and its determinants within the country context? o 1a. How consistent is World Bank support with the latest evidence on supporting nutrition-specific and nutrition-sensitive interventions? o 1b. To what extent is World Bank support aligned to the country situation and the populations or geographies burdened by undernutrition? 10 • EQ2. How is the World Bank implementing multidimensional approaches to support outcomes and intermediate outcomes that improve child undernutrition and its determinants, and strengthen countries’ institutional capacities ? o 2a. To what extent has the World Bank supported the implementation of multidimensional approaches in countries (types of approaches, institutional capacities to ensure successful approaches)? o 2b. To what extent has the World Bank supported policy dialogue, knowledge generation, and the convening of actors at the country level to ensure effective multidimensional approaches that enable institutional capacities for nutrition and exploit sectoral synergies and collaborations with other development partners? • EQ3. To what extent have the World Bank interventions contributed outcomes, intermediate outcomes, and outputs toward the building blocks of the conceptual framework, and what were the factors of success and failure (contextual, design, implementation, benefits to populations and geographies burdened by undernutrition)? Evaluation Scope 4.3 Three parameters determine the scope of this evaluation: time coverage, country perspectives, and conceptual boundaries. • Time coverage: The scope of the evaluation portfolio in countries with high levels of stunting is limited to 10 years of World Bank engagements in nutrition (investment operations, development policy lending, and recipient- executed trust funds) from fiscal year (FY)08 to FY19. The review of knowledge work and World Bank–executed trust funds will be limited to case study countries. The time line of the evaluation aligns with important actions in the evolution of the nutrition agenda (figure 1.2). International Finance Corporation and Multilateral Investment Guarantee Agency operations are outside the scope of the evaluation. A preliminary review suggested that their investments are often upstream of the nutrition interventions at the community and household levels covered by this evaluation’s framework. • Country perspectives: To focus the scope of the evaluation on countries burdened by undernutrition, the evaluation portfolio will be limited to those countries with relevant World Bank operations in the health, social protection, agriculture, water, urban, social development, macroeconomic, and governance sectors that have reported high levels of stunting in children under five 11 (greater than or equal to 20 percent)2 during any year of the evaluation period. The use of stunting rates rather than other undernutrition outcomes to narrow the scope of the portfolio reflects its importance to the Human Capital Index. In these countries, the evaluation will review the portfolio of World Bank support to nutrition, applying a country perspective as opposed to a project-level perspective. This is because of the integrated nature of the portfolio and the need to review the package of interventions that have been supported across projects and sectors in the country programs during the time line. • Conceptual boundaries: The evaluation will be bounded by the conceptual framework and its building blocks presented in figure 2.1. Therefore, it will focus on outcomes, intermediate outcomes, and outputs related to undernutrition and its immediate and underlying determinants. The evaluation will not assess the World Bank’s contribution to higher-level human capital benefits. Obesity is also outside the evaluation scope because it is a relatively newer challenge within the work of the World Bank, and evidence is presumed to be scarce. Obesity also spills into adulthood as a risk factor for noncommunicable diseases and would merit a stand-alone IEG evaluation in the future. 4.4 A preliminary review of the portfolio (see appendix B) shows 390 operations addressing nutrition in the last 10 years in countries with high levels of stunting (International Development Association [IDA]: 75 percent; International Bank for Reconstruction and Development [IBRD]: 6 percent; recipient-executed trust funds: 18 percent; and IDA–IBRD blend: 1 percent). Nutrition is addressed either in their objectives or through interventions in the project components. These operations are within 69 countries: 39 countries in Africa, 10 countries in Latin America and the Caribbean, 8 countries in East Asia and Pacific, 6 countries in South Asia, 4 countries in Europe and Central Asia, and 2 countries in Middle East and North Africa. 5. Evaluation Design 5.1 The evaluation design adopts a multilevel analysis at the global, portfolio, country, and intervention levels. It uses a mixed-method approach that combines quantitative and qualitative evaluative evidence focusing on countries with high levels of stunting. The evaluation design applies the following principles: 2See De Onis et al. (2018). Recent threshold levels for childhood stunting are used to broaden the scope of the portfolio to include countries that would have otherwise been considered to have medium stunting levels by the 2010 World Health Organization (WHO) standard. 12 • Participatory: From the onset of the evaluation, the evaluation team consulted with technical staff across Global Practices and from the Human Capital Project and Nutrition Global Solutions Group to identify key areas in which the evaluation findings can contribute to learning. World Bank staff feedback helped the team frame the evaluation questions and design. This engagement is expected to continue throughout the evaluation to share sampling methods and interpret learning. • Theory-based: As specified previously, the evaluation is based on a conceptual framework (figure 2.1). In the case study countries, nested country-specific theories of change will be developed to examine the building blocks of the conceptual framework and assess the multidimensional contribution of World Bank support (investment operations, development policy lending, knowledge work, and trust funds) to child undernutrition and its determinants. The country-specific theories of change will examine the logic of the conceptual framework in the country-specific contexts, including nutrition-sensitive, nutrition-specific, behavioral change, and enabling environment interventions, as well as outcomes, intermediate outcomes, and outputs. Moreover, the knowledge gained from the case studies and other components of the evaluation will be used to refine the evaluation’s conceptual framework to outline how the building blocks have been applied in the World Bank’s support. • Case-based: A main focus of the evaluation is country learning. The evaluation will include a case-based analysis of the World Bank portfolio in eight countries: four cases will be limited to a desk review of documents complemented by interviews, and four cases will include field visits. The field visits within case study countries will purposefully sample the geographical areas to interview beneficiaries of interventions. The inclusion criteria for the countries are countries with (i) at least one closed and IEG-evaluated project with a nutrition focus in the title or objective, (ii) support for institutional strengthening and behavioral change interventions, and (iii) projects in at least three Global Practices. The design follows a nested approach to address evaluation questions at the country and intervention levels. Included case study countries will vary in terms of changes in their stunting rates during the evaluation period3 and project 3The evaluation will use the average annual change in stunting rates between the two data points closest to 2008 and 2019 that correspond to the evaluation period to cluster case study countries. This responds to the data availability and heterogeneity in the period between data on stunting rates for countries. 13 performance on nutrition-related measures.4 Other considerations will include the availability of impact evaluations of the World Bank’s interventions and other evidence in the country that could support learning and the extent that the country’s experience in improving undernutrition is already documented. Refinement of country selection and sampling will follow IEG methods guidance. Evaluation Components 5.2 The evaluation components at each level are outlined subsequently. See appendix A for the evaluation design matrix. Components at the Global Level • Structured literature review: At the global level, the evaluation will review the available evidence on the effectiveness of nutrition-specific and nutrition- sensitive interventions from systematic reviews only. The review will be organized around outcomes and intermediate outcomes in the conceptual framework (figure 2.1) to consolidate available evidence on interventions to improve undernutrition and its determinants. This exercise will produce an evidence gap map that highlights areas where the existing evidence base is still emerging. In addition, the evaluation will include a review of qualitative literature, such as systematic reviews and ethnographic studies, to examine the available evidence on social and behavioral norms related to the determinants of undernutrition. This exercise will produce a process map that highlights behavioral changes to support determinants of undernutrition. The outputs will be used to benchmark nutrition interventions in the portfolio and case studies against the conceptual framework and evidence from the literature. Components at the Portfolio Level • Portfolio identification, review, and analysis: At the portfolio level, the evaluation will use text analytics and machine learning to identify World Bank operational support for nutrition, including nutrition-specific, nutrition-sensitive, and enabling environment interventions. The portfolio analysis will be restricted to countries with high levels of stunting in any given year during FY08–19, extracting information from project appraisal documents, program documents, project papers, Implementation Status and Results Reports, Implementation Completion and Results Reports, and Implementation Completion and Results 4As a proxy for project performance, the evaluation will use achievement rates of nutrition- related indicators of closed and IEG-evaluated projects. 14 Report Reviews for information on project objectives, interventions, and indicators. The portfolio will be screened against the list of nutrition-relevant projects identified by the Nutrition Global Solutions Group and Global Practices. • Heat map of country investments: For the portfolio of countries with high levels of stunting, the evaluation will correlate data on the country situation related to each of the determinants of undernutrition (that is, access to nutritious food, caregiving resources, WASH, and health services), and the enabling environment, with project portfolio data on nutrition interventions in countries to assess the alignment and consistency of the World Bank’s support. The analysis will use data from the Joint Child Malnutrition Estimates (UNICEF, WHO, and World Bank 2019), Demographic and Health Surveys, and Multiple Indicator Cluster Surveys. This analysis will build on existing work on measuring determinants of undernutrition (World Bank 2019a). • Stocktaking of multidimensional approaches: The evaluation will review the literature on country experiences to qualitatively categorize approaches to coordinate or combine nutrition interventions within countries, including the challenges and rationales for different approaches. The sample will be limited to approximately 10 countries in the portfolio that have peer-reviewed literature on their experiences. The focus would be on the government’s coordination of nutrition at the national and subnational level, and the institutional arrangements (at policy, subnational, and community levels). In the case study countries, the evaluation will review approaches used to synergize and coordinate World Bank and partner investments to support the nutrition program in a country, such as geography, time sequencing, results measured, beneficiary rosters, types of interventions, and financing instruments. • Indicator mapping: For the portfolio of countries with high levels of stunting, indicators used at the project level will be mapped against the conceptual framework to assess the extent to which the World Bank has contributed to and measured the appropriate results and to identify gaps and innovations in the monitoring and evaluation of results. The evaluation will also review the literature to identify indicators at output, intermediate outcome, and outcome as benchmarks to assess World Bank support and to gain insights to improve monitoring and evaluation. Components at the Country Level • Interviews: For case study countries, the evaluation will interview staff from the World Bank (task teams, country management, and experts engaged in nutrition support) and governments, beneficiaries, and development partners in the 15 countries. These will include members of nutrition working groups and civil society who have been engaged in interventions in the case study countries. The interviewees will be selected using a stakeholder analysis, and interviews will focus on learning what interventions have been implemented, how interventions have been converged to support outcomes, and what the main achievements and successes and failures have been. These include factors related to country context, project design, implementation of interventions, targeting populations and geographies burdened by undernutrition, and the integration of support with other sectors and partners. • Country portfolio review: For case study countries, the evaluation will review the package of nutrition interventions in the country program (including their targeting and scale) and multidimensional approaches. The focus will be on the synergies of nutrition interventions in the portfolio implemented through simultaneous or sequential investments in operations, development policy lending, or knowledge work, or through coordinating investments with other development partners. This will involve reviewing World Bank documents from the 10-year evaluation period (Country Partnership Framework or Country Assistance Strategy, project appraisal documents, program documents, knowledge work, and trust funds), country strategies, reports and plans, disaggregated data, and interviews (see interview component). • Institutional mapping: For case study countries, the evaluation will map synergies and collaboration with other development partners and how the World Bank played a role in the convening of actors, the policy dialogue, and knowledge generation to support the governments’ nutrition policies, including targeting, prioritizing, and scaling up programs. This evaluation will be conducted through the review of documents and interviews (see interview component). • Evidence from evaluations: For case study countries, the evaluation will review World Bank impact evaluations and relevant knowledge work, particularly on nutrition-sensitive interventions, where evidence is weaker, to assess (i) the contribution of World Bank support to project impacts and (ii) potential success and failure factors within country-specific contexts. The evaluation will also use efficacy findings and possibly success and failure factors from Project Performance Assessment Reports on nutrition projects. • Contribution analysis: For case study countries, the evaluation will review the outputs, intermediate outcomes, and outcomes achieved by the interventions in the country program against the different building blocks of the conceptual 16 framework. This will involve the development of a country-specific theory of change to assess how the nutrition interventions in the country program have contributed to the building blocks of the conceptual framework. The analysis will also look at the distribution of those outputs, intermediate outcomes, and outcomes in the population and targeted geographies and collect supporting evidence through interviews and available data to fill gaps in project-level reporting. Components at the Intervention Level • Assessment of behavioral change: For case study countries, the evaluation will identify projects with behavioral change interventions, which will be reviewed using the process map developed from the literature and through interviews with beneficiaries. 6. Quality Assurance Process 6.1 Several steps will be undertaken to ensure the quality and usefulness of the evaluation findings. The evaluation will go through IEG’s quality assurance processes. Furthermore, four independent external experts will provide guidance: • Dr. Bruno Marchal (evaluation methods expert at the Institute of Tropical Medicine, Antwerp); • Dr. Olivia Yambi (co-chair, International Panel of Experts on Sustainable Food Systems, and former UNICEF regional nutrition adviser for Eastern and Southern Africa and representative in India, Kenya, and Lao People’s Democratic Republic); • Shawn Baker (chief nutritionist at United States Agency for International Development [USAID], and former director of nutrition at the Bill and Melinda Gates Foundation and vice president for Africa at Helen Keller International); and • Professor James Levinson (former professor at Boston University, Friedman School of Nutrition at Tufts University, Harvard University, and University of Massachusetts Amherst, and former director of nutrition at Tufts University International Food and Nutrition Center, Massachusetts Institute of Technology, and USAID). 6.2 Also, the evaluation team will engage experts in the World Bank’s Global Practices and from the Human Capital Project and Nutrition Global Solutions Group 17 and other thought leaders in bilateral and group consultations throughout the evaluation. 7. Expected Outputs and Outreach 7.1 The evaluation will occur in FY20 and reviewed by IEG and World Bank management in FY21. Committee on Development Effectiveness discussions are expected to take place in the second quarter of FY21. The main output will be a report of up to 20,000 words plus appendixes. 7.2 The evaluation team will continue to engage with a range of key stakeholders within the World Bank, following its participatory principles. Focus group discussions, structured brainstorming sessions, blogs, and workshops will be organized throughout the evaluation to support analyses. The evaluation team will also engage with other internal stakeholders—notably, country management units in selected countries. The consultations are to strengthen the quality, relevance, and ownership of the evaluation findings and foster “process use” opportunities. The dissemination plan for the evaluation will be developed in collaboration with the IEG communications team and the Global Practices, including sharing preliminary findings for the Global Nutrition Summit in Japan in 2020. 8. Resources 8.1 The evaluation will be prepared with an estimated budget of $930,000. The core IEG team members for the evaluation are Jenny Gold (task team leader), Maria De Las Mercedes Vellez (task team leader), April Connelly, Ann Flanagan, and Santiago Ramirez. The work will be conducted under the guidance of Estelle Raimondo (methods adviser), Galina Sotirova (manager), Oscar Calvo-Gonzalez (director), and Alison Evans (Director-General, Evaluation). 18 References Alderman, H. 2016. “Leveraging Social Protection Programs for Improved Nutrition: Summary of Evidence Prepared for the Global Forum on Nutrition-Sensitive Social Protection Programs, 2015.” World Bank, Washington, DC. Berg, A. 1987. Malnutrition: What Can be Done? Lessons from World Bank Experience. Washington, DC: World Bank. Breastfeeding Series Group. 2016. Breastfeeding Series. Lancet 387 (10017). https://www.thelancet.com/series/breastfeeding. Chase, C., A. Bahuguna, Y. Chen, S. Haque, and M. Schulte. 2019. Water and Nutrition: A Framework for Action. Water Global Practice. Washington, DC: World Bank. Denboba, A., R. Sayre, Q. Wodon, L. Elder, L. Rawlings, and J. Lombardi. 2014. Stepping Up Early Childhood Development: Investing in Young Children for High Returns. Washington, DC: World Bank. De Onis, M., E. Borgi, M. Arimond, P. Webb, T. Croft, K. Saha, L. De-Regil, F. Thuita, R. Heidkamp, J. Krasevec, C. Hayashi, and R. Flores-Ayala. 2018. “Prevalence Thresholds for Wasting, Overweight and Stunting in Children Under 5 Years.” Public Health Nutrition 22 (1): 175–179. Development Initiatives. 2018. 2018 Global Nutrition Report: Shining a Light to Spur Action on Nutrition. Bristol: Development Initiatives. Early Childhood Development Series Steering Committee. 2016. Advancing Early Childhood Development: From Science to Scale. Lancet 389 (10064). https://www.thelancet.com/series/ECD2016. Galasso, E., and A. Wagstaff. 2018. “The Aggregate Income Losses from Childhood Stunting and the Returns to a Nutrition Intervention Aimed at Reducing Stunting.” Policy Research Working Paper 8536, World Bank, Washington, DC. Gillespie, S., M. McLachlan, and R. Shrimpton, eds. 2003. Combating Nutrition: Time to Act. Health, Nutrition, and Population Series. Washington, DC: World Bank. Hawkes, C., and M. T. Ruel. 2008. “From Agriculture to Nutrition: Pathways, Synergies and Outcomes.” Agricultural and Rural Development Notes 40, World Bank, Washington, DC. Heaver, R. 2005. Strengthening Country Commitment to Human Development: Lessons from Nutrition. Directions in Development Series. Washington, DC: World Bank. Horton, S., M. Shekar, C. McDonald, A. Mahal, and J. K. Brooks. 2010. Scaling Up Nutrition: What Will It Cost? Directions in Development Series. Washington, DC: World Bank. 19 Jaffee, S., S. Henson, L. Unnevehr, D. Grace, and E. Cassou. 2019. The Safe Food Imperative: Accelerating Progress in Low- and Middle-Income Countries. Agriculture and Food Series. Washington, DC: World Bank. Laviolette, L., S. Gopalan, L. Elder, and O. Wouters. 2016. Incentivizing Nutrition: Incentive Mechanisms to Accelerate Improved Nutrition Outcomes. Washington, DC: World Bank. MacNally, W. 1983. “World Bank Assistance to Food and Agriculture 1974–84.” AGREP Division Working Paper 82, World Bank, Washington, DC. Maternal and Child Nutrition Study Group. 2013. Maternal and Child Nutrition Series. Lancet 382 (9890). https://www.thelancet.com/series/maternal-and-child-nutrition. Maternal and Child Undernutrition Study Group. 2008. Maternal and Child Undernutrition Series. Lancet 371 (9608). https://www.thelancet.com/series/maternal-and-child- undernutrition. Pearson, R., M. Killedar, J. Petravic, J. Kakietek, N. Scott, K. Grantham, R. Stuart, D. Kedziora, C. Kerr, J. Skordis-Worrall, M. Shekar, and D. Wilson. 2018. “Optima Nutrition: An Allocative Efficiency Tool to Reduce Childhood Stunting by Better Targeting of Nutrition-Related Interventions.” BMC Public Health 18: 384. doi:10.1186/s12889-018-5294- z. Rokx, C. 2006. “Governance and Malnutrition, Exploring the Contribution of ‘Good Governance’ to Malnutrition Reduction in Developing Countries.” PhD thesis, Maastricht University. Shekar, M. 2019. “A Growing Nutrition Portfolio to Tackle Malnutrition and Obesity.” Investing in Health (blog), July 15. http://blogs.worldbank.org/health/growing-nutrition-portfolio- tackle-malnutrition-and-obesity. Shekar, M., J. Kakietek, J. Dayton Eberwein, and D. Walters. 2017. An Investment Framework for Nutrition: Reaching the Global Targets for Stunting, Anemia, Breastfeeding, and Wasting . Directions in Development Series. Washington, DC: World Bank. SUN (Scaling up Nutrition) Movement. 2010. Scaling Up Nutrition: A Framework for Action. Geneva: SUN Movement. ———. 2019. Nourishing People and Planet Together: Scaling Up Nutrition (SUN) Movement Progress Report 2019. Geneva: SUN Movement. https://scalingupnutrition.org/wp- content/uploads/2019/11/SUN-Annual-Report-2019-ENG_web_FINAL.pdf. Tanner, J., T. Candland, and W. Odden. 2015. “Later Impacts of Early Childhood Interventions: A Systematic Review.” IEG Working Paper 2105/3. Independent Evaluation Group, World Bank, Washington, DC. 20 UNICEF (United Nations Children’s Fund). 1990. Strategy for Improved Nutrition of Children and Women in Developing Countries. New York: UNICEF. http://ceecis.org/iodine/01_global/01_pl/01_01_other_1992_unicef.pdf. ———. 2015. “UNICEF’s Approach to Scaling Up Nutrition for Mothers and Their Children.” Discussion Paper, Programme Division, UNICEF, New York, June 2015. UNICEF, WHO (World Health Organization), and the World Bank. 2019. Levels and Trends in Child Malnutrition: Key Findings of the 2019 Edition of the Joint Child Malnutrition Estimates. Geneva: WHO. UNCNC21 (United Nations Commission on the Nutrition Challenges of the 21st Century). 2000. Ending Malnutrition by 2020: An Agenda for Change in the Millennium. Geneva: United Nations. https://www.unscn.org/uploads/web/news/2000-FEB-Ending-Malnutrition-by- 2020-Agenda-for-Change-in-the-Millennium-Report.pdf. United Nations Administrative Committee on Coordination Sub-Committee on Nutrition (ACC/SCN). 2000. Fourth Report on the World Nutrition Situation. Geneva: ACC/SCN in Collaboration with International Food Policy Research Institute. WHO (World Health Organization). 2014. Comprehensive Implementation Plan on Maternal, Infant and Young Child Nutrition. Geneva: WHO. World Bank. 2006. Repositioning Nutrition as Central to Development: A Strategy for Large-Scale Action. Directions in Development Series. Washington, DC: World Bank. ———. 2010. What Can We Learn from Nutrition Impact Evaluations? Lessons from a Review of Interventions to Reduce Child Malnutrition in Developing Countries. Independent Evaluation Group. Washington, DC: World Bank. ———. 2013a. Improving Nutrition through Multisectoral Approaches. Washington, DC: World Bank. ———. 2013b. The World Bank Group and the Global Food Crisis: An Evaluation of the World Bank Group Response. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbank.org/sites/default/files/Data/Evaluation/files/food_crisis_eval.pdf. ———. 2013c. World Bank Group Strategy. Washington, DC: World Bank. ———. 2014. Learning from World Bank History: Agriculture and Food-Based Approaches for Addressing Malnutrition. Washington, DC: World Bank. ———. 2015. World Bank Support to Early Childhood Development. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbank.org/sites/default/files/Data/Evaluation/files/early_child_dev_eval.p df. ———. 2016. Future of Food: Shaping the Global Food System to Deliver Improved Nutrition and Health . Washington, DC: World Bank. 21 ———. 2017. SecureNutrition History, Adaptation, and Selected Impact from Inception through December 2017. Washington, DC: World Bank. ———. 2018. The Human Capital Project. Washington, DC: World Bank. ———. 2019a. All Hands on Deck: Reducing Stunting through Multisectoral Efforts in Sub-Saharan Africa. Washington, DC: World Bank. ———. 2019b. “Nutrition-Sensitive Water Supply, Sanitation, and Hygiene.” World Bank, Washington, DC. 22 Appendix A. Evaluation Design Matrix Table A.1. describes the data collection and analysis methods to be addressed for each evaluation question, the required information and sources, and the strengths and limitations of the methods. Table A.1. Evaluation Design Matrix Data Collection and Analysis Key Questions Methods Information Required and Sources Strengths and Limitations Overarching evaluation question: What has been the contribution of World Bank support to improve outcomes and intermediate outcomes of child undernutrition and its determinants in countries burdened by undernutrition? EQ1. To what extent is the World Bank supporting relevant interventions to improve outcomes and intermediate outcomes of child undernutrition and its determinants within the country context? 1a. How consistent is World (1) Structured literature review of (1) Systematic reviews (SRs) of the Although focusing on SR evidence Bank support with the latest evidence on interventions to effectiveness of nutrition-specific and ensures the quality of the studies evidence on supporting produce an evidence gap map, nutrition-sensitive interventions to included, it may limit the evidence nutrition-specific and update the conceptual achieve outcomes and intermediate available for nutrition-sensitive nutrition-sensitive framework on the basis of the outcomes toward each building block interventions, which are less interventions? latest evidence, and create a of the conceptual framework. SRs will studied. checklist to benchmark be gathered from PubMed, Campbell The focus on countries with high interventions. Collaboration, Cochrane, 3ie, IFPRI, levels of stunting limits the (2) Literature review on behavior and so on. evaluation to countries where change interventions to create a (2) Studies on social norms and undernutrition is a main process map to review these behavioral change related to development challenge. interventions in case study determinants of undernutrition. Projects with clear nutrition countries. Project documents of World Bank interventions will be easier to (3) Portfolio identification, review, operations (IDA, IBRD, RETF), from identify in the portfolio. and analysis using text analytics FY08 to FY19, supporting nutrition The assessment of behavioral and machine learning to interventions in countries that have change interventions will be limited categorize nutrition interventions high estimates of stunting in children to case study countries because within projects and conduct under five (greater than or equal to such interventions are often not consistency analysis related to 20 percent) during the evaluation well-described in project the conceptual framework and period. documents. the evidence gap map. 23 Data Collection and Analysis Key Questions Methods Information Required and Sources Strengths and Limitations 1b. To what extent is World (1) Heat map that correlates data on (1) Country nutrition portfolio, data from National-level positive correlates Bank support aligned to the nutrition investments and the Joint Child Malnutrition Estimates between the country situation and country situation and the proxies of determinants of (UNICEF, WHO, and World Bank the World Bank’s support may populations or geographies undernutrition (that is, access to 2019), and country survey data on mask a misalignment at the burdened by undernutrition? nutritious food, caregiving nutrition indicators (United States subnational level. resources, WASH, health Agency for International Deeper analysis of portfolio will be services) and the enabling Development DHS Program limited to case study countries. environment, reflecting the STATcompiler 2020; United Nations country situation. Children’s Fund data set 2019). (2) Case-based review of the (2) Country nutrition portfolio in case nutrition portfolio in the country study countries, World Bank country to assess alignment with the documents (Country Partnership country situation. Framework, Country Assistance Strategy, policy notes, project (3) Structured interviews with World documents, knowledge work), country Bank task teams, experts, country reports and plans on nutrition, and management, government, and disaggregated data. beneficiaries. (3) Information on experiences implementing nutrition support, including behavioral change and enabling environment interventions in case study countries and stakeholder map to identify interviewees. 24 Data Collection and Analysis Key Questions Methods Information Required and Sources Strengths and Limitations EQ2. How is the World Bank implementing multidimensional approaches to support outcomes and intermediate outcomes that improve child undernutrition and its determinants, and strengthen countries’ institutional capacities? 2a. To what extent has the (1) Stocktaking to qualitatively (1) Published case studies, country This is an emerging area in which World Bank supported the categorize multidimensional national nutrition plans, and reports there is growing experience and implementation of approaches in countries and on experiences implementing opportunity for learning. multidimensional approaches inform the identification of multidimensional approaches in a The evaluation will categorize in countries (types of similar approaches in the sample of approximately 10 countries. multidimensional approaches but approaches, institutional portfolio. Information on experiences may not be able to assess their capacities to ensure successful (2) Structured interviews (see implementing nutrition-specific, effectiveness. approaches)? above). nutrition-sensitive, and enabling The feasibility of gathering environment approaches in case feedback will depend in part on the study countries (including targeted willingness of staff and clients to geographies and populations). participate. 2b. To what extent has the (1) Interviews with World Bank task (1) Information on partnerships, Biases that are inherent in World Bank supported policy teams, country management, analytical knowledge generated, and interviews (for example, recall bias dialogue, knowledge experts, and members of a stakeholder map of interviewees for or social desirability bias) will need generation, and the nutrition working groups, case study countries. Development to be carefully managed. convening of actors at the development partners, and partners may include BMFG, DFID, country level to ensure government actors involved in GFF, Japan, Norway, SUN, and USAID. effective multidimensional nutrition dialogue, coordination, (2) Information on synergies and approaches that enable knowledge generation, planning, collaboration efforts with other institutional capacities for and financing in case study development partners and how World nutrition and exploit sectoral countries. Bank convening, policy dialogue, and synergies and collaborations (2) Case-based institutional knowledge have influenced the with other development mapping of development governments’ nutritional policies and partners? collaborations and how the programs. World Bank has played a convening or synergistic role at the country level. (3) Case-based review of the nutrition portfolio in the country, including knowledge work. 25 Data Collection and Analysis Key Questions Methods Information Required and Sources Strengths and Limitations (3) Country nutrition portfolio in case study countries, World Bank country documents (Country Partnership Framework, Country Assistance Strategy, policy notes, project documents, knowledge work), country reports and plans on nutrition, and disaggregated data. EQ3. To what extent have the World Bank interventions contributed outcomes, intermediate outcomes, and outputs toward the building blocks of the conceptual framework, and what were the factors of success and failure (contextual, design, implementation, benefits to populations and geographies burdened by undernutrition)? (1) Mapping of nutrition indicators (1) Literature on nutrition indicators at This evaluation will not use project to the conceptual framework as output, intermediate outcome and outcome ratings or other ratings, as a benchmark for assessing World outcomes levels and on project-level most of the identified projects are Bank support. indicators from the nutrition portfolio not focused exclusively on (2) Case-based assessment of the extracted from ISRs, ICRs, and ICRRs. nutrition. Thus, it is not appropriate achievement rates of indicators (2) Project-level indicators and data on for this evaluation to use these as one measure of effectiveness their achievements from the nutrition ratings as a basis to gauge the at the project level. portfolio extracted from ISRs, ICRs, effectiveness and impact of these and ICRRs. interventions. (3) Case-based review of evaluation evidence to assess project (3) Impact evaluations of World Bank There are likely fewer impact impacts and potential success projects within case study countries, evaluations on nutrition-sensitive and failure factors. PPARs for the closed nutrition interventions. (4) Structured interviews (see projects, and studies on nutrition- above). sensitive interventions in case study The assessment of the contribution countries (for which there are gaps in to outcomes will be assessed the literature). against the conceptual framework, measuring outcomes at different levels. The assessment of the contribution of World Bank support to reductions in stunting is outside the scope of the evaluation. 26 Data Collection and Analysis Key Questions Methods Information Required and Sources Strengths and Limitations (5) Case-based analysis of the (4) Qualitative information on successes contribution of the World Bank and failures of World Bank support in support to outcomes, case study countries (such as related intermediate outcomes, and to country context, design, outputs at different levels of the implementation of interventions, conceptual framework. This will targeting populations and involve the development of geographies burdened by nested theories of change for undernutrition, synergizing support each case study country, linked with other sectors and partners). to the conceptual framework. (5) Information on the nutrition portfolio These nested theories of change in case study countries, including data will examine the logic of how the on outcomes and their distribution in conceptual framework has been geographies and populations and on applied to nutrition and roles of other partners in the targeted behavioral change interventions populations and geographies. in the countries to assess how the country program has supported outcomes for beneficiaries in geographies and populations burdened by undernutrition. Source: United Nations Children’s Fund Datasets, https://data.unicef.org/resources/resource-type/datasets; United States Agency for International Development DHS Program STATcompiler, https://www.statcompiler.com/en/. Note: BMFG = Bill and Melinda Gates Foundation; DFID = U.K. Department for International Development; GFF = Global Financing Facility; IBRD = International Bank for Reconstruction and Development; ICR = Implementation Completion and Results Report; ICRR = Implementation Completion and Results Report Review; IDA = International Development Association; IFPRI = International Food Policy Research Institute; ISR = Implementation Status and Results Report; PPAR = Project Performance Assessment Report; RETF = recipient-executed trust fund; SUN = Scaling Up Nutrition; UNICEF = United Nations Children's Fund; USAID = U.S. Agency for International Development; WASH = water, sanitation, and hygiene; WHO = World Health Organization. 27 Figure A.1. Methodological Design of the Evaluation Data Collection and Analysis Methods Evaluation Questions Global level Structured literature review (on effectiveness of Conceptual Framework EQ1, WHAT and WHY: To what nutrition-specific & extent is the World Bank nutrition sensitive supporting relevant interventions interventions; and behavioral change) to improve outcomes and, intermediate outcomes of child Portfolio identification, review undernutrition and its (countries with high childhood and analysis (on consistency determinants within the country Portfolio level and relevance of nutrition context? Stocktaking of interventions in portfolio) stunting) Indicators mapping multidimensional (indicators from projects approaches (to categorize mapped against the approaches to converge Heat map framework) interventions) (investments against the nutrition situation in countries) EQ2, HOW : How is the World Bank implementing multidimensional approaches to support outcomes and Secondary data sources intermediate outcomes that (country survey data) improve child undernutrition and Case studies (four desk based; four field based) Country portfolio review (on its determinants, and strengthen Institutional mapping & World Bank strategies, countries’ institutional Interviews with interventions, populations and capacities? Country level development partners geographies burdened by malnutrition, multidimensional Contribution analysis (use approaches) theories of change and Interviews with country evidence collected from management, task teams, projects to assess outcomes EQ3, RESULTS: To what extent experts and government against the framework) have the World Bank interventions contributed outcomes, intermediate Evidence from evaluations outcomes, and outputs towards (impacts, and success and the building blocks of the failure factors) conceptual framework, and what were the factors of success and failure (contextual, design, Interventio Field visits and interviews with Assessment of behavioral implementation, benefits to n level Process map for behavioral implementers of interventions change (on interventions in populations and geographies change interventions and beneficiaries projects at local level) burdened by undernutrition)? Source: Independent Evaluation Group. 28 References UNICEF (United Nations Children’s Fund), WHO (World Health Organization), and the World Bank. 2019. Levels and Trends in Child Malnutrition: Key Findings of the 2019 Edition of the Joint Child Malnutrition Estimates. Geneva: WHO. 29 Appendix B. Preliminary Portfolio Review Portfolio Identification Strategy The strategy to identify nutrition projects is summarized in figure B.1. This strategy is preliminary and will be refined during the evaluation. The possible projects were limited to International Bank for Reconstruction and Development and International Development Association lending and recipient-executed trust funds that were active or closed between fiscal year (FY)08 and FY19. Data on basic project features were retrieved from the World Bank’s Business Warehouse (that is, Business Intelligence, Analysis for Office) and merged with data on project development objectives, indicators, and other variables from Implementation Status and Results Reports stored in the World Bank’s Systems Applications Products. Identification Figure B.1. Nutrition PortfolioNutrition Strategy Portfolio Identification Strategy 1. Search 2. Identification 3. Inclusion Evaluation Period Limited to… Relevant Title, Sector and Theme Database C PDO, Components FY08-19 codes (4,260 projects; and Indicators Manual • Agriculture 88 countries) verification with • Education based on key Global Practice Database A • Health concepts of portfolio (10,508 projects) • Social Protection Conceptual (since FY11) by • Public admin. Framework: Health, Ag, Water Business • Water and Nutrition Intelligence, • Private sector • Nutrition-related Global Solutions Analysis for Office • Public sector outcomes Group and SAP Machine • SD, HD and • Immediate and (33 projects added) • IBRD, IDA, and Learning gender underlying RETF • Saliency • Rural determinants of • Project features Scores development nutrition Preliminary (ID, country, • Clustering • Environment • Nutrition-specific nutrition commitments, Scores and sensitive portfolio sector and theme • Matching interventions (390 projects in codes, other) Scores • Enabling 69 countries) • ISR data (PDO, ‘good’ environment indicators, other) projects • Agents of change, Included etc. Additional Financing Database D Parent project met • Text extraction of inclusion criteria Database B Countries with components and (152 countries) high levels of policy area from JME UNICEF/WHO childhood stunting project documents in the Included Stunting rates for >=20%) World Bank’s (357 projects) children under five (88 countries) image bank Source: Independent Evaluation Group. Note: Ag = Agriculture; HD = human development; FY= fiscal year; IBRD = International Bank for Reconstruction and Development; ID = identifier; IDA = International Development Association; ISR = Implementation Status and Results Report; JME = Joint Child Malnutrition Estimates; PDO = project development objective; Public admin. = public administration; RETF = recipient-executed trust fund; SAP = Systems Applications Products; SD = sustainable development; UNICEF = United Nations Children's Fund; WHO = World Health Organization. 30 The basic identification criteria to delimit the potential nutrition projects included (i) sector and theme codes related to agriculture; education; health; social protection; water, sanitation, and hygiene; rural development; public administration; and gender and (ii) the list of countries with high levels of stunting obtained from the Joint Child Malnutrition Estimates (UNICEF, WHO, and World Bank 2019). These criteria limited the database to 4,260 projects in 88 countries with high levels of stunting. For these projects, text of components was extracted from project documents (project appraisal documents for investment financing, project papers for additional financing, and program documents for development policy lending) to enrich the data available on each project. Key nutrition-related concepts and associated keywords in the titles, objectives, indicators, and components of projects were used for the inclusion criteria in a machine- learning exercise. Machine learning provided saliency and clustering scores for the concepts related to outcomes and outputs and nutrition-specific, nutrition-sensitive, and enabling environment interventions linked to the conceptual framework. This helped identify different types of nutrition projects: (i) projects that directly address nutrition in their objectives and (ii) projects that include nutrition interventions in their components. Sample projects were manually reviewed to verify and refine the list of keywords and concepts, including a machine-learning analysis of frequently used phrases in the projects. The projects were also screened against the lists of nutrition-relevant projects previously identified by the Nutrition Global Solutions Group and Global Practices, as part of the active portfolio monitoring. During the analysis phase, the list of projects will be further refined to improve the quality of the portfolio identification strategy. The goal is to identify a representative portfolio of nutrition-related projects to maximize learning related to the evaluation questions and to contribute to ongoing efforts to better track nutrition in projects. Preliminary Portfolio The preliminary portfolio includes 390 projects in 69 of 88 countries with high levels of stunting: 37 percent of the projects have nutrition as a main focus of the project, whereas 63 percent integrate nutrition interventions into one or more of the projects’ components. Most projects are in the Health, Nutrition, and Population Global Practice; followed by Agriculture; Social Protection and Jobs; and Water (figure B.2). Portfolio trends show an increasing emphasis on nutrition during the last 10 years, according to the number of projects approved by fiscal year (figure B.3). Most countries with high stunting rates in 2008 have seen some reduction in their rates during the evaluation period. However, not all those countries received nutrition-related support from the World Bank. In countries with World Bank support (International Development Association, International Bank for Reconstruction and Development, recipient-executed 31 trust fund), the volume of projects to analyze in the evaluation is variable. Figure B.4 presents stunting rates and the number of nutrition projects for countries that have at least two nutrition projects in the preliminary evaluation portfolio. Figure B.2. Preliminary Portfolio of Projects for the Evaluation by Global Practice 2% 1% 1% 3% Health, Nutrition and Population 4% Agriculture 6% Social Protection and Jobs 44% Water Urban, Disaster Risk Management, Resilience and Land 17% Education Macroeconomics, Trade and Investment Governance Social Development 22% Source: Independent Evaluation Group. 32 Figure B.3. Nutrition Portfolio (Project Approvals) over Time 45 40 35 30 No. of projects 25 20 15 10 5 0 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Fiscal year No. of projects Three period moving average (No. of projects) Source: Independent Evaluation Group. 33 Figure B.4. Stunting Rates by Country from 2008 to 2019 and Volume of Projects in the Preliminary Nutrition Portfolio Stunting in children No. of projects Difference under five (percent) 2008 2019 Angola 29.2 37.6 8.4 Djibouti 26.8 33.5 6.7 Papua New Guinea 43.9 49.5 5.6 Mauritania 23.0 27.9 4.9 Nigeria 40.6 43.6 3.0 Chad 38.7 39.8 1.1 Togo 26.9 27.6 0.7 Sierra Leone 37.2 37.8 0.6 Sudan 38.3 38.2 -0.1 Yemen, Rep. 46.6 46.4 -0.2 Madagascar 49.4 48.9 -0.5 Mozambique 43.5 42.9 -0.6 Guatemala 48.0 46.7 -1.3 Burundi 57.6 55.9 -1.7 Timor-Leste 53.9 50.9 -3.0 Congo, Dem. Rep. -3.1 45.8 42.7 Gambia, The -3.1 27.7 24.6 Senegal -3.4 19.9 16.5 Central African Republic -3.5 47.7 44.2 Lao People's Democratic Republic -3.5 43.1 39.6 Indonesia -3.7 40.1 36.4 South Sudan -4.9 36.2 31.3 Cameroon -5.4 37.1 31.7 Pakistan -5.4 43.0 37.6 Zambia -5.8 45.8 40.0 Nicaragua -5.8 23.1 17.3 Lesotho -5.9 39.3 33.4 Vietnam -5.9 30.5 24.6 Rwanda -6.1 44.3 38.2 Country Bangladesh -7.0 43.2 36.2 Cambodia -7.1 39.5 32.4 Mali -7.2 20.8 13.6 Honduras -7.2 29.8 22.6 El Salvador -7.2 37.6 30.4 Liberia -7.5 39.6 32.1 Guinea -7.6 40.0 32.4 Haiti -7.7 29.6 21.9 Zimbabwe -8.0 35.1 27.1 Tanzania -8.6 43.1 34.5 Kenya -9.3 35.5 26.2 Uganda -9.4 38.3 28.9 Ghana -9.6 28.4 18.8 Kyrgyz Republic -9.7 22.6 12.9 India -9.9 47.8 37.9 Congo, Rep. -10.0 31.2 21.2 Bolivia -11.0 27.1 16.1 Benin -11.2 43.4 32.2 Malawi -11.4 48.8 37.4 Armenia -11.5 20.9 9.4 Ethiopia -12.0 50.4 38.4 Nepal -13.2 49.2 36.0 Burkina Faso -14.0 35.1 21.1 Niger -14.2 54.8 40.6 Peru -15.1 28.0 12.9 Comoros -15.8 46.9 31.1 Côte d'Ivoire -17.4 39.0 21.6 Afghanistan -18.4 59.3 40.9 Guinea-Bissau -20.1 47.7 27.6 Tajikistan -21.7 39.2 17.5 -24 -20 -16 -12 -8 -4 0 4 8 12 16 20 24 Difference in stunting in children under five between 2008 a and 2019b (percentage points) | No. of projects Source: Adapted from UNICEF, WHO, and World Bank 2019. Note: The figure shows countries with high levels of stunting and at least two World Bank projects supporting nutrition. The data on stunting are for the years closest to 2008 and 2019 for which country data are available. 34 References UNICEF (United Nations Children’s Fund), WHO (World Health Organization), and the World Bank. 2019. Levels and Trends in Child Malnutrition: Key Findings of the 2019 Edition of the Joint Child Malnutrition Estimates. Geneva: WHO. 35